Evaluation of GRDI Shared Priority Projects

1.0 Introduction

Goss Gilroy Inc. (GGI) is pleased to submit this final draft report for the 2019-20 evaluation of the Genomics Research and Development Initiative (GRDI) Shared Priority Projects (SPPs).

1.1 Background

1.1.1 Overview of GRDI SPPs

GRDI was created in 1999 to establish and maintain core genomics R&D capacity in the following federal departments and agencies (DAs):

• Agriculture and Agri-food Canada (AAFC);
• Environment and Climate Change Canada (ECCC);
• Fisheries and Oceans Canada (DFO);
• Health Canada (HC);
• Public Health Agency of Canada (PHAC);
• National Research Council (NRC);
• Natural Resources Canada (NRCan); and
• Canadian Food Inspection Agency (CFIA) – since Phase V.

Research supported by GRDI seeks to uphold regulatory, public policy, and operational mandates in important areas such as health, food safety, natural resources, agriculture, and environmental protection, while collaborating with academia and the private sector. In Phase V (2011-12 to 2015-16), GRDI supported 73 of these individual DA-led mandated research projects. In Phase VI (2016-17 to 2020-21), GRDI funded 85 research projects in individual DAs to support regulatory, public policy, and operational mandates.

Phase V introduced a new model that included the mobilization of resources for concerted research on issues that are beyond the mandates of single DAs. This model supported the funding of two interdepartmental projects along shared priorities and common goals, referred to as Shared Priority Projects (SPPs). SPPs aim to deliver solutions to enduring and emerging issues for economic, social, and environmental benefits for Canadians. They are intended to leverage existing expertise from the federal research community and are based on a collaborative approach to address priorities identified by the GRDI Assistant Deputy Minister Coordinating Committee (ADM CC) in areas that benefit from an integrated federal genomics R&D approach.^{Footnote 1}

Each SPP is comprised of a series of sub-themes or work packages (hereafter referred to as project activities).

The first round of funding (Phase V) took place from January 2011 to March 2016, and included:

• The Food and Water Safety (FWS) project aimed to improve the ability to detect, diagnose and monitor organisms to ensure a sustainable supply of safe and healthy food and water for human consumption; and
• The Quarantine and Invasive Species (QIS) aimed to improve the ability to detect, identify and understand Canadian biological diversity (including the monitoring of invasive alien and quarantine species) in order to prepare Canadian natural and managed resources, and markets, for global change.

Phase VI SPPs, from April 2016 to March 2021, included two additional projects using the same model:

• The Antimicrobial Resistance (AMR) project uses a genomics-based approach to understand how food production contributes to the development of antimicrobial resistance of human health concern, and to explore strategies for reducing antimicrobial resistance in food production systems.
• The EcoBiomics project develops advanced genomics tools to assess freshwater ecosystem biodiversity and water quality in lakes and rivers, evaluates the health of soil essential to the productivity of agricultural and forestry systems across Canada and investigates soil remediation for the oil and mining sectors.

The end-users for GRDI-funded projects include both internal and external end-users. Internal end-users are the most common type for GRDI-funded projects and include people working inside the federal government, such as laboratory scientists, field inspectors, border agents, trade negotiators and resource managers. External end-users are outside the federal government, and may include industry using a patented technology or revising their processes due to a policy change, and international regulatory agencies using and/or adopting the technology.

1.1.2 Financial Resource Profile

A total of $59.7 million was invested in SPPs in Phase V and $99.5 million in Phase VI. Table 1 shows the spending by participating DAs on Phase V SPPs and Table 2 shows the similar information for Phase VI SPPs.

In addition to these investments, GRDI performance reports for Phase V indicated that an estimated $89.4 million in non-GRDI funding was leveraged from A-base and other sources, such as external collaborators. This amount included cash and in-kind contributions. For Phase VI, it is anticipated that the AMR project will leverage an additional $15 million for a total of $24 million over five years, while it is anticipated that the EcoBiomics project will leverage an additional $6 million for a total of $15 million over five years.

1.1.3 GRDI Horizontal Governance

The ADM CC is an interdepartmental committee comprised of the eight participating GRDI DAs and chaired by NRC. It is responsible for the overall strategic direction of GRDI and ensures that effective priority setting mechanisms are established within DAs and that government objectives and priorities are addressed.

An interdepartmental GRDI Working Group (WG), also chaired by NRC, supports the work of the ADM CC. The WG's mandate is to provide recommendations and advice to the ADM CC regarding strategic priority setting and overall management of GRDI. The WG is responsible for providing direction to GRDI program activities related to operational delivery, implementation planning and investment priority setting. The WG also supports evaluation and reporting requirements related to the Initiative.

The horizontal management of the Initiative is supported by the GRDI Secretariat. The Secretariat supports the ADM CC and the GRDI WG, and communicates the planning cycle, process requirements, financial administration and other project management requirements. The GRDI Secretariat is also responsible for facilitating the SPP planning and peer review processes, ensuring project management plans and funding agreements are in place, and supporting performance management, reporting, evaluations and communications.

1.2 Evaluation Objectives and Scope

The main objective of the evaluation was to gather evidence against five evaluation questions and provide findings, conclusions and recommendations to support the ADM CC in future SPP decision-making. Since three of the evaluation questions pertain to the success of SPPs, reporting on the achievement of SPP outcomes is also an objective of the evaluation.

The last evaluation of GRDI was conducted in FY 2015-16 and completed in FY 2016-17. At the time of data collection, the SPPs funded under Phase V were still underway, although planning for Phase VI had already begun. As a result, the findings and lessons pertaining to the development and selection of SPPs could not be fully integrated into the Phase VI process. As a result, the timing and scope of the current evaluation process were adjusted to allow for more input to the Phase VII SPP development and selection process.^{Footnote 2}

The scope of this evaluation covered both current and past SPPs (Phases V and VI SPPs). The Phase V SPPs ran from 2011-12 to 2015-16 and the Phase VI SPPs are still underway, running from 2016-17 to 2020-21.^{Footnote 3}

1.3 Evaluation Questions

The evaluation was guided by the following evaluation questions:

1. To what extent do the current SPPs align with federal government mandates and current and emerging priorities?
2. Have the SPPs funded as part of Genomics R&D contributed to the development of evidence-based public policy and new knowledge/technologies?
   1. Have the SPPs led to any unique capabilities in the federal public service?
   2. What new scientific achievements have resulted from GRDI SPPs?
3. Have knowledge/technologies been transferred to end-users inside and outside of the federal government?
   1. What has been the impact of the technologies transferred/commercialized?
4. Have the GRDI SPP partners been successful at selecting, managing, collaborating, and achieving results as part of interdepartmental research projects? What changes could be made to have a greater impact?
5. In terms of interdepartmental research collaboration, what lessons learned and best practices have emerged from the SPPs funded as part of Phase V and Phase VI?

1.4 Methodology

The evaluation employed three methods to respond to the evaluation questions. A complete evaluation matrix, which provides the indicators and a crosswalk between the questions and the methods, is presented in Appendix A. The methods for the evaluation included a document and data review, key informant interviews and a bibliometric study. The data collection for the evaluation was conducted between October 2019 and January 2020.

1.4.1 Document Review

The document review was conducted to provide evidence for all evaluation questions. Over 350 documents were provided for review, including those pertaining to GRDI overall, including Initiative-level documents such as agendas and minutes for the ADM CC and WG, Annual Performance Reports, SPP-level documents such as governance and planning documents, workshop materials and reports, as well as documents pertaining to each SPP such as project charters, project budgets and project performance reports.

The document review was conducted by populating an evidence matrix and then preparing summaries for each indicator and question.

1.4.2 Key Informant Interviews

Key informant interviews were conducted to provide evidence against all five evaluation questions (see Appendix B for the interview guides). Only those with knowledge of one or more of the four SPPs were invited to participate as interview respondents for the evaluation. A total of 34 interviews were conducted. This included 33 federal government respondents and one respondent from a federally-funded not-for-profit organization. Of the 34, 19 respondents were members of either the ADM CC or the WG. The balance (15) of respondents were scientists representing one of the eight participating DAs. Table 3 presents the distribution of these interviews by DA as well as by SPP.

An evidence matrix was prepared with the raw interview evidence, organized by respondent type. Evidence was then analyzed by respondent type and, where possible, by SPP.

The interview results presented in the report are identified by respondent type where appropriate. Where there was no difference in opinion by respondent type, the results were presented together. Also, since SPP-specific case studies were not conducted for the evaluation, interview evidence was used and analyzed across SPP where appropriate, or analyzed for each SPP where this made sense. Please refer to Appendix C for SPP specific write-ups.

When interview evidence is presented in the report, the following terminology is used to describe the results: over 90% of respondents = a large majority; >50% to 90% = most or a majority; 25% to 50% = some; and <25% but more than one = a few.

1.4.3 Bibliometric Study

A bibliometric study was conducted by NRC's Library and Information Management Services (LIMS). It aimed to provide evidence to answer evaluation question 3 pertaining to knowledge transfer) and question 4 pertaining to collaboration.

The publication dataset (2011-2018) was provided in Excel by the NRC Audit & Evaluation team for each GRDI SPP.^{Footnote 4} The list included all the metadata necessary for the study, with the exception of author affiliation. A list of GRDI participants/researchers was also provided. LIMS validated, cleaned and removed duplicate publications in each dataset. A publication is defined as a published body of work or a presentation at an official venue. Any author affiliation data was supplemented with internet search result data as appropriate.

The final datasets were approved by the NRC evaluation team. In all, 602 publications were used for the collaboration analysis including 114 for FWS, 220 for QIS, 146 for AMR and 123 for EcoBiomics^{Footnote 5}. A total of 1,008 individual authors were found in the publications data by mapping the names of the researchers provided and using affiliation metadata from Scopus^{Footnote 6}. However, of those, 150 authors could not be associated with an affiliation.

Out of the 602 publications, 145 were indexed in Scopus. Only these publications were used for the bibliometric analysis contributing to findings associated with knowledge transfer (including citation analysis, for example). Out of the 145 publications, 3 were peer-reviewed conference content and included in the analysis.

The datasets were then input into the analysis software VantagePoint, which allows for the creation of various groupings, matrices, graphs, cross-correlations and statistical analyses. Author names and affiliations were cleaned to harmonize variant forms and spellings. Author affiliations were used as disclosed on the publication.

Analyses, visualizations and graphs were generated to answer evaluation questions using Gephi, an open-source social network software. Collaboration graphs were displayed using a ForceAtlas algorithm that emphasizes community detection.

1.4.4 Limitations

The following methodology limitations were noted:

• The data used for the evaluation had to be pieced together using performance reports. This occasionally required manual aggregation and counts of items such as publications. To mitigate this challenge, documentary evidence was triangulated with interview evidence and participating DAs reviewed the SPP write-ups which featured figures derived from manual aggregation. After triangulation with other sources, the performance data in documents was found to be inaccurate (in the case of publications due to the fact that duplicates could not be identified) or not very detailed. The data provided by the GRDI Secretariat for the bibliometric study was used to ensure accurate figures for publications. The examples provided by interviewees were used to provide sufficient level of detail.
• There were no overall summary reports for the completed QIS and FWS projects. Instead, bi-quarterly performance reports, the off-boarding analysis of final outputs and the FWS Knowledge Transfer Workshop deck were used to complete this review.
• With the exception of one external respondent, all interview respondents were currently or had been involved with GRDI SPPs. Thus, there is likely some tendency among respondents to portray a positive outlook on the areas explored during interviews. The rationale for including only one external respondent was that GRDI end-users are largely within the federal government. However, interviewing a larger population of external end-users could have provided valuable insights regarding knowledge transfer and longer-term outcomes.
• Given that the AMR and EcoBiomics SPPs are ongoing, it is not possible to draw strong conclusions about scientific achievements or knowledge transfer (KT) at this point in time. However, the evaluation did include interviews with several scientists involved in both projects where they were asked to comment on expected and potential impacts and KT to end-users.
• The bibliometric study was limited to assessing KT and collaboration. The citation analysis in particular was not used to assess the quality of the research conducted. For example, it is not known how the citations were used or by whom. The analysis assumes that all citations are treated as equal regardless of whether work is cited positively or negatively. A possible mitigation approach would be to combine this analysis with content analysis, but this was outside the scope (budget) for the evaluation. Instead, the citation analysis was only used as evidence of KT and did not assess whether high quality research was conducted.
• The collaboration rates of SPP publications reported by the bibliometric study are likely inflated. In particular, publications include conference presentations and because research results are often presented multiple times at conferences before being published in a peer reviewed journal, there is likely duplication between papers and conference presentations. Note that only 145 of the 602 included publications are indexed in Scopus. Moreover, the reasons for the variance in collaboration rates across DAs was not explored.
• The assessment of impact was limited to what is presented in SPP documentation, such as annual performance reports and the opinions of interview respondents. An analysis of other metrics, such as whether the article was published in a reputable journal, whether other scientists published reviews or comments on a paper and whether those reviews were positive or negative, would give a more reliable understanding of the overall quality of the research conducted. However, this level of analysis was outside the scope (time, budget) of this evaluation. Additionally, many of the tools, processes and methods developed as part of SPPs are not suitable for publication, yet could have impacts. However, these impacts were only assessed through the lens of the opinions of those consulted for the evaluation.

2.0 Findings

The findings from the evaluation are presented in four sections below, beginning with the effectiveness and appropriateness of the development and management of SPPs. This is followed by a discussion of the extent and nature of interdepartmental collaboration. The next section presents the scientific achievements and increased capabilities in the federal government resulting from SPPs. The final section presents the findings related to knowledge transfer (KT).

2.1 Development and Management of SPPs

2.1.1 Development of SPPs

The approach to the development of SPPs is well-documented, but not well understood, likely by those who are not directly involved in the entire process

SPPs were first introduced as part of the Phase V round of GRDI. Broad themes for the research were determined followed by the identification and selection of specific research projects. The approach used in the development of the themes was similar for Phases V and VI and is well documented in various documents, including Governance Frameworks, workshop reports and ADM CC and WG minutes.

To begin, WG members conducted internal/ expert consultations within their own DAs to generate ideas and potential concepts for SPP themes. Following this, an interdepartmental workshop was held where potential themes were discussed. For both phases, this workshop was attended by scientists, policy analysts and managers from participating DAs. From this workshop, a sub-set of themes likely to have significant Canadian benefit, to which several participating DAs could make a meaningful contribution, and which aligned with government and DA priorities, was developed and presented to the ADM CC for discussion and for final decision. The selection of themes was guided by eight criteria:

• Strategic importance or urgency of the opportunity/issue;
• Alignment with Government priorities;
• Relevance of the genomics approach;
• Relevance to the role of federal science and technology;
• Impact/benefits for Canadians (economic, social, and environmental);
• Area of scientific strength, competitive edge both in Canada and internationally;
• Importance of an integrated federal genomics R&D approach; and
• Capacity in more than three GRDI DAs.

Once the themes were developed, a Scientific Coordinator (or Coordinators) for each SPP was identified. These Coordinators then solicited project activity ideas linked with the SPP theme(s) and proposals from GRDI participating DAs. Workshops were held to discuss the potential projects activities and, where appropriate, scientists were asked to make adjustments to their proposals. This iterative approach was described in the documentation as "self-assembling." It is non-competitive and characterized in the Phase VI GRDI Governance Framework (June 2017) as resulting in "strong functional multi-disciplinary teams." As with SPP themes, the proposed suite of research project activities was provided to the ADM CC for consideration. Based on the input provided, the Scientific Coordinator, working with the Project Management Committee, prepared a revised proposal. It then underwent external scientific peer review to validate proposed scientific approaches and for quality control and enhancement purposes. Input from that peer review is used in preparing the final Project Management Plans.

The SPP selection/development process was not well understood by some respondents. During interviews, there was diversity in the descriptions of the process shared by respondents, particularly for how Phase VI themes were developed. Some described the process as being driven by senior management with little input from scientists whereas others described the approach as being both bottom-up and top-down. The bottom-up, top-down process is one in which senior management provided feedback on proposed sub-themes developed at the working level, and then scientists developed projects to address those sub-themes. It is not clear why there was a lack of understanding aside from the likelihood that those respondents were less involved in the entire process.

Most respondents expressed that the current development process is working well and is adequate to meet needs; concerns mostly revolved around the lack of transparency in the selection of project activities

When asked to comment on the process and its relative strengths and weaknesses, most respondents felt that the current process is working well and is adequate to meet the needs of GRDI and participating DAs. Process features that were broadly praised included the involvement of senior management and scientists and the high degree of consultation through workshops.

Those with overall negative impressions of the process were mostly scientists. In terms of process weaknesses, the most commonly cited area, mentioned by a few ADM CC/WG respondents and some scientist respondents was the lack of transparency in the selection of project activities (mentioned by respondents from five different DAs). There was a perception among these few respondents that project activities were pre-determined and that funds were allocated based on existing collaborative networks and capacity and that co-leads took a more directive approach for Phase VI in the selection of scientists and the composition of research teams. There were reported instances where proposed project activities were re-scoped or simply not selected, and it was not clear to the participating scientists why these decisions were made, although other respondents mentioned that there was a fixed budget for each SPP driving the need to select project activities that best contribute to the achievement of overall SPP objectives.

The document review revealed that the last evaluation included a recommendation pertaining to formally defining how SPP project activities are selected, including how and when the input of scientists is considered. However, since the last evaluation was completed after the selection of project activities for Phase VI, a similar concern was raised when Phase VI stakeholders were interviewed for the current evaluation. In response to the recommendation in the last evaluation, the Governance Framework was updated in 2017 with details regarding the selection of project activities including criteria and the expectation that the Scientific Coordinator will explain to scientists why their proposed project activity was not included in the SPP. No other documentary evidence pertaining to the appropriateness of the Phase VI process was available for the evaluation.

Another common area of weakness mentioned by a few ADM CC/WG and scientist respondents was that decision-making by the ADM CC is believed to be influenced by those members with a strong personality.

While most respondents were satisfied with the process to develop themes, sub-themes and select project activities, there were nevertheless several suggestions for how the process could be further improved. In terms of the process overall:

• Encourage earlier and more active engagement of potential end-users, from theme selection through work package development.
• Widely communicate that theme development and project activity selection are not intended to be consensus-based, all-inclusive in terms of DA participation or to advance an individual scientist's or a DA's own research agenda.
• Allow more time for back and forth to narrow the scope of the theme, the development and initiation of project activities to avoid rushing once the funding is available.

In terms of the selection of themes:

• Consider different approaches to identify options for themes such as fore sighting, strategic gap analysis (i.e., mapping existing strengths against future needs and priorities), and seeking guidance from independent advisory bodies such as existing Panels and the Chief Science Officer.
• Optimize the process to ensure themes are not too broad since it is time consuming to narrow the scope. Scoping decisions are then being made by scientists rather than senior management.

In terms of the selection of sub-themes, work packages and project activities:

• Ensure more transparency and oversight for the selection of project activities (including potentially developing and communicating selection criteria).
• Leverage expertise of existing federal organizations to identify best practices that could further strengthen the selection approach.

The development process resulted in SPPs that were well aligned with the priorities of the federal government and that offered additional benefits

A majority of respondents of all types and from most DAs felt that the four SPPs were well aligned with their organization's mandate as well as the mandate and priorities of the federal government. When explaining their rationale for why the alignment was seen to be good, a few respondents explained that the projects represent a balance of both basic research and downstream application as well as protecting people and the environment and enabling economic benefits (e.g., exports, savings from efficiencies).

There was broad recognition among most respondents that SPPs will not match precisely the priorities of all DAs since they are intended to be interdepartmental and cross-cutting. However, a few respondents from two DAs did highlight a misalignment with their own DA's mandate at the sub-theme or project activity level for the EcoBiomics SPP. These few respondents who noted misalignment were also those who felt that the selection of project activities in Phase VI lacked transparency. In essence, they felt that the project activity(ies) put forward by their scientists were not allowed sufficient flexibility to demonstrate alignment to both the theme and the mandate of the individual DA where the scientist worked. These few respondents reported their contributions as relevant and effective to the SPP (and benefits from building genomics capacity), but somewhat limited in supporting the mandate of their DA.

Other benefits offered by GRDI SPPs highlighted by a few respondents each include:

• Creating open science that is available to the public;
• Giving government scientists experience working in shared contexts for the betterment of Canadians;
• Providing innovative solutions to problems that could not have been achieved by one DA; and
• Illustrating horizontal science competency for other programs and organizations to emulate.

There was general agreement with the ADM CC decision that new SPPs develop new areas of research/application rather than build on past SPPs, although a few respondents disagreed

Most respondents agreed with the decision of the ADM CC that Phase VI SPPs should not be a continuation of, or significantly build on, Phase V SPPs^{Footnote 7}. They felt that it was clearly communicated that SPPs would only have funding for five years and that there were many good candidates in other research areas for SPP funding.

However, a few respondents felt that the EcoBiomics SPP was already closely linked with the QIS SPP and that the results of QIS were not appropriately leveraged in the subsequent SPP. According to these respondents, better integrating these two SPPs would have created opportunities to better mitigate risks of detecting pathogens and to support Canada's policy and regulatory obligations with respect to biodiversity protection and management. It is important to emphasize that this opinion was only shared by a small minority of respondents.

Additionally, other respondents made explicit comments about how QIS was appropriately leveraged by EcoBiomics, for example, using the libraries generated by QIS to validate biological classifications in the samples where relevant and the reuse and continued development of the bioinformatics platform.

2.1.2 Management of SPPs

Almost all respondents were of the opinion that Phase VI SPPs are well managed, with some respondents noting that SPP management has improved between Phases V and VI

The document review confirmed that project management artifacts such as project management plans, risk assessment documents and performance reports have been developed. The appropriateness of the management of SPPs was mostly assessed by asking interview respondents their opinions on the matter.

Almost all respondents (of all types, from all DAs) felt that Phase VI SPPs were well managed. Among interview respondents of both types, the following strengths were highlighted (each mentioned by some respondents): regular reporting through ADM CC, frequent project team meetings, role of Management Advisory Committees (MACs) and detailed work plans. A few mentioned that current SPPs are on budget and that funding adjustments have been made as required to ensure efficiency. The documentation pertaining to budgets confirms only minor variances between planned and actual costs of SPPs.

While not directly asked during interviews, some respondents mentioned they believed that the management of SPPs has improved between the two phases. In particular, respondents felt that governance was good, mentioning there were appropriate meeting structures and types in place and that the meeting frequency was also appropriate. They also mentioned that mid-year reporting has been streamlined in Phase VI and that this was appreciated.

In terms of opportunities for improvement, a few ADM CC/WG respondents mentioned that reporting could be improved to better communicate outcomes and impacts for Canadians. A few scientist respondents highlighted challenges with the transition from B-base to A-base funding, including delays and the high level of effort exerted by project managers during the transition.

2.2 Interdepartmental Collaboration

SPPs offer a formal mechanism to enable collaboration and interdepartmental research, although removing barriers (related to IT infrastructure, policy and data management) could further improve collaboration

Through their very nature, SPPs encourage and support interdepartmental collaboration. Most respondents reported that by providing a large amount of funding focused on a single problem, SPPs feature multiple scientists from multiple DAs working together to solve a problem that could not be solved by one DA working independently.

In fact, according to GRDI planning documents, the SPP work packages are designed to leverage and integrate expertise and capacity of multiple DAs. This is leading to interdependencies in workflows, and according to SPP project management plans and some respondents, generating concrete interdepartmental collaborations.

According to some respondents, SPPs also foster trust among scientists from multiple DAs. Some respondents (including ADM CC/WG and scientists) also reported that the relationships formed during Phase V SPPs have been sustained and illustrated to other scientists (i.e., those not involved in Phase V directly) that interdepartmental collaboration can work.

Respondents explained that the interdepartmental collaboration that results from SPPs results in increased networks and access to expertise within government, the development of common protocols and approaches and accelerated achievement of results as opposed to a single DA working on its own. For example, some respondents highlighted that EcoBiomics and QIS SPPs allowed for new collaborations between CFIA and other participating DAs (i.e., DFO, ECCC, NRC) that use the same genomic approaches and techniques, but with which CFIA scientists do not usually interact because they are working on different topics.

Factors highlighted by a few respondents as contributing to the degree of success of interdepartmental collaboration included senior management/DA buy-in, dedicated funding, mechanisms and approaches to centralize data, and reporting/accountability requirements.

In terms of barriers to collaboration, some respondents mentioned insufficient bioinformatics infrastructure (including outdated servers) necessitating work arounds. A few respondents mentioned that Treasury Board (TB) policies were onerous, slowing down and/or hindering the transfer of funds and hiring, and that there are differing approaches to data management (such as common data labels and definitions in databases) among participating DAs.

The collaboration rate on GRDI SPP publications is high compared to other publications in similar subject areas

The bibliometric study explored the extent to which SPP publications^{Footnote 8} were the result of interdepartmental or other types of collaboration. The study found that just over half (53%) of GRDI publications are authored by more than one organization (including government, private sector and academic organizations within and outside of Canada). Among the SPP publications authored by more than one organization, the majority have three to five partners per publication. See Figure 1.

Figure 1: Number of Organizations Collaborating on SPP Publications

Figure 1 - Text version

Pie chart depicting the number of organizations collaborating on SPP publications.

• 1.3 percent (8 total) of SPP publications are authored by 10 or more organizations
• 6.3 percent (38 total) of SPP publications are authored by 6-9 organizations
• 25.6 percent (154 total) of SPP publications are authored by 3-5 organizations
• 19.8 percent (119 total) of SPP publications are authored by 2 organizations
• 47 percent (2863) of SPP publications are authored by 1 organization

At 53%, the GRDI SPP collaboration rate (defined as publications authored by at least two organizations) is low compared to other benchmarks and when considering all subject areas. However, when looking at publications in the more specific and genomics-aligned subject areas of "Agricultural and Biological Sciences Biochemistry" and "Genetics and Molecular Biology," GRDI's SPP collaboration rate is the highest of all benchmarked entities. For example, the SPP collaboration rate is 89% for publications in "Agricultural and Biological Sciences Biochemistry" and 88.4% for publications in "Genetics and Molecular Biology," compared to 73.9% and 69.9% for Canada, respectively. See Figure 2.

Figure 2: GRDI SPP Collaboration Rate in Context^{Footnote 9}

Figure 2 - Text version

Three vertical bar charts depicting GRDI SPP collaboration rate in different contexts, each comparing GRDI to Canada, the U15 Group of Canadian Research Universities, and the world.

In all subject areas:

• GRDI collaboration rate is 52.9 percent.
• Canada collaboration rate is 63.8 percent.
• U15 collaboration rate is 66.6 percent.
• World collaboration rate is 49.4 percent.

In agricultural and biological sciences:

• GRDI collaboration rate is 89 percent.
• Canada collaboration rate is 73.9 percent.
• U15 collaboration rate is 76.3 percent.
• World collaboration rate is 61.8 percent.

In biochemistry, genetics, and molecular biology:

• GRDI collaboration rate is 88.4 percent.
• Canada collaboration rate is 69.9 percent.
• U15 collaboration rate is 71.2 percent.
• World collaboration rate is 61 percent.

All participating DAs and most SPPs have overall collaboration rates above 50%

Considering^{Footnote 10} the collaboration rate by DA (including collaboration with other federal DAs, international, academic and corporate) across SPPs, the highest rate is for DFO at 90.5% followed by NRCan and HC, at 75.9% and 75.0%, respectively. The DAs with the lowest collaboration rates are AAFC^{Footnote 11} (at 54.9%), PHAC (at 59.9%) and CFIA (at 64.1%). These three DAs did, however, report the highest number of publications by far, meaning that the absolute number of co-authored publications is higher for these three organizations than any other. See Figure 3.

Figure 3: Collaboration Rates by DA, Across SPPs

Figure 3 - Text version

Horizontal bar chart depicting collaboration rates by departments and agencies, across SPPs.

• GRDI collaboration rate is 52.9 percent
• DFO collaboration rate is 90.5 percent
• NRCan collaboration rate is 75.9 percent
• HC collaboration rate is 75 percent
• NRC collaboration rate is 71.9 percent
• ECCC collaboration rate is 71 percent
• CFIA collaboration rate is 64.1 percent
• PHAC collaboration rate is 59.9 percent
• AAFC collaboration rate is 56.1 percent

Table 4 presents the collaboration rate by SPP. With the exception of FWS, over half of SPP publications for each project involve more than one organization. The current SPPs show high percentages of co-publications involving multiple non-government Canadian organizations, and over a fifth of publications from completed projects involve international collaboration. Looking at all four SPPs, QIS has the highest collaboration rate (at 57.7%), followed by AMR at 54.8%. FWS has the highest incidence of single-organization authored publications (57.9%) followed by EcoBiomics (47.1%). Recall that both AMR and EcoBiomics are still in progress and their total number of publications will continue to increase.

The collaboration analysis conducted for the bibliometric study included an assessment of the nature and extent of the collaborations between different authoring federal DAs. Figure 4 presents these network diagrams for each SPP. The relative size of the bubbles corresponds to the number of publications by that organization. The thickness of the connecting lines and associated numbers indicate the number of publications co-authored between those organizations.

Figure 4: Nature and Extent of Collaborations

Figure 4 - Text version

A series of network diagrams depicting the nature and extent of the collaborations between different authoring departments and agencies – one diagram per SPP. The relative size of the bubbles corresponds to the number of publications by that organization. The thickness of the connecting lines and associated numbers indicate the number of publications co-authored between those organizations.

FWS network diagram:

• PHAC is the most frequent author for the FWS SPP, followed by NRC. CFIA, HC and AAFC followed by NRC then ECCC.
• NRC and HC, as well as NRC and AAFC have the highest number of co-authored publications, with a recorded volume of 6.

QIS network diagram:

• AAFC is the most frequent author for the QIS SPP, followed by CFIA, DFO and NRCan. The least frequent authors are HC, PHAC, NRC and CGC.
• AAFC and CFIA have the highest number of co-authored publications, with a recorded volume of 23 publications. CFIA and NRCan come second with 16 co-authored publications.

AMR network diagram:

• AAFC is the most frequent author for the AMR SPP, followed by PHAC CFIA, then HC.
• AAFC and PHAC have the highest number of co-authored publications, with a recorded volume of 12 publications. AAFC and CFIA come second with 11 co-authored publications.

EcoBiomics network diagram:

• ECCC is the most frequent co-author for the EcoBiomics SPP, followed by NRCan and AAFC. The least frequent authors are NRC, CCFIA, DFA and PHAC.
• ECCC and NRCan have the highest number of co-authored publications, with a recorded volume of 9. NRCan and NRC come second with 8 co-authored publications.

The figure shows that QIS has the most DAs participating as co-authors whereas AMR has the fewest. The magnitude of co-authorships for EcoBiomics might suggest strong linkages within that project. However, both of these SPPs are still underway and it is too early to know the true extent of collaboration at this time.

For FWS, PHAC had the most publications and had co-authored 13, most often with CFIA or NRC. However, NRC had fewer overall publications, but had co-authored 20 publications, most often with HC and AAFC. ECCC had only one FWS publication, which was not co-authored with another DA.

For QIS, all participating DAs had co-authored publications. CFIA links to all the other DAs and AAFC links to all DAs but one (ECCC). AAFC had the most co-authored publications and the strongest linkage with CFIA (with 23 co-authored publications). CFIA also had a large number of co-authored publications and linkages with three other DAs (NRCan, DFO and AAFC) each including more than 10 co-authored publications.

For AMR (up to and including 2018), only four DAs have co-authored publications, but each DA has co-authored with each of the other DAs. The strongest linkages are between AAFC, and PHAC and between AAFC and CFIA.

For EcoBiomics (up to and including 2018), there have been many co-authored publications amongst the various participating DAs with most DAs having co-authored at least one publication with each of the other DAs. ECCC has the largest number of co-authored publications and strong linkages with NRCan, followed by DFO and AAFC. NRCan and AAFC both have a large number of co-authored publications. NRCan has particularly strong linkages with five DAs (including ECCC, NRC, AAFC, CFIA and DFO). AAFC has strong linkages with ECCC, NRCan and CFIA.

2.3 Scientific Achievements and Increased Capabilities in the Federal Government

The SPPs have contributed to important scientific developments in the field of genomics, and have had a positive impact on the application of genomics in the federal government

The SPPs support the activities of over 150 research scientists across federal DAs, plus hundreds of collaborators and highly qualified personnel. The vast majority of respondents confirmed that the research conducted would not have been possible, or would have been lower-impact and less cutting-edge, without the resources and collaborative model of the SPPs. Some ADM CC/WG respondents described the SPPs as a unique example of horizontal scientific collaboration, which they saw as increased federal capability in and of itself. Participating DAs also reported collaborative developments made possible by the SPP networks beyond the scope of the individual projects. For example, some research components of FWS and QIS continued after the SPPs ended.

The SPPs have created new and increased existing capabilities in federal DAs and agencies, accelerating progress on the use of genomics in government. Some respondents explained that the SPPs have contributed to putting Canada "technologically ahead" on the use of genomic tools for regulatory activities and other purposes. As one respondent described, the SPPs contribute to advancements where "genomic-informed approaches, vastly superior in terms of comprehensiveness and resolving power, are replacing traditional laboratory approaches". In 2016, the leadership teams of the GRDI SPPs were awarded the Public Service Award of Excellence in the category of Scientific Contribution.

The SPPs have generated important scientific achievements in the fields of genomics itself. Importantly, all four SPPs have compiled and disseminated a large amount of genomic data, now accessible to research communities in and outside government. Respondents frequently point to the sheer amount of information generated as a major scientific development, with the potential to support further research and ultimately inform end-users' actions and decisions. For instance, the FWS SPP saw the development of the Integrated Rapid Infectious Disease Analysis (IRIDA) platform: an end-to-end system for the storage, management, analysis, sharing, and reporting of whole genome sequence data for genomic epidemiology. The QIS project led to the development of high-quality molecular DNA sequence datasets for high-risk species, which are used to document organisms found in Canada. The AMR SPP is developing a large collection of bacterial sequences that are enabling a better understanding of the spread of AMR in the food supply. The EcoBiomics project has provided new reference material into ECCC's Canadian Aqua Biomonitoring Network (CABIN), which monitors the health of freshwater ecosystems. The QIS database is used by DFO as a reference library for fish identification and monitoring. The EcoBiomics SPP is expected to yield the first large scale biodiversity genomic database. Across all projects, a majority of participating scientists underlined that the collaborative model of the SPPs is instrumental in generating this foundational material: no one DA could generate the same amount of data, and coordination is vital to develop effective and harmonized approaches for data sharing.

The SPPs have also led to the development of new genomic research methods (including Standard Operating Procedures (SOPs)) and tools that have been successfully implemented in federal DAs and beyond. These outputs, such as new guidelines on sample management and new techniques for genome sequencing, positively impact capabilities in government by accelerating and improving the accuracy of results, and making genomic research more accessible and cost-effective.

In turn, breakthroughs in genomic research and the development of new technologies, through the SPPs, have contributed to improved capabilities within the federal government to detect and study invasive species, food borne pathogens, antimicrobial resistance and the impacts of climate change. SPPs have also deepened scientists' understanding of the ways threats are influenced by different factors like changes in the environment or human activities.

Capabilities developed through the SPPs have contributed, or are expected to contribute, to the work of government DAs

The majority of ADM CC/WG members and scientists interviewed can link SPP developments to actual or expected downstream impacts in the federal public sector. New data, knowledge and technologies developed through the SPPs have impacted (or have the potential to impact) policy, regulations and other government work, as illustrated in the examples below, also taken from the document review for each SPP.

The extent of SPPs' contribution to long-term outcomes is hard to measure since five years is a short time for research to generate socio-economic benefits

Some interview respondents across categories emphasized that five years of scientific research is a short period of time to generate socio-economic outcomes. These respondents questioned whether the type of research conducted through the SPPs should be expected to generate immediate tangible benefits after just five years. Overall, there is limited SPP documentation describing the concrete impacts of knowledge or technologies transferred to end-users, especially outside government. Evidence on the socio-economic benefits of activities was collected mainly from GRDI SPP participants and senior management of departments involved. Respondents also felt it would be too early to identify long-term impacts of ongoing SPPs, but could point to important results achieved to date. AMR and EcoBiomics documentation refer to potential downstream impacts.

Some respondents – among both ADM CC/WG members and scientists – raised issues related to post-project sustainability and opportunities lost after SPP funding runs out. For instance, these respondents noted that there is no plan for continued knowledge transfer (KT) beyond five years, no established mechanism to continue leveraging the network created, and no avenues to continue to pursue commercialization of promising technologies. The respondents were concerned that positive developments could not continue beyond the life of the SPPs, which in their view limits the projects' downstream impacts. For example, although some project activities of QIS are continuing (e.g., work on plant viruses conducted in British Columbia has received funding in 2017), other components have stopped altogether following the end of the project. Furthermore, there are examples of expected potential impacts for both FWS and QIS, where technologies could not be transferred and adopted due to discontinuation of SPP funding.

Some respondents recommended that GRDI SPPs be designed so that specific fruitful components of an SPP can be identified during the project and selected for targeted support beyond the end of the SPP funding period. Alternatively, strong strategies would have to be formulated by interested DAs to continue to support these components beyond the SPP, to more fully deliver on objectives.

The sections below describe some of the mid to long-term outcomes that can be attributed to the completed SPPs.

2.4 Knowledge Transfer (KT)

Considerations for effective KT are built into the SPPs. Evidence shows that it is effective between federal DAs and with the broader research community

Ensuring the creation of value for intended end-users and knowledge/technology transfer are underlying principles of the SPPs. In 2015, the GRDI Innovation Management Strategy (IMS) was developed in part to improve technology transfer and reinforce the full engagement of end-users. The IMS provided guidance on designing SPPs and preparing project management plans "to improve end-user engagement and the uptake of project deliverables" in Phase VI.

Each SPP's KT plan lists potential end-users in and outside government, and describes intended engagement strategies. SPPs (and their different work packages) entail a variety of KT activities including: direct collaboration and communications with other research units and with end-users; workshops; peer-reviewed publications; participation in conferences; involvement in expert meetings, committees, and advisory panels; contribution to databases and libraries, and media releases. SPP researchers are also able to transfer knowledge through their participation in specialized fora in Canada and internationally. Of course, the degree of KT achieved to date varies between the completed and ongoing SPPs: evidence of KT is strong for the completed projects, while KT activities are still ongoing for AMR and EcoBiomics.

The evidence shows that effective KT is taking place between scientists based in participating DAs. Several respondents noted that the ongoing SPPs have seen improvements in terms of coordination, and KT between research groups across DAs. A few respondents emphasized that regulation-related SPPs must continue to be carefully designed and connected to official channels in order to inform regulation development and implementation.

The volume and visibility of research outputs in the form of publications (conference presentation and participation; peer-reviewed articles, books and reports) is an indicator of KT, dissemination and collaboration. The bibliometric study conducted as part of the evaluation revealed that the SPPs have generated a high number of scientific publications. Conference presentations and participation are the most frequent type of research output in the case of GRDI and SPPs. The projects have also generated articles, books, reports and other documents. A bibliometric analysis of titles available in Scopus (i.e., those appearing in peer-reviewed journals) indicates that these publications are being leveraged and cited to a significant extent. The collaborative network analysis of publications illustrates that SPP research teams are connected to their respective research and scientific community.

The detailed results of the analysis of publications per SPP is presented below, along with of key examples of KT activities.

Food and Water Safety

Food and Water Safety - Text version

Pie chart depicting Food and Water Safety SPP-related publication formats.

• 50.9 percent of publications are in the form of conference presentations
• 31.6 percent of publications are in the form of conference participation
• 14.9 percent of publications are in the form of articles
• 2.6 percent of publications are in the form of books and reports

A total of 114 FWS SPP-related publications were identified for analysis in the bibliometric analysis, featuring 216 scientists in 10 countries and 41 organizations, including six Canadian federal DAs as noted above. As is the case with GRDI research outputs generally, most publications are in the form of conference presentations or participation (94 out of 114; 82%). The remaining 18% are composed of articles (15%) and books or reports (3%). About 42% of FWS publications are co-published by multiple organizations.

The bibliometric analysis also looked at citation rates: for 23 publications indexed in Scopus, the average citation per document is 18.1, with a positive high Field-Weighted Citation Impact^{Footnote 12} (FWCI) score of 1.92, comparable to that of GRDI publications overall (1.95).

A few FWS scientists interviewed described that sharing information and data among DAs was challenging at the start of the project. Participants collaborated to reduce barriers, but legal and administrative constraints in some DAs initially hindered the flow of information. This issue was brought to the attention of the ADM CC with a view to develop an overall Memorandum of Understanding (MOU) between DAs, but this did not materialize during the lifespan of the project. This MoU has since been implemented and is being used for collaborative and other, independent work.

Key among KT activities for FWS is the 2-day Workshop held at the National Library and Archives in Ottawa in February 2016. Documentation reviewed about this event shows that the workshop attracted 85 delegates from across Canada, including food industry representatives, researchers, regulators, public health officials and manufacturers of rapid diagnostic tools. Its purpose was to showcase some of the innovations of the FWS project and foster collaboration between researchers and end-users to boost the commercial uptake of tools and processes. Industry, academia and government discussed ongoing challenges in the world of food safety and public health, remaining gaps in science, possible applications for FWS tools and platforms, and continued communication and collaboration.

Interviews and project reports show that members of the FWS project also attended and played a leadership role at international meetings on whole-genome sequence-based infectious disease outbreak investigations. As noted previously, FWS tools for genomic epidemiology and detection of outbreaks are used through the federal surveillance program PulseNet. PulseNet is now engaged in transferring this capacity to provincial public health laboratories.

Quarantine and Invasive Species

The bibliometric analysis shows that QIS has produced a large volume of highly-cited research outputs, involving a very extensive network of collaborators. A total of 220 QIS SPP publications were identified for analysis, from 394 scientists across 37 countries and 173 organizations, including nine Canadian federal DAs. About six in ten QIS publications are in the form of conference presentation or participation (132 of 220; 60%) and almost a third are peer-reviewed articles (29%). Based on the bibliometric analysis of 59 publications indexed in Scopus, the average citation per document is an impressive 50.8, with a positive high FWCI score of 2.09, higher than for GRDI publications in general (1.95).

Quarantine and Invasive Species - Text version

Pie chart depicting Quarantine and Invasive Species SPP publications formats.

• 45.9 percent of publications are in the form of conference presentations
• 28.6 percent of publications are in the form of articles
• 14.1 percent of publications are in the form of conference participation
• 5.9 percent of publications are in the form of books and reports
• 5.5 percent of publications are in other forms

The document review for QIS shows that data sharing, integration and end-user training were successful outcomes of the bioinformatics project activities. The project informed transnational conversations with trade partners on harmonizing terminology about invasive alien and quarantine species. One ADM CC/WG respondent added that QIS information on invasive species and ecosystems has also been valuable to municipal and provincial governments.

Importantly, according to two interviewees, QIS has been a key factor in AAFC receiving funding to digitalize its genome sequence collection (2016 budget, $30 M). QIS data will be put in the public domain via a project with the University of Guelph (funded by AAFC).

Antimicrobial Resistance

A total of 146 AMR SPP publications were identified in the bibliometric analysis, involving 300 scientists across 17 countries and 79 organizations, including four Canadian federal DAs. Since AMR is an ongoing project, it is expected that this number will continue to increase until 2021. Six in ten of AMR publications are in the form of conference presentations or participation (88 of 146; 60%) and over a third are articles (35%).

Antimicrobial Resistance - Text version

Pie chart depicting Antimicrobial Resistance SPP publication formats.

• 34.9 percent of publications are in the form of articles
• 33.6 percent of publications are in the form of conference presentations
• 26.7 percent of publications are in the form of conference participation
• 4.1 percent of publications are in other forms
• 0.7 percent of publications are in the form of books and reports

The bibliometric analysis examined 46 publications indexed in Scopus and the average citation per document is 9.2. AMR publications have a FWCI score of 1.61.

The document review provided evidence of other important KT activities for AMR including monthly meetings with the Canadian Integrated Program for Antimicrobial Resistance Surveillance (CIPARS), as well as participation of AMR researchers in the Joint Programming Initiative on AMR (JPIAMR) and the Transatlantic Taskforce on Antimicrobial Resistance (TATFAR). Project documentation and interviews referred to the fact that AMR researchers have been involved in expert meetings of the United Nations Food and Agriculture Organization and World Health Organization on foodborne antimicrobial resistance.

Research into alternatives to antimicrobials is being conducted at farms, in direct collaboration with producers (e.g., pig producers in Quebec). Respondents also noted that the AMR SPP team works to engage the human health care sector, although this was described as more challenging, given the need to engage provincial jurisdictions and the complexity of health-related activities such as clinical trials and accessing isolates from human origin from provincial public heath institutions (due to confidentiality reasons). Finally, the open software tool, StarAMR, which helps detects AMR genes, was disseminated freely via GitHub^{Footnote 13} and public archives, facilitating its uptake in academia and other government jurisdictions.

EcoBiomics

A total of 123 EcoBiomics SPP publications were identified in the bibliometric analysis, featuring 179 scientists across 8 countries and 46 organizations, including seven Canadian federal DAs (ECCC, AAFC, NRCan, CFIA, NRC, DFO, and PHAC). About 70% of EcoBiomics publications relate to conferences (87 of 123; 71%) and 17% are articles. EcoBiomics scientists interviewed expect that the integration of data during the last year of the SPP, after four years of sampling across different research projects, will generate a large volume of additional research products. For example, some scientists mentioned an upcoming peer-reviewed research paper to be published in the Proceedings of the National Academy of Sciences regarding genomics applied to a very large complex ecosystem, and described this output as innovative and ground-breaking.

EcoBiomics - Text version

Pie chart depicting EcoBiomics SPP publication formats.

• 52.8 percent of publications are in the form of conference presentations
• 17.8 percent of publications are in the form of conference participation
• 17.1 percent of publications are in the form of articles
• 7.3 percent of publications are in other forms
• 4.9 percent of publications are in the form of books and reports

The bibliometric analysis included 21 publications indexed in Scopus: the average citation per document is 12.6 and publications have a high FWCI score of 2.35.

Interview respondents explained that end-users were engaged from the start of the project, given that EcoBiomics built upon previous research efforts such as the QIS SPP, and established relationships. Each EcoBiomics project component benefits from the engagement of various end-users through existing sampling programs across the country. Project documentation shows that organizations engaged from inception include the CABIN network, the National Water Quality Monitoring Program and the Canadian Agri-Food Policy Institute. Respondents added that end-users have been increasingly engaged and consulted since the beginning of the project. Members of the EcoBiomics project engaged regularly with ECCC and the Ontario Ministry of Environment to integrate metagenomics in federal and provincial programs in the Great Lakes.

Engagement of end-users and KT to those stakeholders have been strengthened between Phase V and Phase VI, but could be further improved

The above examples show that KT to end-users has been occurring in the context of the SPPs. Most interview respondents across categories confirmed that KT to external end-users occurred to some extent, but remained an area for further improvement.

Some ADM CC/WG respondents felt a need to increase the focus on KT to end-users by further tailoring project communications to the intended audience and strengthening engagement activities for uptake and application. On dissemination, a few ADM CC/WG respondents suggested that KT could be improved through more meaningful and impactful narratives. End-users can be organizations present in a variety of sectors, including industry for example. The respondents felt that SPP successes are poorly communicated and that the solutions proposed, given their complexity, remain difficult to understand by audiences that do not already possess expert scientific knowledge on genomics. This makes it difficult for stakeholders to leverage SPP outputs in practical initiatives. Respondents thus suggested using more plain language and alternative dissemination mechanisms (such as YouTube videos, podcasts). Some scientists identified live workshops and gatherings as particularly effective both for liaising with end-users and with other GRDI scientists. A few respondents suggested that more regular face-to-face interactions between participating scientists, collaborators and end-users could improve KT and relationship building.

Regarding direct engagement with external stakeholders, some senior managers and participating scientists underlined the importance of early and regular consultations and continuous exchange. Most respondents involved in ongoing SPPs indicated that early involvement of end-users was critical to facilitating KT outside of government. Direct involvement of collaborators in the development of tools and processes, for example, was correlated with uptake in different sectors. However, a few ADM CC/WG respondents noted that transferring knowledge or technology to industry (as intended in the AMR project, for instance) is not consistent with the mandate of most DAs. While NRC is well equipped to establish partnerships with the private sector and regulatory agencies, other DAs do not have a mandate to engage and transfer technology with organizations outside the federal government.

3.0 Conclusions and Recommendations

The evaluation found that the Shared Priority Projects (SPPs) are generally well-developed and well-managed. The overall process to develop and select themes and project activities was found to be largely effective and appropriate. Themes and project activities are aligned with federal government priorities and offer a mix of economic, environmental, health and safety impacts as well as benefits for both regulatory and operational federal departments and agencies (DAs). The selection of themes was found to be particularly well-done: SPPs are selected to generate new knowledge for the benefit of science-based DAs and end-users.

The broad nature of the SPP themes allows for alignment with individual DA mandates. Any areas of inconsistency are due to the fact that SPPs are not intended to be perfectly aligned to the mandate of any single DA. This was well understood by most of those consulted for the evaluation.

Having said that, a few DAs felt some components of their organization's proposed contribution were not given proper consideration and deplored that their proposed project activities were eventually excluded from the SPP or rescoped. These respondents were unsure why this was the case and felt it was not clear what was expected. However, this view was expressed by a minority. Most respondents, and the documentation describing the process to develop themes and select project activities, emphasized that DA participation in SPPs is not intended to be all-inclusive and involve all eight GRDI DAs. In fact, each SPP is required to include the participation of a minimum of three DAs and build an integrated project that maximizes the potential for interdepartmental exchange.

The lack of transparency of the selection process was mentioned by at least one respondent in five different DAs. The last evaluation had a recommendation to formally define how SPP project activities are selected, including how and when the input of scientists is considered and the Governance Framework was updated accordingly. However, since the last evaluation was completed after the selection of projects for Phase VI, the recommendation from the previous evaluation was not acted upon prior to this issue also being raised during the current evaluation.

Other proposed improvements to the development of themes and selection of project activities included: seeking guidance from different analyses (such as a strategic gap analysis) and stakeholders to inform themes; starting the SPP theme development process earlier, and reinforcing messaging that SPPs are not intended to be aligned with DA mandates.

A common opportunity for improvement that emerged pertained to the engagement of end-users, including those within and outside of the federal government, but with a particular emphasis on the former. Currently, the SPP development process includes a series of workshops involving various stakeholders. However, there is room for stronger representation of end-users at SPP development workshops. It is possible that a few DAs choose to send one or two potential end-users, but this is not widespread nor a purposeful approach to selecting workshop attendees. Since the evaluation also found that the transfer of knowledge to end-users is an area of weakness for SPPs, early involvement of end-users in the development of project activities, as well as the development of the SPP's knowledge transfer plan, should be required.

Knowledge transfer was the subject of one of the recommendations from the last evaluation, which called for the development of a plan that would explore how technology and knowledge developed during SPPs would be transferred to, and used by, end-users. The current evaluation confirmed that this has occurred and further found that knowledge transfer occurs effectively through the SPPs, notably via interdepartmental and academic collaborations, the direct contribution of researchers to various activities, and a large volume of highly cited publications. Most interview respondents across categories confirmed that knowledge transfer to external end-users does occur, but remains an area for further improvement. Additionally, respondents indicated a need to better communicate SPP successes through products and activities that make research outputs and findings accessible to different audiences. This requires meaningful narratives suitable to end-user audiences that do not have advance knowledge of genomics, plain language, and alternative dissemination mechanisms to reach diverse stakeholders. Face-to-face interactions were also identified as an important channel for knowledge transfer and relationship building. Early engagement and involvement of stakeholders and end-users through the SPPs would facilitate knowledge transfer and uptake within and outside of government after five years. However, transferring knowledge or technology to industry may not be consistent with the mandate of all DAs.

The evaluation found that interdepartmental collaboration is occurring as a result of SPP funding and support. This is understandable given that the original rationale for the development of SPPs was to conduct research that was beyond the mandate of any one DA. In spite of this, the extent and nature of the interdepartmental collaboration is notable. In fact, the interdepartmental collaboration that is taking place through the SPPs is believed to be unique within the federal government and serving as a model for other science programs.

Not only are scientists producing publications and making presentations at conferences, they are doing this jointly with scientists from other organizations. The overall collaboration rate for SPP publications (i.e., publications authored by more than one organization) is over 50%. In fact, compared to other research bodies working in similar subject areas, GRDI SPP scientists have far more co-authored publications. Additionally, all participating DAs are collaborating on publications at a rate of over 50% (and some DAs at a rate over 75%).

Senior management buy-in, dedicated funding, trust, professional networks that provide access to expertise and shared protocols and approaches were all identified as enablers of collaboration. All these elements are supported by SPP funding and governance mechanisms. Factors hindering collaboration do exist (including bioinformatics infrastructure, TB policies and ongoing differing approaches to data management), but SPP scientists have found work arounds for most of these barriers.

In terms of scientific achievements, SPPs have generated important datasets, as well as ground-breaking methods and tools. Those breakthroughs have created new and strengthened existing genomic research capabilities in DAs, with concrete applications in the areas of research, policy-making and regulations. Respondents were adamant that without the funding and the unique collaborative framework of the SPPs, these developments would not have occurred, at least not to the same extent or as quickly. However, the long-term impacts of SPP knowledge or technologies transferred to end-users, especially outside government, are hard to capture even for completed projects. On the one hand, five years is a short time to see the socioeconomic outcomes of scientific research. On the other, project senior management and scientists are concerned to see promising developments end with the SPPs, as has been the case with aspects of FWS and QIS, thus limiting downstream benefits.

4.0 Management Response and Action Plan

Appendix A – Evaluation Matrix

The 2019-20 Evaluation of GRDI Shared Priority Projects (SPPs) will provide follow-on information since the March 2017 Horizontal Evaluation of the Genomics R&D Initiative that examined the relevance and performance of the program during fiscal years 2011-12 to 2013-14 plus, specifically with respect to work associated with Shared Priority Projects, for fiscal years 2014-15 and 2015-16. Consequently, this evaluation will focus on the period since 2015-16. Further, it will focus only on the SPPs:

• Phase V SPPs:
  • Food and Water Safety (FWS)
  • Quarantine and Invasive Species (QIS)
• Phase VI SPPs:
  • Antimicrobial Resistance (AMR)
  • Metagenomic-Based Ecosystem Biomonitoring (EcoBiomics)

Appendix B – Instruments

Horizontal Evaluation of the Genomics R&D Initiative Shared Priority Projects (SPPs)

Interview Guide –ADM-CC and IWG members

Context

On the behalf of the National Research Council of Canada (NRC), Goss Gilroy is conducting a horizontal evaluation of the Genomics Research & Development Initiative (GRDI) Shared Priority Projects (SPPs). The main goal of the evaluation is to assess the relevance and performance of the SPPs with regard to their expected outcomes. The scope includes both the completed SPPs, including Food and Water Safety (FWS) and Quarantine and Invasive Species (QIS), as well as the two current SPPs, including Antimicrobial Resistance (AMR) and Metagenomic-based Ecosystem Biomonitoring (EcoBiomics). The expectation of the evaluation is to provide evidence-based information to the GRDI ADM-CC.

The evaluation methods include interviews, document and literature review and a review of administrative data. Your responses will be reported in aggregate only, and you will not be personally identified in any reports.

The interview will take approximately 60 minutes.

Thank you for your collaboration.

Background

1. Please briefly describe your involvement with the GRDI over time.

Relevance

2. To what extent are the current SPPs aligned with your department's / agency's current mandate? (1.2)^{Footnote 14}

1. In what respect(s), if any, is there a lack of alignment?
2. Did this lack of alignment exist since the beginning of the SPPs (i.e., inherent to the selection of the SPPs) or was there a change in your department's / agency's mandate in the last three years?

3. Thinking of the process used to select the current SPPs, in your view, did that process sufficiently allow for alignment with your department's / agency's mandate and priorities? Why/why not? (1.1)

1. Thinking more broadly, did the selection process ensure alignment with the priorities of the Government of Canada? Why/why not? (1.1)

Selection and Management of SPPs

4. What were the strengths and weaknesses of the process to select the two currently funded SPPs (EcoBiomics and AMR)? (3.1)

1. How could the selection process have been improved? (3.1)
2. What aspects of the selection process should be maintained for the next round? (3.1)
3. Are there other similar programs from which best practices around selection of shared priority projects can be gleaned? What are these programs?

5. From your knowledge of the two currently funded SPPs, to what extent are they being appropriately managed? (3.2)

[Probe: Management includes oversight, governance, task assignment, reporting, project management in terms of managing budget, timelines and scope]

1. How could the management of the SPPs be improved? (3.2)
2. How could the SPPs be more efficient? (3.3)

6. Did they enhance interdepartmental collaborations as expected? Please explain. (3.4)

1. Did the two currently funded SPPs provide innovative solutions to problems that could not have been achieved by one department / agency? (3.3)
2. To the best of your knowledge, what are the key factors that have facilitated or hindered the partners' abilities to collaborate as part of an interdepartmental project?

Effectiveness

7. Thinking of the four funded SPPs (FWS , QIS, EcoBiomics, AMR), what have been the actual and potential impacts of these projects? (2.3)

1. At your department / agency?
2. At other federal departments / agencies?
3. At other organizations outside the Government of Canada?

[Probe: Any new unique capabilities in the federal public service and scientific achievements]

8. In your view, to what extent has the knowledge and/or technology developed by the funded SPPs met (or are likely to meet) the needs of users, including: (2.6)

1. Your department / agency?
2. At other federal departments/ agencies?
3. At other organizations outside the Government of Canada?

9. For technologies that have been transferred and/or commercialized, what has been the impact? If technologies have not yet been transferred, what are the potential impacts?

[Probe:
QIS LT impacts: Canadian biodiversity and trade protected from the impacts of global change through improved ability to monitor invasive alien and quarantine species
FWS LT impacts: Improved food safety and security in Canada
AMR LT impacts: Informed public health and food production decisions to address one of the most serious global health threats facing the world today; reduced antimicrobial resistance in food production systems
EcoBiomics LT impacts: A comprehensive perspective of water and soil as living systems that sustain ecosystem services and economic activities; environmental management and resource development decisions informed by monitoring of soil and freshwater quality (with the project's objective being to: characterize the complex microbial and invertebrate biodiversity of soil and freshwater through novel metagenomics approaches)]

10. How effective were the approaches to transfer the new knowledge and technology? (2.5)

[Probe: including open data, level of accessibility and transparency of the information provided to end users]

11. What are some of the factors that positively or negatively influence knowledge / technology transfer to end-users? (2.5)

12. How likely is it that the level of impacts achieved to date through the SPP projects would have been achieved in the absence of GRDI SPP funding? (2.7)

Lessons Learned

13. From your perspective, what are the main lessons learned and best practices stemming from the four SPPs conducted to date? (4.1)

Additional comments

14. Do you have any additional comments for this evaluation?

Thank you.

Horizontal Evaluation of the Genomics R&D Initiative Shared Priority Projects (SPPs)

Interview Guide – Project Leads and Researchers – Current SPPs

Context

On the behalf of the National Research Council of Canada (NRC), Goss Gilroy is conducting a horizontal evaluation of the Genomics Research & Development Initiative (GRDI) Shared Priority Projects (SPPs). The main goal of the evaluation is to assess the relevance and performance of the SPPs with regard to their expected outcomes. The scope includes both the completed SPPs, including Food and Water Safety (FWS) and Quarantine and Invasive Species (QIS), as well as the two current SPPs, including Antimicrobial Resistance (AMR) and Metagenomic-based Ecosystem Biomonitoring (EcoBiomics). The expectation of the evaluation is to provide evidence-based information to the GRDI ADM-CC.

The evaluation methods include interviews, document and literature review and a review of administrative data. Your responses will be reported in aggregate only, and you will not be personally identified in any reports.

The interview will take approximately 60 minutes.

Thank you for your collaboration.

According to our records, you were involved in the [NAME OF PROJECT] SPP. Is this correct? If not, how have you been involved in the GRDI/SPPs?

Background

1. Could you please describe your involvement in this project?
2. Could you please explain what need is driving this research? In other words, why is this research being conducted?
3. To

   Relevance

   4. To what extent is the project aligned with your department's / agency's current mandate? (1.2)^{Footnote 15}
   1. In what respect(s), if any, is there a lack of alignment?
   2. Did this lack of alignment exist since the beginning of the project (i.e., inherent to the selection of the SPPs) or was there a change in your department's / agency's mandate in the last three years?

   Selection and Management of SPPs

   5. Were you involved in the selection of the currently funded SPPs? What were the strengths and weaknesses of the process to select the two currently funded SPPs (EcoBiomics and AMR)? (3.1)

   1. How could the selection process have been improved? (3.1)

   6. What are the strengths and weaknesses in how your SPP is being managed? (3.2)

   1. Please comment on: (3.2)
      1. Oversight
      2. Governance
      3. Task assignment
      4. Monitoring and reporting
      5. Strategies to stay on budget, on time and within scope
   2. How could your project make better use of available resources? (3.3)

   7. Is your project enhancing interdepartmental collaborations? Please explain. (3.4)

   1. Is your project providing innovative solutions to problems that could not have been achieved by one department / agency? Please explain. (3.3)
   2. To the best of your knowledge, what are the key factors that have facilitated or hindered the partners' abilities to collaborate on your project? (3.4)

   Effectiveness – Research results to date

   8. Please explain to what extent the project is contributing (or has contributed) to any of the following potential outcomes. Please quantify estimates to the extent possible. (2.3)

   • Improved processes
   • Improved diagnostics/instrumentation
   • Improved products
   • New capabilities
   • Scientific achievements
   • Reduced harm (health), improved care/health, lives saved
   • Reduced health care costs
   • Other cost savings (please provide range of estimates)
   • Enhanced sustainability and management of resources
   • Reduced environmental harm
   • Improved/informed policies, regulations, legislation, programs
   • Other

   Please provide any documents that you would be willing to share with us about the potential outcomes.

   9. Thinking of your SPP, what are the potential long-term impacts of this project? (2.3)

   1. At your department / agency?
   2. At other federal departments / agencies?
   3. At other organizations outside the Government of Canada?

   [Probe:
   AMR LT impacts: Informed public health and food production decisions to address one of the most serious global health threats facing the world today; reduced antimicrobial resistance in food production systems
   EcoBiomics LT impacts: A comprehensive perspective of water and soil as living systems that sustain ecosystem services and economic activities; environmental management and resource development decisions informed by monitoring of soil and freshwater quality (with the project's objective being to: characterize the complex microbial and invertebrate biodiversity of soil and freshwater through novel metagenomics approaches)]

   10. Please describe whether/how the research results have been disseminated, transferred to end-users inside and out of the federal government, and protected (i.e., IP) to date. (2.5)

   1. How effective were the approaches to transfer the new knowledge and technology? (2.5)

   [Probe: including open data, level of accessibility and transparency of the information provided to end users]

   11. What are some of the factors that positively or negatively influence knowledge / technology transfer to end-users? (2.5)

   12. In your view, to what extent are the knowledge and/or technology developed by the project likely to meet the needs of users, including: (2.6)

   1. Your department / agency?
   2. At other federal departments/ agencies?
   3. At other organizations outside the Government of Canada?

   13. How likely is it that the level of impacts achieved to date through the SPP projects would have been achieved in the absence of GRDI SPP funding? (2.7)

   Lessons Learned

   14. From your perspective, what are the main lessons learned and best practices stemming from your project specifically or more generally from the four SPPs conducted to date? (4.1)

   Additional comments

   15. Do you have any additional comments for this evaluation?

   Thank you.

    
   Top of page

   Horizontal Evaluation of the Genomics R&D Initiative Shared Priority Projects (SPPs)

   Interview Guide – Project Leads and Researchers – Completed SPPs

   Context

   On the behalf of the National Research Council of Canada (NRC), Goss Gilroy is conducting a horizontal evaluation of the Genomics Research & Development Initiative (GRDI) Shared Priority Projects (SPPs). The main goal of the evaluation is to assess the relevance and performance of the SPPs with regard to their expected outcomes. The scope includes both the completed SPPs, including Food and Water Safety (FWS) and Quarantine and Invasive Species (QIS), as well as the two current SPPs, including Antimicrobial Resistance (AMR) and Metagenomic-based Ecosystem Biomonitoring (EcoBiomics). The expectation of the evaluation is to provide evidence-based information to the GRDI ADM-CC.

   The evaluation methods include interviews, document and literature review and a review of administrative data. Your responses will be reported in aggregate only, and you will not be personally identified in any reports.

   The interview will take approximately 60 minutes.

   Thank you for your collaboration.

   According to our records, you were involved in the [NAME OF PROJECT] SPP. Is this correct? If not, how have you been involved in the GRDI/SPPs?

   Background

   1. Could you please describe your involvement in this project?

   Management of SPPs

   2. Was the project delivered on time, within scope and on budget? (3.2)^{Footnote 16}

   3. What are the strengths and weaknesses in how your SPP was managed? (3.2)

   1. Please comment on: (3.2)
      1. Oversight
      2. Governance
      3. Task assignment
      4. Monitoring and reporting
      5. Strategies to stay on budget, on time and within scope
   2. How could your project have made better use of available resources? (3.3)

   4. Did the project enhance interdepartmental collaborations? Please explain. (3.4)

   1. Did the project provide innovative solutions to problems that could not have been achieved by one department / agency? (3.3)
   2. To the best of your knowledge, what were the key factors that facilitated or hindered the partners' abilities to collaborate on your project? (3.4)

   Effectiveness – Research results to date

   5. Thinking of your SPP, what are the potential long-term impacts of this project? (2.3)

   1. At your department / agency?
   2. At other federal departments / agencies?
   3. At other organizations outside the Government of Canada?

   [Probe:
   QIS LT impacts: Canadian biodiversity and trade protected from the impacts of global change through improved ability to monitor invasive alien and quarantine species
   FWS LT impacts: Improved food safety and security in Canada]

   6. Please describe whether/how the research results were disseminated, transferred to end-users inside and out of the federal government, and protected (i.e., IP). (2.5)

   1. How effective were the approaches to transfer the new knowledge and technology? (2.5)

   [Probe: including open data, level of accessibility and transparency of the information provided to end users]

   7. What are some of the factors that positively or negatively influence knowledge / technology transfer to end-users? (2.5)

   8. In your view, to what extent are the knowledge and/or technology developed by the project meeting or likely to meet the needs of users, including: (2.6)

   1. Your department / agency?
   2. At other federal departments/ agencies?
   3. At other organizations outside the Government of Canada?

   9. How likely is it that the level of impacts achieved to date through the project would have been achieved in the absence of GRDI SPP funding? (2.7)

   Lessons Learned

   10. From your perspective, what are the main lessons learned and best practices stemming from your project specifically or more generally from the four SPPs conducted to date? (4.1)

   Additional comments

   11. Do you have any additional comments for this evaluation?

   Thank you.

    
   Top of page

   Horizontal Evaluation of the Genomics R&D Initiative Shared Priority Projects (SPPs)

   Interview Guide – Project Leads and Researchers – Completed and Current SPPs

   Context

   On the behalf of the National Research Council of Canada (NRC), Goss Gilroy is conducting a horizontal evaluation of the Genomics Research & Development Initiative (GRDI) Shared Priority Projects (SPPs). The main goal of the evaluation is to assess the relevance and performance of the SPPs with regard to their expected outcomes. The scope includes both the completed SPPs, including Food and Water Safety (FWS) and Quarantine and Invasive Species (QIS), as well as the two current SPPs, including Antimicrobial Resistance (AMR) and Metagenomic-based Ecosystem Biomonitoring (EcoBiomics). The expectation of the evaluation is to provide evidence-based information to the GRDI ADM-CC.

   The evaluation methods include interviews, document and literature review and a review of administrative data. Your responses will be reported in aggregate only, and you will not be personally identified in any reports.

   The interview will take approximately 60 minutes.

   Thank you for your collaboration.

   According to our records, you were involved in two SPPs, including the completed [NAME OF PROJECT] project and the current [NAME OF PROJECT] project. Is this correct? If not, how have you been involved in the GRDI/SPPs?

   Background

   1. Could you please describe your involvement in these projects?
   2. Thinking of the current project, could you please explain what need is driving this research? In other words, why is this research being conducted?
   3. To date, is the current SPP on time, within scope and on budget?

   Relevance

   4. To what extent is the current project aligned with your department's / agency's current mandate? (1.2)^{Footnote 17}
   1. In what respect(s), if any, is there a lack of alignment?
   2. Did this lack of alignment exist since the beginning of the project (i.e., inherent to the selection of the SPPs) or was there a change in your department's / agency's mandate in the last three years?

   Selection and Management of SPPs

   5. What were the strengths and weaknesses of the process to select the two currently funded SPPs (EcoBiomics and AMR)? (3.1)

   1. How could the selection process have been improved? (3.1)

   6. What are the strengths and weaknesses in how your current SPP is being managed? (3.2)

   1. Please comment on: (3.2)
      1. Oversight
      2. Governance
      3. Task assignment
      4. Monitoring and reporting
      5. Strategies to stay on budget, on time and within scope
   2. How could your project have made better use of available resources? (3.3)

   7. Thinking of the completed SPP, what were the strengths and weaknesses of how that project was managed? (3.2)

   8. Is the current project enhancing interdepartmental collaborations? Please explain. (3.4)

   1. Is your project providing innovative solutions to problems that could not have been achieved by one department / agency? (3.3)
   2. To the best of your knowledge, what are the key factors that have facilitated or hindered the partners' abilities to collaborate on your project? (3.4)

   9. Thinking now the completed SPPs, please comment on the degree of interdepartmental collaboration and the factors that facilitated or hindered the partners' abilities to collaborate on that project. (3.4)

   1. Did that project provide innovative solutions to problems that could not have been achieved by one department / agency? (3.3)

   Effectiveness – Research results to date

   10. Thinking of your current SPP, please explain to what extent the project contributed to any of the following potential outcomes. Please quantify estimates to the extent possible. (2.3)

   • Improved processes
   • Improved diagnostics/instrumentation
   • Improved products
   • New capabilities
   • Scientific achievements
   • Reduced harm (health), improved care/health, lives saved
   • Reduced health care costs
   • Other cost savings (please provide range of estimates)
   • Enhanced sustainability and management of resources
   • Reduced environmental harm
   • Improved/informed policies, regulations, legislation, programs
   • Other

   Please provide any documents that you would be willing to share with us about the potential outcomes.

   11. Thinking of the two SPPs, what are the potential long-term impacts of each project? (2.3)

   1. At your department / agency?
   2. At other federal departments / agencies?
   3. At other organizations outside the Government of Canada?

   [Probe:
   QIS LT impacts: Canadian biodiversity and trade protected from the impacts of global change through improved ability to monitor invasive alien and quarantine species
   FWS LT impacts: Improved food safety and security in Canada
   AMR LT impacts: Informed public health and food production decisions to address one of the most serious global health threats facing the world today; reduced antimicrobial resistance in food production systems
   EcoBiomics LT impacts: A comprehensive perspective of water and soil as living systems that sustain ecosystem services and economic activities; environmental management and resource development decisions informed by monitoring of soil and freshwater quality (with the project's objective being to: characterize the complex microbial and invertebrate biodiversity of soil and freshwater through novel metagenomics approaches)]

   12. For both projects, please describe whether/how the research results have been disseminated, transferred to end-users inside and out of the federal government, and protected (i.e., IP) to date. (2.5)

   1. How effective were the approaches to transfer the new knowledge and technology? (2.5)

   [Probe: including open data, level of accessibility and transparency of the information provided to end users]

   13. What are some of the factors that positively or negatively influence knowledge / technology transfer to end-users? (2.5)

   14. In your view, to what extent are the knowledge and/or technology developed by the two projects likely to meet the needs of users, including: (2.6)

   1. Your department / agency?
   2. At other federal departments/ agencies?
   3. At other organizations outside the Government of Canada?

   15. How likely is it that the level of impacts achieved to date through both projects would have been achieved in the absence of GRDI SPP funding? (2.7)

   Lessons Learned

   16. From your perspective, what are the main lessons learned and best practices stemming from your project specifically or more generally from the four SPPs conducted to date? (4.1)

   Additional comments

   17. Do you have any additional comments for this evaluation?

   Thank you.

    
   Top of page

   Horizontal Evaluation of the Genomics R&D Initiative Shared Priority Projects (SPPs)

   Interview Guide – External Stakeholders

   Context

   On the behalf of the National Research Council of Canada (NRC), Goss Gilroy is conducting a horizontal evaluation of the Genomics Research & Development Initiative (GRDI) Shared Priority Projects (SPPs). The main goal of the evaluation is to assess the relevance and performance of the SPPs with regard to their expected outcomes. The scope includes both the completed SPPs, including Food and Water Safety (FWS) and Quarantine and Invasive Species (QIS), as well as the two current SPPs, including Antimicrobial Resistance (AMR) and Metagenomic-based Ecosystem Biomonitoring (EcoBiomics). The expectation of the evaluation is to provide evidence-based information to the GRDI ADM-CC.

   The evaluation methods include interviews, document review and a review of data. Your responses will be reported in aggregate only, and you will not be personally identified in any reports.

   The interview will take approximately 60 minutes. Thank you for your collaboration.

   Background

   1. Please briefly describe your familiarity and involvement with the GRDI over time.

   Relevance

   2. To what extent are the current SPPs aligned with the priorities of the federal government? (1.1)^{Footnote 18}

   1. In what respect(s), if any, is there a lack of alignment?
   2. Did this lack of alignment exist since the beginning of the SPPs (i.e., inherent to the selection of the SPPs) or was there a change in the government's mandate/priorities in the last three years?
   3. Is there overlap with Genome Canada-funded projects? Please explain the nature of the overlap.

   3. Thinking of the process used to select the current SPPs, in your view, did that process sufficiently allow for alignment with the government's mandate and priorities? Why/why not? (1.1)

   Selection and Management of SPPs

   4. What were the strengths and weaknesses of the process to develop the current SPP themes (EcoBiomics and AMR) and the subsequent selection of the projects? (3.1)

   1. How could the selection process have been improved? (3.1)
   2. What aspects of the development/selection process should be maintained for the next round? (3.1)

   5. Are there features of Genome Canada's project selection process that you think could and should be adopted by GRDI? Please explain.

   1. Are there other similar programs from which best practices around development of SPP themes and selection of projects can be gleaned? What are these programs?

   6. To your knowledge, to what extent do GRDI SPPs enhance interdepartmental collaborations? Please explain. (3.4)

   1. Are the two currently funded SPPs providing innovative solutions to problems that could not have been achieved by one department / agency? (3.3)

   7. To the best of your knowledge, what are the key factors that facilitate or hinder partners' abilities to collaborate as part of an interdepartmental project? (3.3)

   Effectiveness

   8. In your view, to what extent has the knowledge and/or technology developed by the funded SPPs met (or are likely to meet) the needs of users, including: (2.6)

   1. Within the federal government?
   2. At other organizations outside the Government of Canada?

   9. For technologies that have been transferred and/or commercialized, what has been the impact? If technologies have not yet been transferred, what are the potential impacts?

   [Probe:
   QIS LT impacts: Canadian biodiversity and trade protected from the impacts of global change through improved ability to monitor invasive alien and quarantine species
   FWS LT impacts: Improved food safety and security in Canada
   AMR LT impacts: Informed public health and food production decisions to address one of the most serious global health threats facing the world today; reduced antimicrobial resistance in food production systems
   EcoBiomics LT impacts: A comprehensive perspective of water and soil as living systems that sustain ecosystem services and economic activities; environmental management and resource development decisions informed by monitoring of soil and freshwater quality (with the project's objective being to: characterize the complex microbial and invertebrate biodiversity of soil and freshwater through novel metagenomics approaches)]

   10. What are some of the factors that positively or negatively influence knowledge / technology transfer to end-users? (2.5)

   11. How likely is it that the level of impacts achieved to date through the SPP projects would have been achieved in the absence of GRDI SPP funding? (2.7)

   Lessons Learned

   12. From your perspective, what are the main lessons learned and best practices stemming from the four SPPs conducted to date? (4.1)

   Additional comments

   13. Do you have any additional comments for this evaluation?

   Thank you.

Appendix C – SPP-Specific Write-ups

C1: Food and Water Safety (FWS)

1.0 Project Objectives and Activities

1.1 Background and Needs

When the FWS SPP was developed, PHAC estimated that about 4 million Canadians suffer from a food-related illness each year. Many are associated with the presence of bacteria in foods, including Campylobacter, Clostridium Perfringens, Listeriosis, Salmonella, E. coli, and Shigella. E. coli and Salmonella, in particular, affect many Canadians.

PHAC, CFIA and their partners play a key role in monitoring potential outbreaks. When cases of infections/symptoms are reported, food samples are brought to labs for tests. Traditional lab tests could take up to five days to produce findings, resulting in a loss of valuable time as contaminated products may continue to be distributed and consumed by the public. The GRDI-funded SPP was conducted to develop techniques in order to reduce lab time required to identify the presence of bacteria by using genomic identification techniques and new microchip technology.

1.2 Research Purpose, Objectives and Activities

The FWS project is a collaborative effort by six departments and agencies^{Footnote 19} to develop the tools and infrastructure needed to apply genomics-based methods for pathogen isolation, detection, characterization, and source attribution. It was coordinated by Sabah Bidawid of Health Canada. The budget totaled $7,567,727 over the span of 2011-2016.

The goals of the FWS were to reduce costs and processing times of genomics-based methods for pathogen isolation, detection and characterization and develop a federally integrated system to manage, store and provide open access to genomic data and related information from food and water-borne pathogens.

Activities within the project were organized under three major themes, (1) Isolation and Detection, (2) Information Generation and (3) Bioinformatics.

Isolation and Detection had three overarching objectives (HC, AAFC, CFIA, NRC, EC, PHAC):

• Reduce pathogen detection turnaround times from 5 days to 8 hours.
• Specific capture of a single, viable, verotoxigenic E. coli (VTEC) cell in an analytical unit of food or water.
• To develop portable and customizable technologies for rapid, field-deployable detection and molecular characterization of food and waterborne pathogens (microchip technology).

Information Generation was to apply genomics in pursuit of four overarching objectives (PHAC, CFIA, NRC, AAFC, EC):

• To enhance the knowledge base resulting from molecular characterizations of retrospectively and prospectively collected VTEC and Salmonella enterica (SE) strains from a Canadian context.
• To obtain traditional and genomic bacterial characterizations to better understand sources, distribution, and public health significance of VTEC and SE in Canada.
• To apply genomics to rapidly identify and differentiate VTEC and SE strains.
• To develop, validate and transfer new and enhanced genomics-based tools based on discriminatory markers for source attribution, source tracking, and risk assessment of water and food with respect to VTEC and SE quality.

Bioinformatics addressed three overarching objectives (PHAC, CFIA, HC, AAFC):

• To develop the computing and informatics infrastructure that will be required for storing, managing, and analyzing next-generation data for enhanced epidemiological analysis.
• To develop analytical tools for extracting the necessary information for integration within enhanced methods for identification, differentiation and characterization of VTEC and SE.
• To develop training programs to enhance Bioinformatics knowledge and skills across the participating departments and agencies.

1.3 Alignment with Federal Government Mandates and Priorities

According to the 2013 Mid-Year Performance Report for SPPs (1 April 2013 to 30 September 2013, p.60), the FWS project was aligned with existing and emerging coordinated priorities of the departments and agencies involved in pursuit of its goals.

Section 1.2 of the PHAC PAA (2015) covered Health Promotion and Disease Prevention. Under this section of the PAA (Program Activity Architecture), there was Sub-Sub-Program 1.2.1.3: Food-borne, Environmental and Zoonotic Infectious Diseases, with which the FWS was aligned.

At the time FWS SPP was launched, other government institutions were also mandated to protect Canadians from preventable health risks. The Canadian Food Inspection Agency's (CFIA's) goal was to partner with research organizations to improve food safety and disease control systems. Health Canada (HC) also had a role in supporting food safety surveillance. In the Sub-Program 2.2.1: Food Safety of the PAA, it stated that the HC Food and Drug Regulations provide the regulatory framework to develop, maintain, and implement the Food Safety program. FWS was also aligned with research mandates of the other departments involved (AAFC, NRC and ECCC). For example, AAFC's PAA (2015) included the assurance systems area, which covered issues of food safety and the NRC's PAA included an obligation to develop and advance technologies to enhance the prosperity of Canadian industries in support of federal priorities.

1.4 Partners, Collaborators and Stakeholders

The FWS SPP was a collaborative effort by six federal departments and agencies (AAFC, CFIA, ECCC, HC, NRC and PHAC). The project involved over 50 researchers from these organizations.

1.5 Financial Profile

The financial profile for GRDI and non-GRDI funds (leveraging data) in support of the FWS project is presented in Tables C1.1 and C1.2 below. Over the five-year period, the total investment in support of the FWS project was about $22.5 million. GRDI funding accounted for approximately 34% of the total project cost and non-GRDI funds from departmental budget (A-base) and other sources represented 66% of total project funding.^{Footnote 20}

2.0 Achievements

2.1 New Scientific Achievements

The following points summarize the progress achieved in the various components of the project.

Isolation-Detection project. Two milestones were achieved. The work that focused on the initial sample preparation and doing a partial or complete purification to isolate the target pathogens from the particulate food matrix which can interfere with downstream applications were completed. Second, progress has been made with the re-design of the Articulated Blade centrifugal platform hardware and m-CHAS microfluidic cartridge. The new design allows fabrication by injection molding using industrial methods.

Microfluidics assay platform. The CFIA partnered with NRC and HC to develop a new automated test for the rapid identification of E. coli O157:H7 colonies recovered from food samples. This new approach adapts a simple Cloth-based Hybridization Array System (CHAS) developed at CFIA for the identification of E. coli O157:H7 through the detection of genomic markers using the PCR (Polymerase Chain Reaction) technique, to a novel microfluidic platform developed at NRC which fully automates all of the reagent addition steps. This new microfluidic CHAS format obviates the labour-intensive operations required to complete a CHAS procedure and significantly increases analyst productivity, ensuring that laboratory results are available in a timely fashion to inform risk management decisions during food safety investigations.

Use of microchips to identify bacteria. The first years of the FWS project saw the NRC team refine its silicon photonic wire sensor array platform. Microarray chips containing up to 128 sensors were built. Multiplexed detection and identification of pathogenic E. coli in suspension was demonstrated using monoclonal antibodies (MAbs) developed at the CFIA labs. The research teams established and demonstrated a test protocol for multiplexed serotyping of E. coli in food samples prepared by HC labs.^{Footnote 21}

Information Generation theme. Sequence polishing experiments continued in FY2015-16 to generate comprehensive, fully closed genome resources with consistent and curated gene product naming for future use (i.e., enhanced genome qualities relative to what are otherwise publically available). Working with the Bioinformatics theme, all FWS VTEC genomes were being leveraged to develop and validate an E. coli in silico typing tool (ECISTR) and other novel bioinformatics tools. The Information Generation theme, in partnership with the Bioinformatics theme generated the most comprehensive cataloguing and molecular characterization of food and waterborne Shiga toxin-producing E. coli (STEC) and SE in Canada to date and understanding of what VTEC and SE pathogens exist within Canada's borders, where, when and why.

Bioinformatics theme. A pipeline (SigSeekr) was developed to identify signature genomic sequences for a given bacterial strain (e.g., a VTEC strain associated with a foodborne illness outbreak), enabling the rapid development of strain-specific PCR primers which can be deployed to frontline testing laboratories during food safety investigations. The project developed a strain-specific PCR method targeting a common E. coli O157:H7 lab control strain. A new system has been developed to enhance Canada's ability to quickly identify genomic fingerprints for diagnostic and subtyping purposes by identifying target signatures in arbitrary pathogen genomes or groups of genomes based on the Integrated Rapid Infectious Disease Analysis (IRIDA) platform. These systems were implemented in provincial laboratories.

2.2 Development of Evidence-based Public Policy and New Knowledge/ Technologies

The FWS had several new implications in this area:

• Genomic approach to assess effects of food chemicals on the development of allergies: HC researchers aimed to develop genomics-based tools to assess chemicals for their potential to develop and enhance allergy and contribute to increased allergic diseases in Canada. Research conducted in 2015-2016 has yielded data of interest to international toxicologists, regulators at HC as well as those under the Government of Canada's Chemicals Management Plan involved in the evaluation of novel food chemical contaminants and additives, including colouring agents and nanomaterials.
• Safety of prebiotics in infants: In this project activity, HC researchers assessed the impact of fructooligosaccharides on infant gut bacterial community during weaning and over the long term in rats to develop genomics-based methods for assessing bacterial community composition associated with feeding fermentable materials in infant formula. This project activity has already increased awareness among HC regulators of the physiological outcomes potentially associated with feeding fermentable materials, especially as they apply to infant formula.
• Identification and characterization of microRNA in serum and milk to measure the health effects of fungal toxins and chemical contaminants in food: HC researchers set out to identify and characterize microRNA in serum and milk associated with dietary exposure to fungal toxins and chemical contaminants currently detected in foods. This work will enable the generation of important regulatory toxicology data to inform the risk assessment process, enhancing HC's ability to detect and respond to the presence of fungal toxins and chemical contaminants in food consumed by Canadians.
• Genomics approach to predict pulmonary pathology induced by nanomaterials: Nanomaterials can induce harmful effects in animals. Based on the results, an Adverse Outcome Pathway detailing the toxicity pathways for lung fibrosis induced by nanomaterials was developed and submitted to the Organization for Economic Co-Operation and Development to support risk assessment activities.
• Genomics approach to understand the Respiratory Syncytial Virus vaccine: Respiratory Syncytial Virus is a common and highly contagious virus that infects the respiratory tract of infants and young children, and is the most common cause of bronchitis. This project activity aimed to better understand vaccine-induced toxicity and establish regulatory tools to assess the safety of this vaccine. Preliminary research data has been communicated to HC regulators responsible for evaluating biological drugs in Canada.
• Next generation sequencing detection of de novo mutations to identify germ cell hazards: De novo mutations are associated with a diverse array of genetic phenotypes and are increasingly being recognized as contributing to a wide range of human diseases. A reference genome Muta ^TM Mouse was completed using whole genome sequencing and a bioinformatics pipeline has been implemented to establish a mutation spectrum.
• Genomics approach for the standardization and risk assessment analysis of stem cell-based health products: In this project activity, HC researchers developed diagnostic tools to enable a thorough evaluation of the risks and benefits associated with the therapeutic use of human mesenchymal stem cells. Two new biomarkers were identified. These successfully validated biomarkers will form the basis for the development of diagnostic tests for evaluating stem cell-based health products.
• Development of practical toxicogenomics methods for hazard identification and risk assessment of environmental chemicals: In this project activity, HC researchers developed and validated timesaving and more cost-effective risk assessment genomics-based methods to predict whether a chemical causes DNA damage or other adverse genetic effects. Data was provided to the Health and Environmental Sciences Institute for a submission to the United States Food and Drug Administration's biomarker qualification program to validate these genome tools. Significant contributions were made to the United States Environmental Protection Agency's Next Generation Risk Assessment publication on the application of genomics in human health risk assessment.
• Genomics knowledge to strengthen public health programs and activities related to infectious and chronic disease Enteric diseases: Mostly done through PHAC project activities, they added to the development of genomic technology in a laboratory network, detection and genomic epidemiology of priority pathogens, and supporting communities, hospitals and the international response in regards to antimicrobial resistance.

2.3 Knowledge and/or Technologies Transferred to End-users

A total of 51 knowledge transfer workshops with end-users were conducted according to the 2015-2016 performance report.

The effectiveness of the E. coli identification technologies (lab tests) has been demonstrated and the lab tests are in the final validation phases. The 795 Polymerase Chain Reaction has been deployed to the CFIA food microbiology testing laboratory network, which will eventually be utilized by provincial labs as well.

• The microchip development has also been advanced and prototypes have been developed. This technology is used in the final stage of bacterial testing via a portable device. Uptake started with CFIA labs but eventually should make it way to other government labs.
• The Bioinformatics theme was successful in developing two bioinformatics software programs: SigSeeker for the identification of signature sequences in E. coli, and Neptune which is a single algorithm that can identify signature sequences in any organism and can do genotype-phenotype association. One software platform was rolled out to PulseNet Canada for molecular surveillance of foodborne disease and has been taken up by the provinces to integrate their databases and servers into one large network. The other being FoodNet, which is used by PHAC in their surveillance activities.

Figure C1.1 below shows the web of knowledge transfer between stakeholders:

Figure C1.1: Knowledge Transfer Web

Figure C1.1: Knowledge Transfer Web - Text version

Web diagram of knowledge transfer between stakeholders.

Stakeholders include 13 academic institutions, 3 provincial/state government departments, 4 Canadian federal departments and one international scientific organization.

PHAC has knowledge transfer connections with 15 stakeholders:

• University of Alberta
• Dalhousie University
• Simon Fraser University
• University of British Columbia
• University of Maryland, USA
• University of Lisbon, Portugal
• University of Manitoba
• University of Guelph
• University of Lisbon
• BC Public Health
• CDC Atlanta, USA
• CFIA
• AAFC
• NRC
• HC

A high volume of transfers, ranging between 10-13, occurred with BC Public Health, Simon Fraser University, University of British Colombia, University of Maryland and University of Lisbon.

NRC also has knowledge transfer connections with 11 stakeholders.

• University of Ottawa
• University of Malaga, Spain
• Université of Paris, France
• INRS
• North Carolina Deptarment of Agriculture, USA
• CDC Atlanta, USA
• Consejo Superior de Investigaciones Cientificas
• HC
• AAFC
• PHAC
• CFIA

The volume of transfers between NRC and stakeholders ranges from 6 to 2, with the highest amount of transfers occurring between NRC and AAFC and HC

AAFC has knowledge transfer connections with 9 stakeholders:

• University of Ottawa
• University of Malaga, Spain
• Université de Paris, France
• University of British Columbia
• Consejo Superior de Investigaciones Cientificas
• North Carolina Department of Agriculture, USA
• CDC Atlanta, USA
• NRC
• PHAC

The volume of transfers between AAFC and stakeholders ranges from 6 to 2, with the highest amount of transfers occurring between AAFC and NRC.

CFIA has knowledge transfer connections with 7 stakeholders

• University of British Columbia
• University of Lisbon
• University of Guelph
• BC Public Health
• HC
• PHAC
• NRC

The volume of transfers between CFIA and stakeholders ranges from 4 to 1, with the highest amount of transfers occurring between CFIA and PHAC and NRC

HC has knowledge transfer connections with 8 stakeholders.

• INRS
• University of Lethbridge
• University of Ottawa
• University of Guelph
• University of Manitoba
• CFIA
• NRC
• PHAC

The volume of transfers between HC and stakeholders ranges from 6 to 1, with the highest amount of transfers occurring between HC and NRC.

2.4 Impact of Technologies Transferred

Acute bacterial foodborne and waterborne illness are estimated to cost Canada more than $12 billion annually. The FWS SPP was conducted to improve food surveillance systems with the goal of quicker and more accurate detection of foodborne and waterborne pathogens. The program had two significant impacts.

1. It developed an integrated federal system (PulseNet Canada) to manage, store and provide open access to genomic data of verotoxigenic E. coli and Salmonella enteritidis for detection and characterization from a variety of food, water and environmental matrices. This system gives users access to information and data in a timelier manner. This includes information regarding pre-sequenced pathogen strains that are more readily available across regions, which reduces the effort needed for surveillance and allows for quicker reactions to detection to protect public health. It also allows for the continued collaboration and sharing of best practices across Canada. For example, information available through PulseNet regarding a discovered fungal disease attacking crops was used to identify the same disease found to be present in fish. This information has since been used to take measures to protect aquatic life. This may not have occurred without the central database.
2. It developed methods that reduce isolation-detection turnaround times from more than five days to less than eight hours and improve the identification of pathogen sources. Standardized genomic sequencing for known pathogen strains allows for investigators to more quickly and accurately identify pathogens, reducing the amount of time needed to confirm pathogen presence. The portability of these new technologies has made it much easier to test food and water either at the source or the point of entry into the country, which has and will continue to improve the response time and accuracy of response to foodborne and waterborne pathogens.

The evidence shows that these partial benefits alone (over 10 years) were expected to exceed GRDI investments for the FWS project, in the amount of $17M during the course of Phase V of GRDI (see table C1.3).

Unfortunately, it is difficult to assess the actual impact of the FWS SSP as an economic analysis has not been conducted. Furthermore, there seems to be little information beyond the 2015-2016 performance report on actual uptake of the technologies developed during the project.

3.0 Interdepartmental Research Collaboration

3.1 Effectiveness, Efficiency, and Extent of Interdepartmental Collaboration

Interview respondents agreed that the collaboration between participating departments and agencies was useful and important for project success and that GRDI facilitated active collaborations, such as the exchange of samples, protocols, information, actual collaborative work, and working meetings on specific tasks. Prior to GRDI, according to interview respondents, there was distance and boundaries between departments and agencies that GRDI helped to alleviate through the FWS project. GRDI provided the financial support and project framework (project charter) to allow this collaboration.

The GRDI first round of SPPs were initiated in Phase V of GRDI to respond to a need for interdepartmental collaboration. The main advantage of the collaboration is that it leveraged expertise from various areas. The FWS project was multidisciplinary, involving genomics research and hardware/software development, which was facilitated by the involvement of various departments, agencies and research groups.

The bibliometric analysis conducted for the 2019-20 evaluation illustrates how interdepartmental collaborations occurred as a result of the project. According to findings:

• Three federal organizations are on the top 20 collaborators graph: PHAC, HC and NRC.
• There is a strong hub of Canadian institutions involved in GRDI-FWS related project activities.
• Top institutions by number of co-authorship publications are also the top institutions by number of collaborators.

The report also provides the following findings:

• 42% of publications show collaborations with at least one other institution.
• 32.5% are single-authored, and are most frequently associated with conference participation.
• 57.9% are from a single-institution; this is the highest single-author rate for all four priority datasets.
• 29.8% of publications have 3 or more institutional affiliations, and are associated with both journal articles and conference participation.
• 33.3% of publications feature collaborations with Canadian, non-federal organizations, usually academic.
• 20.2% of FWS collaborations feature international partners.

It was also agreed by interview respondents that without the collaborative framework provided by GRDI, the FWS project would not have realized its successes.

3.2 Lessons Learned and Best Practices Related to Interdepartmental Collaboration

Best Practices

The following are taken from the Innovation Management Document developed in the Phase V SPPs.

• Collaboratively develop a potential end-user list/database, which would initially serve the purpose of identifying organizations/individuals from which input would be solicited regarding project design.
• To facilitate effective project development and execution, at a November 2014 Workshop GRDI participants agreed it would be helpful to have a group of "subject matter experts" who could be called upon, as required, to support GRDI SPPs. This community of Subject Matter Experts would be guided by the Innovation Management Guiding Principles.
• Specific to the FWS, the IRIDA team created a novel macro-enabled spreadsheet tool for manually capturing and curating metadata in the form of a 'Metadata Manager', with pull-down menus for consistent data entry. This tool was used to generate the public archive submission data; and has been modified for other projects such as the GRDI-AMR. The IRIDA team also partnered with the National Center for Biotechnology Information (NCBI) to streamline their batch data upload procedures which lead to an easier process that also helps promote open access to all data for end-users.
• The FWS also shed light on the importance in standardization of annotations of data.

Challenges and Solutions

• More up-front work to determine roles and responsibilities could have saved work downstream.
• More communication with external stakeholders via teleconferencing. It was suggested that regular calls every six months be scheduled to share ideas and insights. Some external stakeholders tended to "hoard" information, which stems from traditional tendencies in interdepartmental projects.
• Increase the number of external end-users in earlier phases of the project to increase knowledge transfer and provide feedback from a larger variety of perspectives.

C2: Quarantine and Invasive Species

1.0 Project Objectives and Activities

1.1 Background and Needs

According to the World Conservation Union, invasive species are the second biggest threat to biodiversity after habitat loss. The impact of Invasive alien species (IAS) on native ecosystems, habitats and species is severe and often irreversible, and can cost billions of dollars each year. The estimated damage worldwide from invasive species is more than $1.4 trillion (5% of the global economy).^{Footnote 24 Footnote 25} Studies that included over 100 species suggested that the average costs associated with each invasive species within North America ranged from $14 million to $39 million annually.^{Footnote 26} A 2006 study of the projected economic impact of the nuisance related to nonindigenous species on Canadian fisheries, agriculture and forestry sectors (which examined only 16 nuisance species) estimated annual costs at between $13.3 and $34.5 billion.^{Footnote 27} Similarly, the CFIA estimated the annual impact of invasive species to be $30 billion, $20 billion in the forest sector, $7 billion in the Great Lakes, and $2.2 billion in the agricultural sector.^{Footnote 28} Furthermore, globalization, climate change, and international trade increases have elevated IAS introduction risks.

Rapid detection and preventive actions at borders are less costly than control, management and eradication of the invasive species, losses in biodiversity and in intrinsic ecosystem values and economic costs associated with activities that are dependent on terrestrial and aquatic ecosystems. Research and monitoring, exchange of information, border control through inspections of international shipments, customs checks and quarantine measures and cooperation are some of the key guiding principles of the United Nations Convention on Biological Diversity (COB) for the mitigation of the impact of alien species that threaten ecosystems, habitats or species.

The SPP entitled "Protection of Canadian biodiversity and trade from the impacts of global change through improved ability to monitor invasive alien and quarantine species" (hereafter referred to as QIS) was a collaborative research effort that followed these guiding principles of targeting the development of genomic knowledge and tools to help address the increasing number and severity of quarantine and invasive species entering Canada.

1.2 Research Purpose, Objectives and Activities

The QIS SPP applied genomics to increase the speed, sensitivity and specificity of detection, thereby enhancing the prevention/eradication efforts in order to minimize the environmental and economic losses caused by the introduction and spread of invasive species. The QIS SPP intended to enhance the capacity of regulatory departments and agencies (DAs) in order to strengthen Canada's role and capacity in protecting its resources and expanding market access for Canadian products by having the most modern tools to detect quarantine and invasive species. The SPP was divided into five project activities (referred to as sub-projects). Sub-project lead(s) are highlighted in bold.

Sub-Project 1: Optimization and standardization of nucleic acid extractions (led by CFIA, all DAs)
Developing, optimizing and standardizing methods that would allow for efficient extraction of DNA for: 1) Preserved and archived tissues originating from the various federal collections; and 2) Bulk samples collected in the field for use in sensitive direct detection.

Sub-Project 2: Barcoding of aquatic invasive species of highest risk to Canadian native fauna and trade (led by DFO, with ECCC)
Developing DNA reference sequences to facilitate detection of high risk aquatic invasive species that can cause serious harm to commercial and recreational fisheries, threaten aquaculture productivity, impact habitat quality and ecosystem productivity, displace native species, and alter food web dynamics. This sub-project comprised three themes: 1) Live food and aquarium trade, 2) Aquatic invertebrates in shipping pathways, and 3) Aquatic invasive parasites.

Sub-Project 3: Barcoding of Quarantine and Invasive Species in Terrestrial Ecosystems (led by AAFC, with CFIA, and NRCan)
Generating DNA barcode libraries that would provide the baseline identifiers for confirmation of identities and also provide reliable taxonomic digital libraries that are central for the development of the next generation of early detection technologies. This sub-project comprised five themes: 1) Insects and mites, 2) Nematodes, 3) Fungi, 4) Viruses and Phytoplasma, and 5) Invasive plants.

Sub-Project 4: Direct detection of quarantine and invasive species (led by CFIA, with AAFC, DFO, ECCC, NRC, and NRCan)
Evaluating the application of Next Generation Sequencing (NGS) methods for the detection of diverse groups of invasive species from bulk environmental samples. This sub-project comprised four themes: 1) Forest and agricultural insect pests, 2) Viruses infecting tree-fruit and grapevines, 3) invasive toxic cyanobacteria, and 4) Invasive species pathway analysis.

Sub-Project 5: Bioinformatics (led by AAFC, all DAs)
Creating a cyber-infrastructure platform and workflows accessible to all participating DAs to store information related to environmental samples and individual specimens, manage and to analyze data generated by the other four QIS sub-projects.

1.3 Alignment with Federal Government Mandates and Priorities

As a party to the United Nations COB, the Government of Canada is committed to the prevention, control or eradication of invasive species that threaten ecosystems, habitat, and indigenous species.^{Footnote 29} The federal, provincial and territorial governments produced An Invasive Alien Species Strategy for Canada (IAS Strategy) in 2004.^{Footnote 30} This strategy is aimed at reducing the risk of invasive species to the environment, economy and society, and promoting environmental values such as biodiversity and sustainability.

Federal science-based DAs support the IAS Strategy and respond to invasive terrestrial and aquatic invasive species with a number of programs and horizontal initiatives that also contribute to federal strategic outcomes.^{Footnote 31} With shared responsibility toward prevention and management of IAS, ECCC is leading the invasive animal species (Sub-Program Wildlife Habitat Conservation and the Federal Sustainable Development Strategy), DFO the aquatic invasive species issues (Aquatic Invasive Species Regulations), CFIA the invasive plants and other plant pests (Action Plan for Invasive Alien Terrestrial Plants and Plant Pests); and NRCan the forest pests (Sub-Program Forest Disturbances Science and Application).^{Footnote 32} QIS genomic knowledge and tools were envisioned to support the following key federal responsibilities in:

• Resolving or preventing trade dispute in cases for which trading partners are claiming that Canadian products contain restricted biological organisms by confirming their absence;
• Maintaining or expanding market access opportunities for products by confirming the absence of biological organisms (or species) restricted by trading partners;
• Reducing the quarantine period for imported commercial biological material (e.g., imported plants for agricultural production) by detecting the presence or absence of restricted biological organisms or species;
• Protecting agri-food, fishery and forestry resources and associated domestic and international markets from alien species; and
• Preventing biodiversity loss caused by alien species.

Finally, the provision and use of genomic-based information by the QIS project to support policy development and implementation is also consistent with the current 2014 federal Science, Technology and Innovation Strategy.^{Footnote 33} The project contributes to the three pillars for Canada outlined in the Strategy (people, knowledge, innovation) and directly supports the research priority Environment and Agriculture and most of the associated focus areas.

1.4 Partners, Collaborators and Stakeholders

The QIS SPP primarily involved scientists from six federal DAs: AAFC, CFIA, DFO, EC, NRC and NRCan. The project was led by AAFC and composed by five sub-projects and themes composed of theme lead(s) and PIs from participating DAs. The QIS SPP was highly cooperative involving collaborators and end-users that are internal and external to the Canadian federal government. A broad range stakeholders and end-users have been engaged, including representatives of academia, governments and industry. Nearly 140 research collaborators from over 60 organizations have participated during the course of the SPP, including informal and formal collaborations with external collaborators (41% of collaborators were from international organizations).^{Footnote 34}

The SPP also had a research collaboration agreement with researchers from the University of British Columbia (UBC). This partnership generated complementary data for the QIS SPP and for the Tree Aggressors Identification using Genomics Approaches (TIAGA) project on forest health genomics.^{Footnote 35} This large-scale project funded by Genome Canada aims to increase Canada's capacity in forest disease diagnostics and pathogen detection and monitoring by developing and translating genomics resources into applications (e.g., diagnostic tools for on-site use or delivery as part of a high-throughput service offering to end-users).

The Biodiversity Institute of Ontario (BIO) at the University of Guelph was also a major partner for data processing, exchange and research collaboration. On August 12, 2012, a Memorandum of Understanding (MOU) was signed between AAFC and the University of Guelph, through the BIO, aimed at building an accurate reference library of DNA barcodes of insect species held at AAFC's Canadian National Collection of Insects, Arachnids, and Nematodes that will be part of the Barcode of Life Database (BOLD). Their Canadian Centre for DNA Barcoding (CCDB)^{Footnote 36} conducted initial sequencing for the QIS SPP and gave a 50% discount on the cost of sequencing for aquatic specimens. The QIS SPP also formalized an agreement with the Canadian Barcode of Life Network^{Footnote 37} funded by Genome Canada for data exchange.

1.5 Financial Profile

The QIS SPP timeframe was five years from FY 2011-12 to 2015-16, with a total GRDI funding amount of $8M. In addition, the participating DAs contributed to the SPP with in-kind contributions estimated at $14M, which represent 64% of total planned project costs (TableC2.1). AAFC received the lion share of GRDI funding (40%, including 20% for shared bioinformatics) but also contributed the most to the SPP with in-kind contributions (35%).

2.0 Achievements

2.1 Development of Unique Capabilities in the Federal Public Service

Within the federal government, the QIS SPP supported the collaborative work of over 50 research scientists, professionals and technicians, post-doctoral fellow as well as graduate and undergraduate students across six federal DAs.^{Footnote 38} In terms of developing unique capabilities in the federal public service, the project has so far contributed to the:

• Mobilization of federal resources to pursue collaborative research on invasive species;
• Successful implementation, not without technical and administrative challenges, of the first horizontal cyber-infrastructure bioinformatics platform to manage and analyze genomic data generated and accessible to all participating DAs, and;
• Establishment of a national extensive and high-quality genomic database that can be used by all DAs (and now by external organizations) interested in biodiversity and species identification.

2.1 New Scientific Achievements

The following points summarize the scientific achievements of the SPP:^{Footnote 39}

• Optimization and standardization: Nearly 30 Standard Operating Procedures (SOPs) for the extraction of DNA from field and bulk samples and from specimens in reference collections have been developed, optimized and standardized:
  • SOPs for DNA extraction from formalin fixed tissues and from ethanol preserved tissues;
  • SOP for plant DNA extraction from herbarium material, insect, soil and water.
• Barcoding of high risk aquatic invasive species: Delivered 11,800 DNA barcode sequences:
  • Canadian freshwater fish species in Canada have been vouchered, sequenced and databased through QIS activities;
  • With respect to the theme on aquatic invertebrates, a collection of DNA sequences and associated digital information assembled through QIS activities now contains nearly 3,000 specimens and have been added to the AAFC database;
  • Regarding aquatic parasites, barcode sequences were obtained from hundreds of specimens representing 4 target taxa and were used to create a reference DNA database for species identification.
• Barcoding of QIS in Terrestrial Ecosystems: insect, mite, nematode, fungi, plant virus, phytoplasmas and invasive plant samples. Important new samples from Canada and trading partners have been acquired, vouchered, databases, sequenced and deposited in federal natural history collections.
• Direct detection of quarantine and invasive species: QIS advanced experimental and bioinformatics protocols; generated data to assist accurate and rapid identification of target organisms (e.g., insects, plant viruses, cyanobacteria, freshwater fish) in environmental samples. The proof-of-concept for direct detection of forest insect pests and associated reference database has been tested during the course of the QIS SPP. Manuscripts on assay development for DNA extraction from bulk insect samples were published. But according to one respondent, insects and other groups (Nematodes, Fungus, Viruses and Phytoplasmas and Invasive plants) were too diverse so it had been decided to focus on high-risk invasive species aquatic ecosystems and to and high quality data sets were produced. However, the development of a toolkit for rapid detection of regulated pathogens and invasive species was not completed during the five-year period of the project.
• Bioinformatics: Implementation and use of a bioinformatics platform to manage and analyze genomic data generated and accessible to all participating DAs.

Between 2012-13 and 2014-15, more than 16 research tools and processes were produced by the QIS sub-project and transferred to end-users.

Since the end of the SPP, scientists pursued research using QIS data and bioinformatics workflows. Several publications were completed or are still in preparation (according to the latest QIS off boarding report made available for the evaluation – no date). For example, AAFC and NRCan recently published a manuscript on the establishment of foreign species of beetles in Canada based on 8,000 species beetles and dataset of over 5,000 sets of DNA sequences.

2.2 Development of Evidence-based Public Policy and New Knowledge/Technologies

As reported in the previous evaluation and by the GRDI secretariat,^{Footnote 40} the following success stories provide examples of impacts of QIS on evidence-based public policy:

• GRDI data on the Downy Mildew pathogen of soybeans helped to keep CAN$85 million of annual exports in Malaysia. According to some respondents, this was made possible because of effective interdepartmental collaborations in the development of libraries of DNA barcodes, DNA extraction methods, diagnostic tools and communication channels. Notably, the QIS project enabled federal scientists to leverage historic collections of specimens by developing SOPs to extract DNA for the development and use of new identification technologies.
• GRDI fungus data also helped to avoid false positive results related to Canadian commodities with a US Select Agent that could have created potential trade issues and disruption causing a negative significant impact on the Canadian corn exports.
• GRDI nematode data allowed authoritative identification of a previously misidentified nematode in yellow peas for the negotiation of export market access issues with India.
• NGS direct detection methods have been applied to virus infecting tree-fruit and grapevines and the value and potential for this method to reduce costs, time and increase the ability to detect new virus variants has already been demonstrated.
• GRDI QIS aphid database and rapid detection tools are tested to help control the spread of virus spread by aphids colonizing strawberry plants.

The interviews for the current evaluation confirmed that QIS played an important role in the development of science and technology based initiatives in support of public policy. In the 2016 budget, AAFC received $30M from the Treasury Board to digitize its collection. In 2017, $80M was allocated for the construction of a new CFIA research center, the Centre for Plant Health. This laboratory located in Sidney BC is Canada's only post-entry quarantine, research and diagnostic facility for tree fruit, grapevine and small fruit and is responsible for testing plants for viruses. (Centre for Plant Health, Sidney Laboratory). The linkage between QIS and this center was recognized by senior management.

2.3 Knowledge and/or Technologies Transferred to End-users

• While the tracking of the level of use of the SOPs is not available, the SOPs were successfully transferred to all participating DAs and to several external research university groups.
• Guidelines, training and support were provided to end-users and collaborators.
• Web-based SharePoint sites hosting the most current versions of documents and some data for access by all project DA participants and Advisory Boards.
• Agreements to share data stemming from the QIS both within and outside the federal family.
• Deposits of project data in federal genomics databases and public libraries used by DAs and academic and provincial governmental research organizations outside the federal government (see Figure 2).
• Knowledge transfer workshops with stakeholders/end-users were also held.

The SPP produced a large volume of peer-reviewed publications or presentations, which contribute to the dissemination of information to end-users.

The bibliometric analysis identified a total of 220 GRDI publications associated with the QIS SPP. Of 220 EcoBiomics publications used for a collaboration network analysis, about 60% were conference presentations (46%) or conference participations (14%), and the remainder were mostly articles (29%). For 59 publications indexes in Scopus, the average citation per document was 50.8, with a positive high Field-Weighted Citation Impact (FWCI) score of 2.09. QIS publications have 15% of the share of top 10% highly cited publications in the field.

Figure C2.1: Collaborative Network Analysis — QIS SPP (with > 3 co-publications)

Figure C2.1 - Text version

Web diagram of QIS SPP collaborative network analysis.

5 stakeholders include 13 academic institutions, 3 provincial/state government departments, 7 Canadian federal departments and one RTO, CNRS, France.

AAFC has collaborative connections, with 11 stakeholders:

• Auburn University, Australia
• University of Saskatchewan
• Universidad Veracruzana, Mexico
• Harvard University, USA
• University of California, USA
• USDA, USA
• NRCan
• HC
• PHAC
• CGC

The volume of transfers between AAFC and stakeholders ranges from 23 to 2, with the highest amount of transfers occurring between AAFC and CFIA.

CFIA has collaborative connections with 11 stakeholders:

• University of British Columbia
• Université de la Rochelle, France
• University of Guelph
• FP Innovations
• Royal BC Museum
• CNRS, France
• HC
• NRC
• NRCan
• AAFC
• DFO

The volume of transfers between CFIA and stakeholders ranges from 23 to 2, with the highest amount of transfers occurring between CFIA and AAFC.

DFO has collaborative connections with 11 stakeholders:

• University of Guelph
• Saint Mary's University
• Dalhousie University
• University of Windsor
• University of Washington, USA
• University of Toronto
• Université de la Rochelle, France
• Royal BC Museum
• BC Ministry of Environment
• CNRS, France
• CFIA

The volume of transfers between DFO and stakeholders ranges from 22 to 2, with the highest amount of transfers occurring between DFO and CFIA.

BCCC has collaborative connections with 2 stakeholders:

• Concordia University
• Dalhousie University

The volume of transfers between BCCC and stakeholders ranges from 8 to 3 with the highest amount of transfers occurring between BCCC and Concordia University.

HC has collaborative connections with 2 stakeholders:

• CFIA
• AAFC

The volume of transfers between HC and stakeholders ranges from 8 to 7, with the highest amount of transfers occurring between HC and CFIA.

NRC has a collaborative connection with 1 stakeholder:

• CFIA

The volume of transfers between NRC and CFIA is 4.

PHAC has a collaborative connection with 1 stakeholder:

• AAFC

The volume of transfers between PHAC and AAFC is 3.

CGC has a collaborative connection with 1 stakeholder:

• AAFC

The volume of transfers between CGC and AAFC is 6.

Those publications feature 394 scientists from 173 institutions and 37 countries. Over half were published collaboratively between at least two organizations (58%) and more than a third of all publications involved three organizations or more (36%). QIS has the highest collaboration rate among all four SPPs. Academic collaborators from Canada and abroad are numerous (Figure C2.1, above). QIS scientists have also provided scientific advice and played a prominent role in national or international genomics-related committees.

2.4 Impact of Technologies Transferred

According to most respondents and documents, the QIS SPP contributed to position Canada at the world level on the use of genomic tools for regulatory activities. For many, the QIS SPP allowed Canada to stay competitive by having provided capacity and barcoding data for projects using Next Generation Sequencing (NGS) of environmental samples in fields such as metagenomics:

• The SOPs and microfluidic tools prototyped for direct detection developed under QIS (from environmental, field and bulk samples) served as the basis for the EcoBiomics SSP.
• Similarly, the QIS bioinformatics platform and associated lessons learned benefited to the EcoBiomics SSP which is currently developing a platform with improved computing, bioinformatics analytical tools, data storage and sharing capabilities.

The QIS SPP off-boarding and interviews also reported spin-off projects and use of QIS outputs:

• Multiple DNA extraction SOPs were transferred to user groups and follow-up with end users has demonstrated improvement with DNA extraction results and increased diagnostic efficiency.
• Additional specimens were successfully sequenced and made accessible in AAFC's internally developed web application (SeqDB) used for biological collection management and integrated DNA sequence tracking as well as in public repositories (i.e. BOLD, GenBank).
• The QIS protocols are now the standard SOP in AAFC laboratories and incorporated in research programs for all current and future projects with DNA-barcoding and taxonomy aspects.
• References will be used in a new project on direct detection of marine invertebrate invaders in BC. Three new projects funded by DFO and the BC government stemmed from QIS expertise/ capacity developed on aquatic invertebrates.
• ECCC is now incorporating parasites into STAGE pollution project/STAGE funding to discriminate parasite in cumulative stress project.
• Funding was awarded to an AAFC scientist to complete an in-depth systematic study for rust fungus species complexes identified in preliminary analyses of the DNA barcodes from QIS.
• QIS protocols are now used in metagenomic work at CFIA as a reference database/tool to monitor fungi in specific sites of interest.
• Collaboration among teams of researchers in DFO (Moncton, Nanaimo) for new projects.
• Invasive plant barcode collection available to CFIA diagnostic lab and generated two spin-off projects funded by CFIA. QIS SOPs will be used in these projects. It also contributed to characterization of new viruses, contributed to generating new information on virus distribution and new potential pathways.
• NRCan and CFIA are doing analysis of DNA recovery by size of organism undertake to determine if levels can be used to count organisms in mass extractions.

However, the QIS goal of implementing a field diagnostic tool to test for regulated pathogens and invasive species and to monitor new potential organisms has not yet materialized. According to some respondents, the research on viruses by CFIA and AAFC at the Centre for Plant Health will eventually lead to the development of improved diagnostics tools by robotizing the genome sequencing of plants for viruses. This technology could ultimately be transferred to producers and importers to monitor is plants are virus-free over time after importation. Thus, the integration of genomic diagnostic tools into the overall process of regulators and other organizations involved in dealing with industry, market access and trade issues has not yet materialized.

3.0 Interdepartmental Research Collaboration

Respondents unanimously confirmed that the QIS SPP increased their interdepartmental collaborations and in many cases such collaborations would not have been possible in the absence of the SPP. A dedicated budget, site access and sample collection through participating DAs, a common set of SOPs and a willingness to collaborate were all highlighted as keys to successful collaboration. In both evaluations, evidence of increased collaborations were mentioned by respondents from all participating DAs. Also, many highlighted continued collaboration opportunities after the end of the QIS SPP. These views are illustrated in the following quote:

"Yes, it enhanced interdepartmental collaboration. Within EcoBiomics and QIS, SPPs allowed new collaborations, with departments that use the same approaches and technique but that we usually don't interact with because there are not working on the same topics or needs."

As noted by one respondent, the GRDI SSPs are attractive and bring a lot of attention and focus from partners enabling leveraged funds from different sources. In 2016, together with the FWS SPP team, the QIS SSP team received a Public Service Award of Excellence for their scientific contribution and for interdepartmental management.^{Footnote 41} The collaborative network analysis provides an illustration of interdepartmental connections through co-publications. All DAs involved in the project are represented in the co-publication network. As shown in Figure C2.2, AAFC and CFIA dominate the collaboration map for the QIS, with stronger connections between CFIA, NRCan, DFO and EECC. HC, PHAC and ECCC have fewer co-publication linkages to other partners.

Figure C2.2: Collaborative Network Analysis QIS SPP (Federal Departments only)

Figure C2.2 - Text version

Web diagram of IS SPP collaborative network analysis (Federal Departments only)

Federal departments include: AAFC, CFIA, NRCan, HC, PHAC, BCC, DFO, NRC, CGC

AAFC and CFIA have the largest amount of collaborative connections.

AAFC has collaborative connections with 7 department agencies:

• CGC
• NRC
• NRCan
• DFO
• CFIA
• PHAC
• HC

The volume of transfers between AAFC and stakeholders ranges from 23 to 2, with the highest amount of transfers occurring between AAFC and CFIA.

CFIA has collaborative connections with 7 department agencies:

• AAFC
• NRC
• NRCan
• DFO
• ECCC
• PHAC
• HC

The volume of transfers between AAFC and stakeholders ranges from 23 to 2, with the highest amount of transfers occurring between CFIA and AAFC.

3.1 Lessons Learned and Best Practices Related to Interdepartmental Collaboration

Best Practices

The foremost best practice mentioned by almost all respondents is the interdepartmental collaborative nature of the projects. In particular, the following aspects were highlighted:

• Dedicated funds for interdepartmental collaboration for a large collaborative project. One respondent noted that the QIS model is currently considered to foster interdepartmental discussions in the creation of a new federal research hub.
• Senior-level governance (ADM-level with a Management Advisory Committee). This feature was seen to be instrumental and considered a component of project success because it brought attention and focus to the common goals of the project.
• Strong project management approach, structure and tools. These were considered by respondents to contribute to the success of QIS by setting clear expectations and deliverables.
• Knowledge management tools (project charter and management templates/tools, guidelines and practice notes).

Lessons learned

Accorded to all respondents, issues with the shared IT infrastructure and integrated network for the implementation of the bioinformatics component were the main factors that hindered the collaboration between DAs and thus impacted progress of the QIS SPP. Bioinformatics for data storage, exchange and analysis constituted the major bottleneck for genomics projects and developing an integrated infrastructure common to all participating DAs was a challenge. Issues related to having SSC on board was outside of the scope of evaluations, but according to some respondents, the project team did a good job of positioning the GRDI database needs as a model for SSC to consider for infrastructure integration of scientific data sharing and collaboration.

A cyber intrusion and related IT shutdown at NRC (in 2014) had severe implications on key components of the project. QIS scientists had issues accessing their sequencing data results as access could take up to 6 months after sending samples to NRC. These IT issues caused delays in the sequencing work and transfer of data to the central database. As a result, a team of scientists, SSC representatives and senior managers have agreed to work closely together on the issues to improve IT infrastructure.

Interdepartmental financial and administrative issues were also reported during the last evaluation. The hiring of HQP was complicated and impacted the delivery of the QIS SPP. Genomic R&D highly depends on the contribution of post-doctoral researchers. In a specific example, ECCC and DFO had initial challenges in establishing sequencing capacity and expertise since their planned joint hiring/paying of a shared post-doctoral researcher was delayed (by 9 months) by administrative hurdles. Among key lessons learned from the last evaluation, financial and administrative processes involved in the delivery of the project should be verified (pre-arranged and pre-approved) and harmonized across participating DAs. Also, the admissible expenses for the O&M GRDI funds should be reviewed in order to fund staff working on the SPP.

Also, the admissible expenses for the O&M GRDI funds should be reviewed in order to fund staff working on the project.

Finally, for many respondents, the end of funding after five years had created lost opportunities as it is not possible to solve such complex problems, finalize and transfer technology for regulatory purposes. The following quote illustrates the continued need for supporting QIS interdepartmental collaboration:

"A lot of things put in place under QIS are ready to be used and being used by other groups — it's a good legacy. However, if we were able to do things differently, I would say extend the funding period for a couple more years or maybe focus on some sub-groups for a couple extra years of funding."

"Some groups were pretty much done documenting the molecular diversity [and near ready to develop and field-test diagnostic tools]. I was a proponent of continuing the QIS. We expressed that. But [senior management] decided to move on [to another project theme]."

C3: Antimicrobial Resistance (AMR)

1.0 Project Objectives and Activities

1.1 Background and Needs

Modern medical and veterinary practices depend the availability of antimicrobials and the development of resistance to antimicrobials is a serious global health threat. Antimicrobial-resistant infections are becoming more frequent and increasingly difficult to treat.^{Footnote 42} In 2019, the Chief Public Health Officer of Canada noted that with no action, annual worldwide human deaths attributable to antimicrobial resistance could reach 10 million by 2050.^{Footnote 43} Each year in Canada, more than 18,000 hospitalized patients acquire infections resistant to antimicrobials and "deaths directly related to Clostridium difficile alone have increased five-fold in the past decade."^{Footnote 44}

During the 2015 project planning workshops for the AMR SPP, federal departments identified overlapping human, animal and environmental antimicrobial issues, taking into account a One Health approach. Given widespread use of antibiotics in food production and the potential impacts of this practice on the presence of resistant bacteria in food, the project was focussed on this area of AMR study.^{Footnote 45} The project is part of Canada's response to AMR: the project fits within the innovation pillar of the 2015 Antimicrobial Resistance and Use in Canada: A Federal Framework for Action.

1.2 Research Purpose, Objectives and Activities

The AMR SPP project's timeframe is 2016-2021, with a total GRDI funding amount of $9M. The project lead is based at AAFC, and the project also involves CFIA, HC, PHAC and NRC.

The project uses genomics to better understand how food production contributes to the development of antimicrobial resistance. The project also explores strategies for reducing antimicrobial resistance in food production systems. The two overriding project goals are thus defined as follows:

1. gain understanding of the critical activities that contribute to antimicrobial resistance development in food production systems, and critical exposure pathways by which AMR bacteria reach humans
2. contribute to the validation of economically sustainable technologies, practices and policies to mitigate AMR development in food production systems.

The project includes five work packages that contribute to different aspects of these two broader objectives.

Work package 1: Whole Genome Sequencing (led out of CFIA, with AAFC and PHAC)
Using bacteria collections, samples and sequencing/informatics capacities within departments (PHAC, HC, AAFC, EC, CFIA), the team sequences and evaluates a large number of bacteria for resistance to antibiotics.

Work Package 2: Toolkit (led out of PHAC, with CFIA, HC and AAFC)
This work package aims to advance the technology and methods used to obtain and analyze genomics data concerning AMR. The team works to develop laboratory, bioinformatics, and visualization/modelling tools.

Work Package 3: AMR Transmission Dynamics (led out of PHAC, with AAFC, HC and CFIA)
Experts generate improved intelligence specifically on the genetic basis for AMR in resistant bacteria.

Work Package 4: Mitigation (led out of AAFC, with NRC)
The team evaluates strategies for mitigating AMR in food production systems.

Work Package 5: Risk (led out of PHAC, with HC and CFIA)
The team works on developing a suite of AMR risk assessment products that can be used for policy-making.

1.3 Alignment with Federal Government Mandates and Priorities

AMR is a very concerning issue that directly impacts the health of Canadians and the safety of the food production chain. The Federal Framework for Action explains that addressing the growing threat of antimicrobial resistance is a shared responsibility and the federal government's role includes brokering knowledge and facilitating innovation in this field.

Project planning documents and key informant interviewees underline the importance of the federal government conducting research now to inform the upcoming development of policies and regulations regarding AMR. Research into AMR is also aligned with the federal mandate for surveillance and stewardship on food safety and human health. The project is specifically intended to contribute to the Canadian Antimicrobial Resistance Surveillance System (CARSS) and Canadian Integrated Program for Antimicrobial Resistance Surveillance (CIPARS), both under the responsibility of PHAC.

The project is also aligned with the individual mandate of the primary departments involved, as described in the Federal Framework for Action:^{Footnote 46}

• PHAC provides national leadership on the public health aspects of antimicrobial resistance and use, and works with domestic and international partners in areas of surveillance, laboratory analysis, infectious disease outbreaks, awareness and public health guidance development.
• Health Canada regulates the approval of antimicrobial drugs for sale in Canada that are used in humans and animals, and is responsible for establishing policies and standards related to the safety and nutritional quality of the food supply.
• CFIA enforces Canadian regulatory requirements for the health and safety of animals and the food supply. Regulatory requirements are enforced through inspection, surveillance, and/or licensing/registration programs for livestock feeds, veterinary biologics, animal health and foods. The CFIA contributes to the development of national biosecurity standards and leads the on-farm food safety recognition program.
• AAFC supports the development and adoption of industry-led animal care, biosecurity, on-farm food safety assurance systems, and research into alternatives to antimicrobials. It also monitors trade and market access activities. AAFC works closely with the animal production industry to ensure its sustainability in an increasingly competitive global marketplace.

NRC's contacts with industry and the department's genomics capability are leveraged under work package 4, to help assess the nature of AMR across the "farm to fork" continuum.

1.4 Partners, Collaborators and Stakeholders

The project primarily involves scientists from five federal departments and agencies: AAFC, CFIA, HC, PHAC and NRC (for one work package). A broad range of federal stakeholders and end-users have been engaged, including representatives of academia and industry. The nature and extent of this engagement and collaboration is described in more detail below.

1.5 Financial Profile

Allocation of SPP GRDI funding per department is depicted in Table C3.1 below.

2.0 Achievements

2.1 Development of Unique Capabilities in the Federal Public Service

Within the federal government, in 2018-19, the AMR SPP supported the work of over 80 research scientists, professionals and technicians, along with five post-doctoral/visiting fellows, 23 graduate students, 12 undergraduate students and two administrative officers. In terms of developing unique capabilities in the federal public service, the project has so far contributed to:

• The creation of a science community on AMR through interdepartmental collaboration and the adoption of common protocols for sequencing. Multiple work packages contribute to the development of tools and techniques that have been shared through the Integrated Rapid Infectious Disease Analysis Project (IRIDA) platform and are being mobilized by researchers across departments.
• The integration of genomics in day-to-day activities, namely in improving the detection and prediction of resistance in CFIA labs. For instance, the CFIA was able to add surveillance for AMR genes in their routine genetic analysis for types of disease-causing E. coli and Salmonella. Interviewees noted that this builds CFIA's capacity for the introduction of risk assessments and new regulations in the future;
• The integration of genome sequencing into surveillance and monitoring programs, and into models at PHAC and CFIA;
• The successful piloting and deployment of Nanopore technologies for sequencing on a large scale in HC, PHAC and CFIA laboratories;
• PHAC's capacity to develop risk assessments related to food;
• Useful information being provided to HC and PHAC's drug directories.

In terms of forthcoming developments, interviewees at AAFC are confident that science generated from the AMR SPP will inform policies and evolving regulations in the agricultural sector, and will have impacts on issues of related to trade of food products.

2.2 New Scientific Achievements

Although AMR is still ongoing, several scientific achievements have been reported to date, including:

A large volume of whole genome sequences provided to project partners. The SPP has contributed to a very large collection of whole genome sequences to study AMR in the Canadian context. The latest performance report for AMR indicates that "the development of a large collection of bacterial sequences to be used to gain a better understanding of the spread of AMR in the food supply is well underway, with the completion of over 6,000 draft bacterial genomes and 48 shotgun metagenomes."

A greater understanding of how food production contributes to the development and transmission of AMR. For instance, researchers showed that antibiotic resistance genes in soils increased following fertilization with organic matter from food waste compost or yard waste compost. Furthermore, CFIA has been able to apply genomic technology to analyze the existing collections of bacteria obtained through surveillance to understand AMR in pork, poultry, and beef. For example, the use of antimicrobials in poultry was shown to clearly increase antimicrobial resistant bacteria in poultry litter, some of which could pose a risk to human health.

Identification and testing of natural alternatives to antimicrobials and methods to reduce and prevent AMR in food production. ARM researchers discovered that some essential oils and fruit extracts can help combat infections in poultry. AMR SPP researchers also showed that bacteria from conventional swine setting were resistant to a larger number of antimicrobials than the isolates originating from antimicrobial-free pigs and that "anaerobic digestion of manure showed potential to eliminate AMR bacteria, with the added benefit of generating clean burning biogas and providing producers with a new revenue stream." Findings from AMR SPP work package 4 have been mobilized in further non-GRDI research projects. For example, a company (in collaboration with Université Laval and AAFC) received NSERC funds for a project about the use of cranberry extract as an alternative to antibiotics in swine production.

New or improved research processes. The 2018-2019 year-end performance report for SPPs indicates 14 research tools and 12 research processes were developed or improved through the AMR project, including:

• Low-cost sequencing approaches and different methods for the preparation of samples;
• A new multiplex Polymerase Chain Reaction (PCR) assay for genes conferring resistance to certain antimicrobials (cephalosporins and fosfomycin);
• A heat map analysis tool with in-house programming for the swine production continuum;
• An integrated graph database for real-time prediction of anti-microbial resistance and predictive marker discovery for virulence and AMR;
• A mouse model for studying horizontal transfer of AMR genes in the intestine.

2.3 Development of Evidence-based Public Policy and New Knowledge/ Technologies

Interview respondents confirmed that using advanced genomic technologies is essential in the study of AMR and the generation of new knowledge. AMR is still ongoing, but according to the majority of interviewees, the knowledge generated through the AMR SPP will impact regulations and policies in the near future. The AMR 2018-2019 performance report indicates that legacy resources and publications from the project will significantly contribute to "filling knowledge gaps" and can be applied to "inform future policy and decision-making". However, at this stage, the impact of the AMR SPP on public policy remains to be seen.

Below are some examples of technologies developed under the project. Reports do not always indicate which departments or organizations are using those products, but state that end-users of AMR SPP technologies include research teams and science-based agencies in agri-food, food safety and public health interested in tracking AMR:

• The StarAMR technology detects AMR genes and is designed to function in sync with other existing resources, including CARD and ResFinder created by university researchers. According to one AMR scientist, StarAMR has been downloaded 7,000 times. New Genome Canada funding was awarded in June 2018 to further support its uptake. The tool is being deployed by PHAC, CFIA and academic collaborators (Dalhousie and McMaster). Codes are disseminated freely via GitHub and public archives.
• The "Mob-suite" is an analytical tool that can predict AMR phenotype as well as the mechanisms of transmission associated with resistance. This software is freely available to GRDI-AMR and other researchers.
• Work continues on developing an Integrated Assessment Model (IAM). Preliminary simulations were carried out to test the validity of the model. A draft Quantitative Microbial Risk Assessment (QMRA) model was completed for Salmonella Heidelberg in poultry.
• AMR researchers developed the statistical framework PhenoRes, which predicts the minimum inhibitory concentration for 13 antimicrobials in Salmonella based on genome sequence data.

The project is still ongoing and most mid or long-term impacts of the research have not manifested yet. However, researchers as well as senior management within departments are adamant on the importance of this work and expect that the technologies, tools and findings coming out of the SPP will contribute to building the government's capacity to address AMR and develop new policy and regulations. Some participating scientists raised questions about the continuation of the work beyond the funding period for the SPP, but had no doubt as to the relevance of the work up to this point.

2.4 Knowledge and/or Technologies Transferred to End-users

Knowledge transfer activities

The updated knowledge transfer plan for AMR states that knowledge and methodologies developed through the project are meant to be fully transferred to relevant federal programs by the end of the funding period. The knowledge transfer plan also indicates that in the final year of the project, "a workshop will be held to focus on the communication of project outputs with end-user groups and to create strategies whereby there is effective transfer and uptake of the tools and knowledge produced by the project within end-user communities."

So far, AMR researchers have organized and participated in a variety of meetings to transfer information between the teams working on different work packages. Data analysis tools ready for adoption are hosted on IRIDA. The project transfers whole-genome sequencing data to publicly available sequence databases, such as the US National Center of Biotechnology Information (NCBI) and the Comprehensive Antibiotic Resistance Database (CARD).

AMR bioinformaticians and genomics researchers also work directly with end-users conducting regulatory and surveillance activities within the federal government. AMR scientists have held monthly meetings with the Canadian Integrated Program for Antimicrobial Resistance Surveillance (CIPARS). In 2018-2019, for instance, a large proportion of Shiga-toxin producing E. coli collected from a veal surveillance program were predicted to be resistant to multiple antimicrobial agents and strains, and were identified for transfer to the CIPARS for further characterization.

Interviewees noted that the project involves direct partnerships with external end-users in industry. Research into alternatives to antimicrobials is being conducted at farms, in direct collaboration with producers (e.g., pig producers in Quebec). Certain antimicrobial strategies identified through the project are currently being used on the ground (e.g., as part of the Sustainable Beef Production Guidelines). Interviewees also noted that the AMR SPP works to engage the human health care sector, although this was described as more challenging, given the need to engage provincial jurisdictions and the complexity of health-related activities such as clinical trials and accessing isolates from human origin from provincial public heath institutions (due to confidentiality reasons).

The AMR SPP also led to various instances of collaboration with university-based researchers. For instance, AMR SPP participants worked with colleagues at Dalhousie University on building a Salmonella virulence database, and studying the relationship between virulence and resistance genotype. Members of the AMR project were also invited to participate as co-applicants in the Genome Canada project "Antimicrobial Resistance: Emergence, Transmission, and Ecology (ARETE)."

The AMR SPP also includes a variety of other knowledge transfer activities, such as media interviews, press releases, community presentations and public-facing documents. Overall, performance reporting for 2018-19 alone indicates five end-user consultations and a public meeting; 12 instances of AMR experts providing science advice (including to senior management); one material transfer agreement; three knowledge transfer workshops with stakeholders/end-users, and 25 requests for research results, papers, collaborations. Examples of such activities include:

• Participation in the Canadian Bioinformatics Workshop on Infectious Disease Genomic Epidemiology workshop held in Toronto (July 10-12, 2018).
• Participation in the Regional Research Users Meeting on Food Innovation at the Guelph Research and Development Centre on June 18, 2018.
• Presentation to the Ontario Ministry of Agriculture, Food and Rural Affairs (OMAFRA) in March 2018 on the Integrated Assessment Model and characterizing the relationship between livestock antimicrobial use and antimicrobial resistance in people in Ontario.

AMR produced a large volume of peer-reviewed publications or presentations, which contribute to the dissemination of information to end-users. The bibliometric analysis identified a total of 187 GRDI publications associated with the AMR SPP. For 46 publications indexes in Scopus, the average citation per document was 9.2, with a positive high Field-Weighted Citation Impact (FWCI)^{Footnote 47} score of 1.61. Of 146 AMR publications used for a collaboration network analysis, about 60% were conference presentations (33.6%) or conference participations (26.7%), and the remainder were mostly articles (34.9%). Those publications feature 300 scientists in 79 institutions and 17 countries. Over half were published collaboratively between at least two organizations (54.8%) and almost a third of all publications involved three organizations or more (29.5%). Academic partners, especially Canadian universities, are numerous. See Figure C3.1.

Figure C3.1: Collaborative Network Analysis - AMR SPP

Figure C3.1 - Text version

Stakeholders include 20 academic institutions, 7 provincial/state government departments,4 Canadian federal departments and 2 RTOs and one corporate organization.

AAFC has 23 collaborative connections Stakeholders include:

• Tsinghua University, China
• University of Hong Kong, Hong Kong
• Western University
• University of Lethbridge
• Colorado State University, USA
• Université de Sherbrooke
• University of Calgary
• University of Manitoba
• University of British Columbia
• Dalhousie University
• Simon Fraser University
• University of Alberta
• Harvard University, USA
• University of Gothenburg, Sweden
• University of Helsinki, Finland
• CFIA
• PHAC
• Calgary Laboratory Services
• USDA, USA
• Feedlot Health Management Serviced Ltd
• Alberta Agriculture and Rural Development
• Centre de recherche en sciences animales de Deschambault

The volume of transfers between AAFC and stakeholders ranges from 12 to 1, with the highest amount of transfers occurring between AAFC and PHAC

PHAC has 23 collaborative connections. Stakeholders include:

• University of Manitoba
• University of British Columbia
• Dalhousie University
• Simon Fraser University
• University of Lethbridge
• University of Alberta
• University of Calgary
• University of Guelph
• University of Prince Edward Island
• University of Saskatchewan
• Colorado State University, USA
• CDC Atlanta, USA
• Agriculture and Rural Development
• BC Public Health
• Calgary Laboratory Services
• Inst national de santé publique du Quebec
• Public Health Ontario Labs
• Alberta Provincial Lab for Public Health
• CFIA
• AAFC
• HC
• Centre de recherche en sciences animales de Deschambault
• European Commission Joint Research Centre, Italy

The volume of transfers between PHAC and stakeholders ranges from 23 to 1, with the highest amount of transfers occurring between PHAC and the University of Guelph

CFIA has 20 collaborative connections. Stakeholders include:

• University of Gothenburg, Sweden
• University of Helsinki, Finland
• Harvard University, USA
• University of Bologna, Italy
• University of Lisbon, Portugal
• Tsinghua University, China
• University of Hong Kong, Hong Kong
• Western University
• University of Calgary
• University of Manitoba
• University of British Columbia
• Simon Fraser University
• University of Guelph
• USDA, USA
• AAFC
• PHAC
• HC
• Calgary Laboratory Services
• Alberta Agriculture and Rural Development
• European Commission Joint Research Centre, Italy

The volume of transfers between CFIA and stakeholders ranges from 11 to 1, with the highest amount of transfers occurring between CFIA and AAFC.

HC has 19 collaborative connections. Stakeholders include:

• University of Manitoba
• University of British Columbia
• Dalhousie University
• Simon Fraser University
• University of Alberta
• University of Calgary
• University of Guelph
• Université de Sherbrooke
• University of Prince Edward Island
• University of Saskatchewan
• PHAC
• CFIA
• AAFC
• Inst national de santé publique du Québec
• Public Health Ontario Labs
• Alberta Provincial Lab for Public Health
• BC Public Health
• CDC Atlanta, USA
• Calgary Laboratory Services

The volume of transfers between HC and stakeholders ranges from 13 to 1, with the highest amount of transfers occurring between HC and the University of Guelph.

Internationally, AMR experts have provided scientific advice and played a prominent role in the Joint Programming Initiative on AMR (JPIAMR) and the Transatlantic Taskforce on Antimicrobial Resistance (TATFAR). Canadian experts have also been involved in meetings of the United Nations Food and Agriculture Organization and World Health Organization on foodborne antimicrobial resistance.

2.5 Impact of Technologies Transferred

As noted previously, the AMR SPP is still ongoing. The impacts of the technologies and methods developed through the project and shared with end-users is hard to capture at this time. However, interviewees are confident that the network of scientists working on AMR is contributing to national and international progress, by enhancing stakeholders' understanding of AMR (within and outside government) and exploring strategies to prevent the development of ARM in food systems. Respondents also underlined the importance of developing effective risk-assessment tools that can leverage metagenomics, given a foreseeable need to develop new policies and regulations.

The most recent performance report for the AMR SPP contains a statement of socio-economic benefits accrued from project results. This section describes the large collection of whole genome sequences as an "invaluable resource" for studying AMR in the Canadian context "now and in the future". The report also highlights cost efficiency gains that have resulted from new sequencing approaches deployed to project partners: "In the first year of the project, we closed seven genomes for $1,000/genome, in the past two years, almost 1000 genomes were closed at a cost of $85/genome." The report also refers to groundbreaking research conclusions regarding the development of AMR through the food production chain, and alternatives that can help prevent it.

3.0 Interdepartmental Research Collaboration

All AMR interviewees had very positive views on the interdepartmental collaboration achieved through this SPP and felt that the same level of progress would not have been achieved as rapidly without the funding and collaborative structure of the project. One interviewee explained: "we are doing science now that we could not do if we did not have a relationship." The fact that researchers could leverage the capacity and expertise in multiple departments while also engaging with end-users directly was key in many of the project's achievements. According to respondents, interdepartmental collaboration through this SPP allowed for the sequencing of a large volume of data based on samples held by multiple departments. Collaboration has also been beneficial in supporting a One Health approach that connects animal, human and environmental health. The AMR SPP has allowed participating researchers to leverage expertise and effectively disseminate valuable information, processes and technologies across departments. Interview respondents indicated that interdepartmental collaborations contributed to creating a strong science policy community on AMR in Canada, positioning the country as a leader in the field. Challenges mentioned by participating scientists included difficulties sharing data across departments because of IT issues.

The collaborative network analysis provides an illustration of interdepartmental connections through publications. Four of the five departments involved in the project are represented in the network. All four organizations have collaborated on publications. As shown in Figure C3.2, AAFC and PHAC dominate the collaboration map for the AMR SPP, but the connections between AAFC, PHAC, and CFIA are strong. HC has fewer publications overall and fewer co-publication linkages to other federal partners.

Figure C3.2: Collaborative Network Analysis - AMR SPP - Federal Departments only

Figure C3.2 - Text version

Web diagram of AMR SPP Collaborative Network Analysis - Federal Departments only

AAFC and PHAC have 12 co-publications
AAFC and CFIA have 11 co-publications
PHAC and CFIA have 8 co-publications
PHAC and HC have 4 co-publications
CFIA and HC have 3 co-publications

3.1 Lessons Learned and Best Practices Related to Interdepartmental Collaboration

Interviewees familiar with the AMR SPP described improvements between the FWS SPP and AMR in terms of management practices and collaboration. More specifically, managers learned how to effectively manage multi-collaborator projects with better defined work packages, and more frequent teleconference meetings. These improvements led to increased awareness and involvement of participants, and better integration of various research projects to support SPP objectives.

The first AMR SPP annual performance report (2015-2016) captured lessons learned from each work package. The notes relate to details in various dimensions including a few staffing/capacity challenges; some difficulties in tracking expenditures and instances of overhead; delays and practical challenges; issues in communication, etc. The performance report also indicates that lessons learned were to be examined during the annual GRDI-ARM workshop. None of these issues appear to have generated major concerns in the conduct of the project. These elements were not raised by interviewees.

C4: EcoBiomics

1.0 Project Objectives and Activities

1.1 Background and Needs

Canadian ecosystems are being degraded or used unsustainably by land use disturbances linked to economic development activities in a variety of sectors such as agriculture, fisheries, urbanization, water consumption and natural resource development (e.g., forestry, pulp, mining, energy/oil and gas). Together with anthropogenic disturbances, climate change is also affecting the habitat of species resulting in biodiversity changes and loss of species. Over the past decades, this decline of biodiversity is recognized globally as one of the most important environmental issues facing humanity. In response to threat and its obligations as a party to the United Nations Convention on Biological Diversity, the first Canadian Biodiversity Strategy was developed in 1995.^{Footnote 48} In support of this strategy and departmental mandated objectives, the federal science-based departments and agencies (DAs) are working in advancing science to monitor, assess, predict and respond to these ecosystems and biodiversity changes.

The advancements in the fields of microbial, environmental and applied metabarcoding and metagenomics are now providing "an unprecedented ability to characterize the enormous diversity of microorganisms and invertebrates sustaining soil health and water quality."^{Footnote 49} Such tools allow for moving from the sequencing of individual species (like in the QIS SPP) to sequencing and studying interaction of communities of organisms with their environment. By taking advantage of advancements in genomic approaches and tools, the EcoBiomics project was established to respond to threats to the integrity of Canadian ecosystems, biodiversity and human health from anthropogenic environmental change and disturbances.^{Footnote 50}

1.2 Research Purpose, Objectives and Activities

The EcoBiomics SPP applies metagenomics to assess and monitor soil and water quality by integrating metagenomics analysis of microbial and invertebrate communities of several ecosystems across Canada. Leveraging the bioinformatic platform developed under QIS, the SPP advances a government-wide platform to centralize and harmonize sequencing data, bioinformatics / metagenomic analysis workflows. The EcoBiomics SPP has three overarching objectives:^{Footnote 51}

• Develop standard soil and water methods and a federal Bioinformatics Platform to harmonize analyses of metabarcoding , metagenomics and metatranscriptomics data across federal departments/agencies, and communicate results to Canadians;
• Establish genomic observatories and comprehensive biodiversity baselines for assessing future changes to water and soil biodiversity at long-term environmental monitoring sites in Canada;
• Develop new knowledge to improve water quality and soil health by comprehensively characterizing aquatic microbiome, soil microbiome, and invertebrate zoobiome, and testing hypotheses to enhance environmental assessment, monitoring, and remediation activities.

The SPP includes six work packages organized in themes that contribute to these broader objectives. In total, the sixteen research project activities are sampling and analyzing soil microbiome, aquatic microbiome and invertebrate Zoobiome across seven study sites across Canada (Figure C4.1). A number of project activities from different themes are co-located to enable interdepartmental collaboration and interdisciplinary research at the soil-water interface: Wood Buffalo Park (Fort McMurray, Alberta), Ells River (Alberta) and South Nation River watershed (Ontario).

Figure C4.1: Locations of EcoBiomics sampling sites (genomics observatories)

Figure C4.1 - Text version

Different sized circles on the map depict the different concentrations of EcoBiomics sampling sites in Canada. The highest concentration is in Northern Alberta Oil Sands area and Southern Ontario Great Lakes area. Other smaller concentrations are scattered in Ontario, southern Quebec and New Brunswick.

According to respondents and project documents, these locations called genomics observatories were selected with a view to leverage existing field study sites of ongoing monitoring programs (as well as from established collaborations with local economic and environmental stakeholders. The six themes and participating DAs are presented below. Theme lead(s) are highlighted in bold.

Theme 1: Sampling /Nucleic Acid Preparation (led by CFIA, with all DAs)

• Evaluating extraction methods used by different DAs for standardization, improving soil sampling and nucleic acid extraction methods using microfluidics (NRC's Power Blade Platform), and evaluating and establishing SOPs and protocols.

Theme 2: Soil Microbiome (led by NRCan, NRC, ECCC, AAFC, CFIA)

• Sequencing microbial communities from multiple soil and roots samples from ongoing experimental projects in oil sands areas (Alberta), in boreal forests (Ontario and Quebec) and from long-term field trials in agriculture (Ontario). Improving the understanding of the effect of anthropogenic disturbances on soil nutrient cycling and water quality and modelling the role of soil microbiome in land reclamation/remediation for proper sustainable resource development.

Theme 3: Aquatic Microbiome (led by NRC, with ECCC, CFIA, NRCan, AAFC, and PHAC)

• Assessing the temporal and spatial impacts on soil/aquatic microbiota of eutrophication and land use disturbance over long time periods and under different (climate/management) scenarios from the two projects and associated study sites:
• Biodiversity impacts associated with harmful algal (cyanobacterial) blooms (HABs) from agricultural and urban sources at Lake Erie/Thames/Leamington, Lake Champlain/St. Lawrence watershed, South Nation watershed (NRC, EC, NRCAN, AAFC, CFIA, PHAC);
• Comparative analysis of soil and water ecosystems in forestry: Atlantic rivers (northern New Brunswick), mining (Athabasca) watershed disturbed and natural settings (NRCan, EC, NRC).

Theme 4: Invertebrate Zoobiome (led by ECCC, with DFO, CFIA, NRCan, PHAC, and AAFC)

• The theme focused on identifying key target groups of invertebrates by identifying key subsets of markers, developing field methods for sampling and transportation of sampled material, sampling and developing cases for DNA-sequence-based invertebrate biodiversity data in relation to the principles of cumulative effects assessment. Field sites were selected to maximize synergy between departmental capacities (e.g. salmon habitat (DFO), forest habitat (NRCAN), river monitoring (EC) in Atlantic Canada; agricultural land-use mitigation (AAFC), river monitoring (ECCC), flying insects (PHAC; CFIA) in the South Nation (Ontario) experimental watershed study (AAFC)).

Theme 5: Bioinformatics Platform (led by AAFC, with NRC)

• Leveraging the QIS platform, this theme focused on providing a shared computing and storing infrastructure through the development and support of a high-performance computing (HPC) based and a Web-based data management, integrated genomics workflow and analysis software platform.

Theme 6: Environmental Assessment Integration (co-led by ECCC and AAFC, with all DAs)

• Integrating results coming from other themes and demonstrating the added value of metagenomics in improving environmental assessment, monitoring and remediation activities.

1.3 Alignment with Federal Government Mandates and Priorities

The Canadian Biodiversity Strategy^{Footnote 52} explains the need and responsibilities for Canada to address the growing threat of the degradation of ecosystems and the loss of species and genetic diversity which result from human activities. The SPP is supporting Canada's obligations with respect to the Convention on Biodiversity.^{Footnote 53} The Federal Sustainable Development Strategy^{Footnote 54} is monitored by Canadian Environmental Sustainability Indicators (CESI) (such as water quality and regional water ecosystems, biodiversity, climate change, natural resource protection and use, pollution and waste and environmental human health) by several DAs mostly with individual contributions to specific indicators confirming that the protection of biodiversity and ecosystems is a shared responsibility.

The federal government's role in brokering knowledge and facilitating innovation is aligned with research priorities outlined in the federal government's new strategy: Seizing Canada's Moment: Moving Forward in Science, Technology and Innovation,^{Footnote 55} which recognizes protection of the environment as an important societal challenge and identifies environment, and climate change research and technology as research priorities.

SPP planning documents and key informant respondents underline the importance of the federal government – which is a priority across federal DAs – conducting research to provide baseline information and more accurate biodiversity indicators using innovative and cost-effective genomics approaches and tools. The complementary activities and responsibilities (science, regulatory, resource management, and climate change) are expected to be enhanced through integration of work in federal DAs. Thus, the EcoBiomics SPP is envisioned to establish improved capacity to assess and support Canada's effort to minimize economic and environmental threats as well as to improve emergency preparedness and the long-term sustainability of resource industries. However, for some respondents, the exclusion of regulated species in the SPP is a lost opportunity since they believe the SPP could have played a bigger role at monitoring preventing regulated invasive species and pathogens with implications for international market commitments.

1.4 Partners, Collaborators and Stakeholders

The SPP primarily involves scientists from seven federal DAs: AAFC, CFIA, DFO, ECCC, NRC, NRCan, and PHAC. A broad range of federal stakeholders and end-users have been engaged, including representatives of academia, governments and industry. The nature and extent of this engagement and collaboration is described in more detail below.

1.5 Financial Profile

The EcoBiomics SPP timeframe is five years from FY 2016-17 to 2020-21, with a total GRDI funding amount of $9.3M (Table C4.1). In addition, the participating DAs contributed to the project with in-kind contributions estimated at $5.7M, which represent 38% of total project costs. AFFC received the largest share of GRDI funding (31%) but also contributed the most to the project with in-kind (44%).

2.0 Achievements

2.1 Development of Unique Capabilities in the Federal Public Service

Within the federal government, in 2018-19, the EcoBiomics SPP supported the collaborative work of over 60 research scientists, biologists, bioinformatics specialists, technologists, and additional undergraduate students across seven federal DAs. In terms of developing unique capabilities in the federal public service, the project has so far contributed to the:

• Establishment of project-specific bioinformatics platform and data management web application (SeqDB) available to all participating GRDI DAs which includes many commonly used bioinformatics tools for metagenomics and metabarcoding analyses and a harmonized approach to data management. According to several respondents, this capacity is unique in the world in terms of having a cross federal government platform that allows harmonization of data collection and analysis and comparing results between DAs.
• Progress in bioinformatics analyses that are already showing the advantages of metagenomics methods over conventional methods. Such advanced analytical capabilities contributed to better characterizing the massive microbial and invertebrate biodiversity in soil and water as well as generating international data and metadata standards that can serve as a valuable baseline for future comparison.
• Development of metagenomics capacity at NRCan, ECCC and DFO. For example, the SPP influenced NRCan in securing funds for the maintenance of metagenomics infrastructure and expertise with the hiring of a permanent lead scientist. Also, the SPP enabled the use of metagenomics at DFO and at ECCC in complement to established bio-monitoring methods.

For a number of participating DAs, the EcoBiomics SPP was instrumental to enabling and/or advancing environmental metagenomics monitoring and analytical capabilities.

"EcoBiomics is allowing us now to look at things like networks of service availability within ecosystems, and then you also visualize and use those for monitoring purposes, something that we could never have contemplated even five years ago. It's transformational."

2.1 New Scientific Achievements

The following points summarize the progress achieved in the various components of the SPP:^{Footnote 56}

• Since 2018, the establishment of the Bioinformatics Platform and its SeqDB data management system allowed for the collection of 7,457 soil, water, and invertebrate samples across Canada.
• In FY 2018-19, the EcoBiomics project continued active research on 16 soil microbiome, aquatic microbiome and invertebrate zoobiome projects across Canada.
• To date the centralized sequencing facility at NRC-Saskatoon has completed MiSeq and HiSeq Illumina DNA sequencing runs that have generated many millions of DNA sequence reads for bioinformatics analyses.
• Continued work for m QIS to advances the development of microfluidic device and a chip with the NRC.
• Improved and harmonized protocols for water sample collections by scientists across DAs, and contribution to the development of international standards.
• Significant advances to the Bioinformatics Platform and associated training opportunities led to a growing use of the Platform to support bioinformatics analyses.
• Bioinformatics analyses are showing the advantages of metagenomics methods over conventional methods for better characterization of the enormous microbial and invertebrate biodiversity in soil and water.
• The importance of characterizing this biodiversity has been demonstrated for soil microbial community changes associated with remediation and reforestation needs for disturbed soils in the oil sands Athabasca region, and for recovery of boreal forest soils affected by disturbances such as wildfire, pest infestation, and biomass removal practises.
• Also, studies of agricultural soils in Ontario are revealing the impact of agricultural management practices (e.g. crop rotation) on soil microbial communities and the need to consider these microbial changes in managing soil nitrogen and phosphorus cycles.
• Preliminary results also suggest the reservoir of soil microbial diversity can be an important contributor to aquatic microbiome in rivers such as the Thames River and South Nation River in Ontario, which are the focus of phosphorus reduction management actions.
• Aquatic metagenomics projects are investigating harmful algal blooms in transboundary lakes (Erie, St. Clair, Champlain) collaborating with federal/provincial water quality monitoring programs to study soil and river microorganisms being seeded downstream into lakes.
• Invertebrate zoobiome research continued in collaboration with Canadian Biomonitoring Network (CABIN) activities to identify advantages of metabarcoding for establishing benthic invertebrate diversity baseline conditions in key areas such as the Peace-Athabasca Delta downstream of the oil sands, and the South Nation River in Ontario in advance of an experimental removal of riparian habitats. Further invertebrate research is studying invertebrate communities in salmon guts in Atlantic Canada streams as a new way to assess salmonid habitat suitability.
• A research paper looking at natural dynamics within a very large complex ecosystem using genomics to be published in the Proceedings of the National Academy of Sciences is considered as innovative and ground-breaking.

Importantly, almost all interviewees highlighted that the full integration of all samples and analyses across project themes will be completed only a few years after the end of the Ecobiomics SPP. Cutting edge scientific achievements are expected but more time and effort is needed to process data and disseminate the knowledge needed to inform mitigation strategies and measures to preserve soil and water biodiversity. As noticed by one scientist:

A lot of the science will come after the end year of funding.

2.2 Development of Evidence-based Public Policy and New Knowledge/ Technologies

Interview respondents confirmed that using advanced genomic technologies is essential in the study of EcoBiomics and the generation of new knowledge. EcoBiomics is still ongoing, but according to the documentation and respondents, multiple case studies derived from workshops in the next 2-3 years are anticipated to demonstrate metagenomics applications for making better science-based decisions, in a faster and more cost-effective way, for protecting biodiversity and ecosystems across Canada. As mentioned by some respondents, an additional 10 to 15 years will be needed to observe long-term outcomes such as enhanced sustainability and management of resources and reduced environmental harm.

2.3 Knowledge and/or Technologies Transferred to End-users

The updated knowledge transfer (KT) plan for EcoBiomics mentions that consultations were held during the development of the EcoBiomics SPP. However, some respondents highlighted the need to engage end-users earlier in the scoping of the objectives and more effectively during the whole project life cycle in order to 1) better align the objectives and deliverables, and importantly, 2) ensure uptake of knowledge and tools by end-user at the end of the funding cycle. As noted by some respondents, the EcoBiomics SPP is the very first step of a very ambitious yet cutting-edge research program that will require additional funding and collaborative efforts to address such complex issues of importance to Canadians.

The integration work by the sixth theme near the end of the SPP is expected to produce deliverables of interest for end-user engagement and to "formalize the linkage of the EcoBiomics Bioinformatics Platform to other existing soil and water databases in environmental programs." It is also envisioned that new metagenomics tools, data and analysis will be considered and incorporated into some programs within AAFC, CFIA, DFO, ECCC, NRCan and NRC. In particular, the EcoBiomics project activities are intended to be used as demonstrations to facilitate longer-term partnerships with:^{Footnote 57}

• ECCC's national CABIN Network for monitoring benthic invertebrate diversity and National Water Quality Monitoring Program by collaborating on common study sites to establishing weblinks or data sharing with CABIN databases and their website.
• The Canadian Environmental Assessment Agency (CEAA), and other federal and provincial regulators, responsible for conducing environmental impact assessments to develop guidance for incorporating metagenomics considerations.
• Regulatory risk assessment programs for chemicals and biotechnology products by developing guidance for considering metagenomics methods for improving environmental risk assessments for protecting soil and aquatic biodiversity.
• CFIA and PHAC to demonstrate bioinformatics screening capabilities to identify microbes or invertebrates of pest and or quarantine concern to regulatory agencies.

Interview respondents highlighted that publications are produced as much as possible during the course of the SPP. The annual workshop meetings are also reportedly a great opportunity to interact with end-users from other DAs. The participation of end-users is reported as very good to date considering that the protection of biodiversity is a shared responsibility across multiple SBDAs with no clear mandate for environmental stewardship of natural resources industries. Based on the QIS SPP experience and outcomes, most respondents were advocating to make EcoBiomics data open and available to the public once scientific and technological advancements are published.

To date, the SPP has produced a large volume of peer-reviewed publications or presentations, which contribute to the dissemination of information to end-users. The bibliometric analysis identified a total of 123 GRDI publications associated with the EcoBiomics SPP. Of 123 EcoBiomics publications used for a collaboration network analysis, 53% were conference presentations, 18% conference participations, 17% articles, 5% books and 7% other types of publications. For 21 publication indexes in Scopus, the average citation per document was 12.6 with a positive high Field-Weighted Citation Impact (FWCI) score of 2.35. EcoBiomics publications has the highest share (50%) of the top 10% highly cited publications in the field.

Those publications feature 179 scientists from 46 institutions and 8 countries. Over half were published collaboratively between at least two organizations (53%) and more than a third of all publications involved three organizations or more (36%). Academic collaborators from Canada and abroad are numerous (Figure C4.2). EcoBiomics experts have also provided scientific advice and played a prominent role in national or international genomics-related committees.

Figure C4.2: Collaborative Network Analysis – EB SPP (with >2 co-publications)

Figure Figure C4.2 - Text version

Stakeholders include 12 academic institutions, 1 provincial/state government departments, 7 Canadian federal departments and 1 RTOs.

ECCC has 15 collaborative connections. Stakeholders include:

• Sydney Inst Marine Science, Australia
• University of New South Wales, Australia
• University of New Brunswick
• University of Guelph
• Macquarie University, Australia
• University of Minnesota, USA
• University of Saskatchewan
• Royal BC Museum
• CSIRO Australia
• NRCan
• DFO
• NRC
• CFIA
• AAFC
• PHAC

The volume of transfers between ECCC and stakeholders ranges from 14 to 2, with the highest amount of transfers occurring between ECCC and the University of Guelph

NRCan has 11 collaborative connections. Stakeholders include:

• University of Guelph
• McMaster University
• INRS
• Université de Montréal
• University of New Brunswick
• DFO
• CFIA
• PHAC
• NRC
• ECCC
• AAFC

The volume of transfers between NRCan and stakeholders ranges from 12 to 1, with the highest amount of transfers occurring between NRCan and McMaster University.

AAFC has 13 collaborative connections. Stakeholders include:

• University of Waterloo
• McMaster University
• Macquarie University, Australia
• University of New Brunswick
• University of New South Wales, Australia
• University of Guelph
• INRS
• DFO
• ECCC
• PHAC
• NRC
• CFIA
• Food Standards Australia New Zealand, Australia

The volume of transfers between AAFC and stakeholders ranges from 6 to 1, with the highest amount of transfers occurring between AAFC and University of Waterloo as well as AAFC and Food Standards Australia New Zealand, Australia.

DFO has 9 collaborative connections. Stakeholders include:

• McMaster University
• University of Guelph
• University of New Brunswick
• ECCC
• NRCan
• NRC
• AAFC
• PHAC
• CFIA

The volume of transfers between AAFC and stakeholders ranges from 5 to 1, with the highest amount of transfers occurring between DFO and NRCan

NRC has 8 collaborative connections. Stakeholders include:

• NRCan
• ECCC
• AAFC
• CFIA
• INRS
• Université de Montréal
• University of Toronto
• University of Saskatchewan

The volume of transfers between NRC and stakeholders ranges from 6 to 2, with the highest amount of transfers occurring between NRC and NRCan

PHAC has 6 collaborative connections. Stakeholders include:

• University of Guelph
• AAFC
• CFIA
• NRCan
• DFO
• ECCC

The volume of transfers between PHAC and stakeholders ranges from 3 to 1, with the highest amount of transfers occurring between PHAC and AAFC.

2.4 Impact of Technologies Transferred

As noted previously, the EcoBiomics SPP is still ongoing and the integration and analysis of samples collected across Canada is still to come. The impacts of the technologies and methods developed through the SPP are hard to capture at this time.

While some respondents are confident that the network of scientists working on EcoBiomics is contributing to national and international progress in developing advanced assessment tools, some others have reiterated that metagenomics-based environmental assessment is at a fundamental stage within the innovation spectrum. According to many respondents, an additional 10 to 20 years of research work are needed to characterize the complexity of biodiversity of ecosystems and connections with anthropomorphic and environmental changes, and for technologies to be transferred to end-users. Most respondents recognized that the scope of EcoBiomics is cutting-edge science but very fundamental considering SPPs are intended to address national priorities within a five-year funding timeframe.

The most recent performance report for the EcoBiomics SPP (2018-19) contains a statement of benefits accrued from project results. The main benefit reported is the establishment of the bioinformatics platform, the increased number of samples that have been sequenced, and bioinformatics training has been provided for an increasing number of EcoBiomics participants who are communicating these results to end-user communities and contributing to socio-economic benefits. For example, representatives from the ECCC's CABIN network and the National Water Quality Monitoring Program continue to be engaged. Project teams also engage with ministries and with the Canada's Oil Sands Innovation Alliance (COSIA) to monitor changes in environmental requirements and their impacts.

Documents also report regular engagement with water quality monitoring program end users in ECCC and Ontario Ministry of Environment to ensure EcoBiomics water sampling integrated with priority federal and provincial water monitoring programs in the Great Lakes. The International Group on Earth Observations Biodiversity Observation Network (GEO BON) was created and several presentations have been made by leading members of Global Omics Observation Network (GLOMICON) to promote the need for metagenomic data standards and raise awareness nationally and internationally. Recently, the project engaged a new potential end user (Canadian Agri-Food Policy Institute, CAPI).

EcoBiomics scientists also served on the Health Canada federal/provincial committee revising the Canadian Recreational Water Quality Guidelines and contributed to the development of Biodiversity Information Standards as well as different working groups in Canada and in Europe (International Bioeconomy Forum: Microbiome Working Group, Genomic Standards Consortium conference and European Horizon 2020 project).

Of the 16 sub-projects, some are already starting to engage end-users and having short-term impacts on the way federal programs are managing soil and water. In the longer term, it is expected that genomics tools will lead to the identification of new organisms or DNA sequences that could be an early warning for problems in soil and water management and evidence for prescriptive recommendations to the industry. This would include tools for the prescriptions to the agriculture and forestry sector for site restoration, including for the remediation of mining and oil sites. As of today, results have already informed new prescriptions to the agricultural and forestry sectors on site restoration, for instance the remediation of mining and oil sites.^{Footnote 58}

• The industrial pilot scale project (led by NRCan) at Canadian Natural Resources Limited (CNRL) oil sands reclamation sites in Fort McMurray is advancing the genomics expertise at NRCan on examining the disturbance/effect of oil sands operation on the environment with biomonitoring at Fort McMurray. The SPP is considered as a success story within the department because the preliminary results already contributed to the identification of restoration solution of oil sand sites by "providing operational science-based guidelines for governments and corporations to accelerate the return of perturbed sites to a resilient and functioning ecosystem."
• The application study using genomics tools established in 2013 at AAFC's Harrow Research and Development Centre contributed to remediation and improved nutrient management for agricultural soils and crop practices.

Metagenomics capacity has been reported by some respondents to have influenced other programs dealing with ecosystem biodiversity and natural resources services. It is also envisioned that the UN report on ecosystem would benefit using such an approach. Eventually, the platform and bioinformatics software are envisioned to be open-source.

3.0 Interdepartmental Research Collaboration

All respondents had very positive views on the interdepartmental collaboration achieved through this SPP and felt that the same level of progress would not have been achieved as rapidly without the funding and collaborative structure of the project. Here are some quotes that illustrates the added value of interdepartmental collaboration:

"We are now collaborating with scientists that we would have not collaborated with by other means. For example, collaboration between forestry and agriculture. This allows to benchmark biodiversity across settings. This is quite new."

"The advantage of collaborating is to make available costly and specific expertise and infrastructure (DFO Lab sample treatment and NRC Saskatoon for sequencing). Sample handling and bioinformatics is done internally here in the lab, but the analyses are done by another group, we are sharing expertise."

Figure C4.3: Collaborative Network Analysis – EcoBiomics SPP (Federal DAs only)

Figure C4.3 - Text version

Web diagram of EcoBiomics SPP Collaborative Network Analysis Federal Departments only.

• AAFC and DFO have 3 co-publications
• AAFC and ECCC have 5 co-publications
• AAFC and PHAC have 3 co-publications
• AAFC and NRC have 4 co-publications
• AAFC and CFIA have 5 co-publications
• ECCC and DFO have 5 co-publications
• ECCC and NRCan have 9 co-publications
• ECCC and CFIA have 5 co-publications
• ECCC and PHAC have 1 co-publication
• PHAC and NRCan have 1 co-publication
• PHAC and NRC have 3 co-publications
• NRC and DFO have 2 co-publications
• NRC and NRCan have 8 co-publications
• NRC and CFIA have 3 co-publications
• CFIA and NRCan have 8 co-publications
• CFIA and DFO have 3 co-publications

Also, through interactions made possible with the EcoBiomics SPP, some respondents reported that spin-off projects have been developed with the same collaborators that are not funded by GRDI. Outside GRDI, these projects were reported to be highly rated particularly from the point of view interdepartmental cooperation. For example, there is now a strong collaborative relationship between AAFC, ECCC and NRCan on soil research. The collaborative network analysis provides an illustration of interdepartmental connections through co-publications. All DAs involved in the project are represented in the co-publication network. As shown in Figure C4.3, ECCC, NRCan and AAFC dominate the collaboration map for the EcoBiomics, with stronger connections between CFIA, the NRC and DFO. PHAC has fewer co-publications.

3.1 Lessons Learned and Best Practices Related to Interdepartmental Collaboration

Best Practices

The foremost best practice mentioned by almost all respondents is the interdepartmental collaboration aspects of the SSP through having different DAs coming from different mandates and sectors all working on a common goal and specific objectives (regular communications and real integration of sampling data to analysis). For many, the provision of a specific common funding and management structure for interdepartmental collaborations establishing common protocols, procedures, workflows and technology should be considered as a best practice. For many respondents, the EcoBiomics model is quite unique and has been recently published as a best practice^{Footnote 59} and should be considered in other priority fields within the federal government. Because of QIS and EcoBiomics, the Canadian federal government is now a reference for data standards and definition in environmental genomics.

Respondents familiar with SPPs from phases V and VI described improvements between the QIS SPP and EcoBiomics SPP in terms of management practices, collaboration and integration of research practices. Several interview respondents identified a paper published on the EcoBiomics SPP that provided a model for governments around the world to incorporate metagenomics into monitoring programs. For many, it is just not making sense to pursue research outside the integrated environment developed under the project. According to many interviewees, the GRDI approach/model is a best practice and should be expanded at the level of the whole government, with more funding, over a longer period of time and should be considered with the upcoming federal scientific funding.

Lessons learned

The main lesson learned highlighted by respondents were the planning and funding of the IT component from the beginning. The IT platform, while it had improved over time, is still an issue, and remains a challenge and limiting factor to project success. The management of the infrastructure by Shared Services Canada (SSC) is an ongoing issue for accessing and updating the software to keep pace with the evolving needs of the SPP. For many respondents, the real challenge is finding and retaining HQP for such SPPs. An interdepartmental Bioinformatics Working Group was established within the SPP in 2017-18 that continues to be used to facilitate discussion and sharing of experiences using bioinformatics pipelines across DAs.

Even if EcoBiomics is already starting to lead to applications, the federal government needs to support genomics for environmental applications in the long-term to see benefits, like the human genome projects that took several years to complete. Some respondents stressed that the end of funding after five years for such long-term scientific research program will create inefficiencies and lost opportunities in terms of science innovation. In addition, the nature of metagenomics and the complex implementation of the projects and the overall regulatory environment add further challenges to achieving long-term impacts, and assessing those impacts. As interviewees noted:

"Our computing infrastructure that we've had allocated within the project will not be available at the end of the project, so we're now looking at how we can make sure that stuff doesn't basically just simply disappear at the end of the SPP funding. Ecobiomics work is quite computationally intensive, and it does generate a large amount of data for storage, and obviously that needs to be maintained"

"The continuation of SPPs over time are key as it is not possible to solve the problems, finalize and transfer technology within 5 years. Having all the DAs collaborating on a priority, this priority issue established should be supported beyond a 5-year window."

Importantly, many respondents highlighted the need to better engage several end-users within and outside the federal government. Some stressed the need to have end-users involved earlier in the project both in the selection of SPP sub-themes or projects and in the planning of knowledge transfer activities. Such end-user engagement is required to maximize the uptake of scientific and technological development and to ensure continuity and leverage GRDI SPP investments after the five-year funding ends.

Appendix D – Best Practices and Lessons Learned

This appendix presents a summary of the best practices and lessons learned gleaned from the evaluation of the GRDI SPPs. These findings were supported by interviews and documents but do not represent a consensus or majority view. Some findings were only shared by one source (or one individual).

Best Practices

• Interdepartmental collaboration as a best practice in managing and conducting science within the federal government.
  • Trust between scientists and DAs builds over time.
  • Sharing of methods and data, as well as highly qualified personnel (HQPs) such as post-doctoral fellows.
  • Scientists working together on a common theme and a common goal rather than separately working towards a common goal.
  • Pushing scientists outside of their comfort zone to find commonalities, complementarities.
• Governance, structured management and coordination practices.
  • Senior-level leadership in the governance structure to provide focus and prioritization of science and investments.
  • Dedicated coordination secretariat led by one DA.
  • Communication, monitoring of progress and integration.
  • Having a common working structure and budget provide the mutual dependency among participating scientists needed for improved communication and support as a team for interdependent and critical project deliverables.
• Process used for the development and selection of SPP themes and project activities.
  • Top-bottom for common general themes and mandate to be substantiated from the bottom, had the advantage of bringing solutions forward by gathering people across DAs.
• Funding model effective for leveraging available capacities and infrastructure of participating DAs.
• Creation of documentation and training on publication guidance, IP and rules of engagement across SPPs, code of conduct/ ethics code was a really good practice and should be implemented from the beginning of SPPs and are of value for other programs across the federal government.
• Learning from previous Phases to introduce continuous improvement and efficiency gains, exploring:
  • How to leverage bioinformatics capacity developed from Phase V SPPs
  • Potential for a common bioinformatics platform tools and techniques developed in Phase V
  • Potential of incorporating NRC expertise.
• Learning from early years of SPPs and apply those lessons to later years.
  • In the case of AMR and EcoBiomics, the first Annual Performance Report mentioned lessons stemming from: staffing/capacity challenges; difficulties in tracking expenditures; delays and practical challenges encountered in sequencing projects and others; the importance of collaboration and local knowledge; communication challenges and instances of overhead, etc.

Best practices identified for specific SPPs include:

• QIS / Ecobiomics:
  • The production and transfer of standardized protocols and procedures, technology and research best practices to all participating DAs.
• FWS / AMR:
  • Defined work packages and frequent teleconference meetings lead to increased awareness and involvement of participants, and better integration of various research project activities for the benefit of the overall project goal.
  • The opportunity to respond to ADM information requests and adjust the content and delivery of presentations to different audiences.

Lessons learned

• Costs for IT infrastructure (and bioinformatics) are significant and should be expected and planned for.
• Early and robust engagement of external partners who are necessary to facilitate the research.
  • Access to a dedicated high-performance data clusters server in Dorval was identified as positive, but still an ongoing challenge.
  • The main challenge is getting Shared Services Canada to understand complexity of requirements resulting on outdated servers and bioinformatics software not matching the needs of current research.
• Process to develop themes should be started earlier and allow for sufficient back and forth so that the main goal is sufficiently narrow to facilitate the development of sub-themes and project activities.
• Expectations of participating DAs should be managed regarding potential role(s) in SPPs and focus is on what is best for Canada and the federal government as a whole.
• Early engagement of stakeholders and end-users for SPPs, both within and outside the federal government.
  • Would allow better identification of the problem, input to decision-making, improved relationship for mutual benefits and better uptake in the future.
  • Translation and transfer did not materialize as expected.
  • Possibly centralize the role of translating outputs into practice and creating the awareness within the public service of the research projects and their achievements.
• Hiring and retaining HQPs a major challenge, including: keeping the good HQP, especially in the context of undetermined positions/non-permanent projects; the mismatch between SPP projects length and post-graduate student training cycle resulting in loss in HQPs during the course of projects.
• Develop collaborations outside of the federal government, in particular, with Genome Canada and universities to better align the thematic areas, selection process and funding directed to priority issues for Canada.
  • Would allow for improved access to HQPs (post-docs) and funding capacity to offer employment for period of time longer than current SPPs funding cycle
• Succession planning needed to account for life events (e.g., retirement) of the collaborators as it can cause disruption and other issues during the course of a 5-year project.
• Lack of continuity in SPP funding after five years as a major challenge to avoid losing investments, opportunities and momentum in achieving targeted objectives and outcomes.
  • Unfinished technology transfer to the industry after five years resulted in DAs needing to fund the project completion because of industry engagement and expectations.
  • Continue funding the part(s) of the SPP(s) that has the most promise (which can deliver an impact in 2-3 years) and support the area to be continued so it doesn't get halted.
• Transition from b-base to a-base
  • Constitutes a risk for SPPs because it can be used in different ways by participating DAs.
  • Changed funding allocation timing and caused delays in the availability of funds for SPPs.
• GRDI should examine other federal and international interdepartmental initiatives to benchmark and identify best practices.

Lessons learned identified for specific SPPs include:

• QIS / Ecobiomics:
  • Lost opportunity in having QIS data not fully integrated to the new infrastructure in Dorval.
  • Opportunities for other genomics research projects in other fields of interest for the federal government (including projects that were excluded from the scope of the SPPs like toxicogenomics/virology) to benefit from standardized IT/bioinformatics platform developed under EcoBiomics.
• AMR / FWS:
  • Better communication among participating DAs is the most important lesson, by having a regular call and find better ways to communicate.
  • Staff time constraints as many participating scientists were contributing to FWS in addition to other projects. The size of research groups should be reduced because the number of scientists and expectations were too high.
  • There are opportunities for AMR network of scientists to contribute to the emerging needs nationally and internationally for risk-assessment tools using metagenomics (including the forthcoming pan-Canadian action plan related to mitigating the risks for AMR).
• Need more opportunities or a formal forum for operational management discussions among research of each work package, not just among project executives.
• Need annual face-to-face meetings where students and post-docs can present work to update.