Horizontal evaluation of the genomics R&D initiative (GRDI)

Alternative format

PDF version (2.1 MB)

 

Prepared for:
National Research Council

Prepared by:
Goss Gilroy Inc.
Management Consultants
Suite 900, 150 Metcalfe Street
Ottawa, ON K2P 1P1
Tel: (613) 230-5577
Fax: (613) 235-9592
E-mail: ggi@ggi.ca

Date:
November 15, 2016

Executive Summary

Program description

The Genomics Research and Development Initiative (GRDI) was created in 1999 and is currently implementing its Phase VI round of funding. The program's objective is to establish and maintain core genomics R&D capacity in the following federal departments and agencies (DAs):

  • Agriculture and Agri-food Canada (AAFC);
  • Environment and Climate Change Canada (ECCC);
  • Fisheries and Oceans Canada (DFO);
  • Health Canada (HC);
  • Public Health Agency of Canada (PHAC);
  • National Research Council (NRC);
  • Natural Resources Canada (NRCan); and,
  • Canadian Food Inspection Agency (CFIA) – since Phase VI.

A total of $59.7 million was invested in support of genomic R&D as part of Phase V. Research supported by GRDI seeks to uphold regulatory, public policy, and operational mandates in important areas such as health, food safety, sound management of natural resources, a sustainable and competitive agriculture sector, and environmental protection, with strong collaborations with university and private sectors. In Phase V (2011-14), GRDI supported 73 of these individual DA-led mandated research projects. As well, the program funded two shared priority projects (SPPs) between January 2012 and March 2016: Food and water safety in Canada through an integrated federal genomics initiative (also referred to as the Food and Water Safety or FWS project); and Protection of Canadian biodiversity and trade from the impacts of global change through improved ability to monitor invasive alien and quarantine species (also referred to as the Quarantine and Invasive Species or QIS project).

There are two types of end-users for GRDI-funded projects: internal and external end-users. Internal end-users are the most common type for GRDI-funded projects and are those inside federal government (e.g., scientists in the labs, inspectors on the ground, border agents, trade negotiators). External end-users are those outside federal government (e.g., industry using a patented technology, industry changing their processes due to policy change, international regulatory agencies using/adopting the technology).

Scope and methodology

GRDI foundational documents for Phase VI funding require a horizontal evaluation of the program in 2015-16. The evaluation focused on Phase V of GRDI, which spanned fiscal years 2011-12 to 2013-14. However, fiscal years 2014-15 and 2015-16 were included insofar as work associated with SPPs continued to be conducted during these years. The evaluation addressed nine questions in three issue areas: relevance; performance- effectiveness; and performance – efficiency and economy. The methodology employed multiple lines of evidence, including both qualitative and quantitative and relying on both primary and secondary sources of data. The main limitations of the methodology included: validity and reliability of administrative data; and a likely underestimation of the extent of collaborations in SPPs assessed through the bibliometric study.

Overall evaluation findings

The evaluation found that, while the needs of DAs for genomics R&D funding have evolved, there is a continued need for GRDI. For many DAs, the need continues to relate to capacity building (in both basic and advanced techniques). Several DAs have also seen the need move towards the application of genomics-based technologies and approaches. In addition, there is clear alignment of GRDI with the mandates and priorities of participating DAs and the federal government more generally. The evaluation confirmed that the program is consistent with, and contributes to, participating DAs' legislated mandates related to the health and safety of Canadians as well as the sustainability of Canada's natural resources (via regulatory activities) and support for industry (via economic development and regulatory activities). The evaluation further found that there is little duplication with other similar organizations or federal programs.

In spite of large amounts of funds being leveraged and some DAs making investments in genomics R&D, it is clear that the need for continued support for these types of projects does not show signs of abating. In the absence of GRDI, the evaluation found that DAs' abilities to deliver on their mandates in the future might be negatively impacted as it is unclear what amount of DA funding for genomics R&D would be available given other DA priorities.

The continued need for GRDI is taking place within a context of a decreasing real value of the funding (due to inflation since inception in 1999 without commensurate increases in funding), funding being shared among more participating DAs, SPPs vying for part of the GRDI funding envelope and increasing costs associated with the research. Research costs are increasing because more effort is now being focused on applications and transferring knowledge to end-users, which are more resource intensive phases of R&D.

In terms of effectiveness, the evaluation found that Phase V projects have been successful in the development of innovative knowledge and technologies and influencing evidence-based public policy. The program has produced expected outputs and exceeded targets related to these. Moreover, the evaluation confirmed that many GRDI projects have successfully transferred knowledge and technologies to end-users, both internal and external to the federal government. Many more GRDI projects have the potential to similarly impact end-users. Nevertheless, there continue to be opportunities, especially for the SPPs, to build on the lessons and best practices from Phase V in subsequent phases of the program. Some of these have already been addressed. In particular, content outlined in the Innovation Management Strategy (IMS), though intended for SPPs, offers examples and tools to ensure end-user engagement and take-up of GRDI-funded knowledge and technologies that could be considered beyond the SPPs.

In addition to the transfer of knowledge and technologies, the evaluation found evidence that GRDI-funded projects are likely to have real and lasting longer-term impacts, most of which remain anticipated or potential impacts at this time. These impacts are significant and are expected to lead to billions of dollars in benefits for end-users (through avoided costs, increased trade, etc.). More specifically, the evaluation found that GRDI-funded projects are likely to result in: improved public health and wellness; avoided health system costs; efficiencies and avoided losses for the public and private sectors; environmental sustainability; improved detection of invasive species; improved strains/traits of plants, trees and animals of commercial value; and improved fisheries tracking and management.

The introduction of interdepartmental SPPs in Phase V has proven to be a strong feature of the program and the evaluation found that there is a continued need for this type of interdepartmental collaboration. The two SPPs funded in Phase V were found to be well-managed and have achieved significant results, which likely would not have occurred in the absence of GRDI funding for the interdepartmental nature of the projects. As well, the degree of interdepartmental collaboration stemming from the SPPs has been significant. However, there is no consistent or planned approach related to what will happen to ongoing SPPs when the GRDI funding ends. Consequently, there is a risk that the results achieved to date and the potential for further impacts might be slowed or otherwise diminished. Some concerns were also raised about the selection of SPPs (including the role of scientists), and a review of related documents confirmed some lack of transparency in terms of how sub-projects are selected. As well, since sub-projects are ultimately directed by the Scientific Coordinator from one participating DA (with support from Sub-Project Leads), there is the potential for the perception of conflict of interest. Finally, the QIS case study suggests that mid-year reporting may be overly burdensome (on top of annual reporting). Both case studies brought the suggestion that there should be more interaction between the various scientists contributing to the SPP goals.

Other challenges associated with the SPPs included those pertaining to: end-user engagement; sharing of data/materials and building bioinformatics capacity; management of IP ; a short timeframe for projects of this magnitude; interdepartmental financial and administrative issues; and delays associated with the hiring of highly qualified personnel. The IMS was developed with a focus on the first three challenges in particular (among other matters), and is being applied during Phase VI SPPs.

In terms of efficiency and effectiveness, the evaluation found that the program is both efficient (in terms of its horizontal governance) and effective (in terms of its use of collaborations, leveraging of funds and delivery approach). The evaluation did not reveal any significant opportunities for improvement in program governance or delivery. As well, the evaluation found that the program has made improvements in terms of its performance measurement approaches and systems. However, a few challenges remain, including reporting inconsistencies and the lack of mechanisms to compile and/or track supporting evidence by project.

Recommendations and Management Response

Recommendations Management response
1. The ADM CC should consider exploring and formally defining how SPP sub-projects are selected (including how and when the input of scientists is considered). Part of this selection process should consider and evaluate the potential for conflict of interest during the approval process as well as the input of scientists.

Accepted.

The GRDI will update its Governance Framework and articulate the SPP planning process more clearly, particularly vis-à-vis sub-project development and selection.

2. The ADM CC should consider requiring SPPs funded in Phase VI and beyond to develop a plan exploring how the technology/knowledge developed during the project will be transferred to, and used by, end-users outlining follow-on work that should be conducted by the partners to ensure such transfer and identifying potential funding sources to maximize uptake.

Accepted.

SPPs will include a technology/knowledge transfer plan that identifies clients and end users who will be engaged throughout the project.

SPP teams will periodically review and update the initial technology/knowledge plan in order to maximize uptake. At the end of the project funding period, GRDI will identify follow-on work required to fully implement the plan, identifying potential resources and fostering external linkages as appropriate

3. The ADM CC should consider and implement additional opportunities to increase communication between DAs participating in SPPs to improve collaboration and joint problem solving, for example by supporting interactive mid-year sessions.

Accepted.

Internal communications plans to facilitate collaboration and joint problem solving will be required in SPP Management Plans. GRDI will support additional interactive team sessions

4. The ADM CC should explore additional opportunities to improve performance measurement and reporting, including: Accepted.
  • Ongoing monitoring on Phase V SPPs (and the next evaluation should also explore what has happened with these projects). SPP leads should be consulted regarding what would be the most meaningful set of indicators to track that are manageable in terms of reporting burden
In consultation with Phase V and Phase VI SPP leads, GRDI will develop meaningful methodologies for examining the ongoing legacy and impact of SPPs. SPPs will be required to identify end-users and stakeholders that have benefited, or could benefit, from project outputs for follow-up assessments. An adoption and impact study of Phase V SPPs will be undertaken as part of the next GRDI evaluation.
  • Implementing a database to capture information by project, which would allow for searching and analysis by key project characteristics (such as project type, impact area, DA, GRDI funding phase, GRDI funding, DA-leveraged funding, HQP , etc.) and would encourage projects with similar objectives to communicate with one another. This database should be accompanied by a detailed definition of variables and indicators to be reported.
GRDI will define key project characteristics to be reported, and develop a database which can be used to foster interdepartmental communication and support reporting activities.
  • Streamlining SPP reporting, for example by revisiting the number of indicators to be reported in SPP mid-year reports. Interactive mid-year sessions rather than detailed mid-year reports, and revisiting the number of indicators, are examples of opportunities for streamlining and improvements in communications.
GRDI will streamline SPP mid-year reporting to the ADM CC and will revisit the variables and indicators reported annually for mandated and SPPs.
 

1.0 Introduction

Goss Gilroy Inc. (GGI) is pleased to submit this revised final report that presents the results of the horizontal evaluation of the Genomics Research and Development Initiative (hereafter referred to as GRDI or "the program").

1.1 Overview of GRDI

GRDI was created in 1999 to establish and maintain core genomics R&D capacity in the following federal departments and agencies (DAs):

  • Agriculture and Agri-food Canada (AAFC);
  • Environment and Climate Change Canada (ECCC);
  • Fisheries and Oceans Canada (DFO);
  • Health Canada (HC);
  • Public Health Agency of Canada (PHAC);
  • National Research Council (NRC);
  • Natural Resources Canada (NRCan); and,
  • Canadian Food Inspection Agency (CFIA) – since Phase VI.

Research supported by GRDI seeks to uphold regulatory, public policy, and operational mandates in important areas such as health, food safety, sound management of natural resources, a sustainable and competitive agriculture sector, and environmental protection, with strong collaborations with university and private sectors. In Phase V (2011-14), GRDI supported 73 of these individual DA-led mandated research projects. Based on a recommendation from the last evaluation of GRDI, Phase V introduced a new model that included the mobilization of resources for concerted research on issues that are beyond the mandates of single DAs. This model supported the funding of two highly coordinated interdepartmental projects along shared priorities and common goals, referred to as shared priority projects (SPPs). These SPPs took place from January 2012 to March 2016.

  1. Food and water safety in Canada through an integrated federal genomics initiative (also referred to as the Food and Water Safety or FWS project): Aimed to improve the ability to detect, diagnose and monitor organisms to ensure a sustainable supply of safe and healthy food and water for human consumption; and
  2. Protection of Canadian biodiversity and trade from the impacts of global change through improved ability to monitor invasive alien and quarantine species (also referred to as the Quarantine and Invasive Species or QIS project): Aimed to improve ability to detect, identify and understand Canadian biological diversity to prepare Canadian natural and managed resources and markets for global change.

There are two types of end-users for GRDI-funded projects: internal and external end-users. Internal end-users are the most common type for GRDI-funded projects and are those inside federal government (e.g., scientists in the labs, inspectors on the ground, border agents, trade negotiators). External end-users are those outside federal government (e.g., industry using a patented technology, industry changing their processes due to policy change, international regulatory agencies using/adopting the technology).

1.1.1 Financial Resource Profile

A total of $59.7 million was invested in support of genomic R&D as part of Phase V. Table 1 shows the actual spending by participating DAs on mandated research projects. Table 2 shows the spending by participating DAs on SPPs.

Table 1: Actual program expenditures, by fiscal year, by DA ( $000s )Table note *
GRDI Participating DAs FY 2011-12 FY 2012-13 FY 2013-14 Total %
AAFC 5,700 4,800 4,800 15,300 30%
CFIA - - - - -
DFO 855 720 720 2,295 4%
ECCC 950 800 800 2,550 5%
HC 1,900 1,600 1,600 5,100 10%
NRC 5,966 4,800 4,800 15,566 30%
NRCan 1,900 1,600 1,600 5,100 10%
PHAC 1,900 1,600 1,600 5,100 10%
Coordination and common functions N/A 313 314 627 1%
Total 19,171 16,233 16,234 51,638 100%
 

Source: GRDI Administrative Data – Fiscal Years 2011-12, 2012-13, 2013-14

Table 1 notes

Table note *

Note: Some numbers may differ slightly from figures in NRC Departmental Performance Reports as a result of differences due to report timing, data coding and/or categorization.

Return to footnote * referrer

Table 2: Actual program expenditures for SPPs, by fiscal year, by DA ( $000s )
GRDI Participating DAs FY 2011-12 FY 2012-13 FY 2013-14 FY 2014-15 FY 2015-16 Total %
QIS
AAFCTable note * 169 726 707 775 773 3,149 40%
CFIA 72 319 365 347 331 1,434 18%
DFO 67 280 284 275 281 1,187 15%
ECCC 29 129 108 135 163 564 7%
HC - - - - - - -
NRC 53 342 261 270 250 1,176 15%
NRCan 8 66 129 107 95 405 5%
PHAC - - - - - - -
Total – QIS 399 1,862 1,854 1,910 1,893 7,917 100%
FWS
AAFC 10 127 160 106 103 507 7%
CFIA 23 198 225 232 215 893 12%
DFO - - - - - - -
ECCC 5 85 66 52 52 259 3%
HC 7 109 117 85 77 395 5%
NRC 97 501 480 549 542 2,170 29%
NRCan - - - - - - -
PHAC 185 785 762 818 795 3,344 44%
Total – FWS 327 1,805 1,810 1,842 1,784 7,568 100%
Total 726 3,667 3,664 3,752 3,677 15,485 -
 

Source: GRDI Secretariat

Table 2 notes

Table note *

Note: 50% of the GRDI funding allocated to AAFC was for the shared bioinformatics platform.

Return to footnote * referrer

 

In addition to these investments, GRDI performance reports for Phase V indicate that an estimated total of $89.4 million in non-GRDI funding was leveraged from A-base and other sources (e.g., external collaborators). This amount includes cash and in-kind contributions. Please refer to Table 3 in Section 2.1, below.

1.1.2 Program Activities

GRDI investments were used primarily to support R&D and dissemination activities for 73 mandated research projects and two SPPs. The remaining funds were used to support the horizontal coordination of the program (i.e., the GRDI Secretariat), as well as other activities related to the administration of the program.

1.1.3 GRDI Horizontal Governance

An interdepartmental Assistant Deputy Minister Coordinating Committee (ADM CC), chaired by NRC, is responsible for the overall strategic direction of GRDI. It ensures that effective priority setting mechanisms are established within departments and agencies and that government objectives and priorities are addressed. For Phase V, the committee includes members from each of the organizations receiving funding, and guest representatives from Industry, Science and Economic Development (ISED) Canada and Genome Canada. Note that CFIA did not receive funding for mandated research prior to Phase VI but did have a representative on the GRDI ADM CC and Working Group in Phase V and earlier.

An interdepartmental GRDI Working Group (WG), also chaired by NRC, supports the work of the ADM CC. The mandate of the WG is to provide recommendations and advice to the ADM CC regarding strategic priority setting and overall management of GRDI. The WG is responsible for providing direction to GRDI program activities related to operational delivery, implementation planning and investment priority setting. The WG also supports evaluation and reporting requirements related to the Initiative.

The horizontal management of the initiative is supported by the GRDI Secretariat, including support to the ADM CC and the GRDI WG and communication to departments of the planning cycle, process requirements, financial administration and other project management requirements. The GRDI Secretariat is also responsible for facilitating the SPP planning and peer review processes; ensuring project management plans and funding agreements are in place; and supporting performance management, reporting, evaluation, and communications.

1.2 Objective of the Evaluation

Since GRDI's implementation, two external evaluations (2006 and 2010) have been conducted. These evaluations have confirmed the relevance and continued need for an initiative that supports genomics R&D at a federal level as well as the need for the program to be managed horizontallyFootnote 1. Recommendations from those evaluations have been applied towards the governance of GRDI and progression of subsequent GRDI research projects.

GRDI foundational documents for Phase VI funding require a horizontal evaluation of the program in 2015-16. The horizontal evaluation has the following main objectives:

  • Examine the overall initiative with a focus on SPPs and horizontal governance;
  • Identify the impacts of GRDI projects on end-users; and
  • Assess the core issues of relevance and performance of the program (according to the 2009 TB Policy on Evaluation's core evaluation issues).

The evaluation focused on Phase V of GRDI, which spanned fiscal years 2011-12 to 2013-14. However, fiscal years 2014-15 and 2015-16 were included insofar as work associated with SPPs continued to be conducted during these years. The evaluation addressed the following nine questions in three issue areas:

Relevance:

  1. How have the needs of federal departments / agencies (D/A) for GRDI funding evolved since its inception and to what extent is the Initiative still aligned with current needs? To what extent is there a continued need for interdepartmental research collaboration (i.e., shared priority projects)?
  2. To what extent do the activities and outcomes of GRDI align with the priorities of the federal government and the priorities of partner departments and agencies?
  3. To what extent is GRDI consistent with federal roles and responsibilities?

Performance – Effectiveness

  1. To what extent have the research projects funded as part of Phase V contributed to the development of evidence based public policy and innovative knowledge / technologies? To what extent and under what conditions were this knowledge and these technologies transferred to end-users inside and outside of the federal government?
  2. What impacts have been generated from the use of GRDI's research results by end-users and external stakeholders? What success / hindrance factors have influenced the use of knowledge and technologies by end-users and external collaborators? To what extent would those research results have occurred without the funding from GRDI?
  3. To what extent have the GRDI partners been successful at selecting, managing, collaborating, and achieving results as part of interdepartmental research projects?
  4. What lessons learned and best practices can be identified in terms of interdepartmental research collaboration based on the two projects funded as part of Phase V?

Performance – Efficiency and Economy

  1. To what extent have the horizontal governance components of GRDI (i.e., ADM CC, Interdepartmental Working Group, GRDI Secretariat) been efficient?
  2. To what extent does the current delivery model for GRDI allow for a cost-effective use of federal government resources? Are there alternatives that are more cost-efficient?

Please refer to Appendix A for the evaluation matrix, which outlines the evaluation questions, judgment criteria and the associated indicators and methods.

The evaluation was managed by the Evaluation Division at NRC (hereafter referred to as the project authority) and overseen by an Interdepartmental Evaluation Working Group (IEWG) comprised of representatives of each participating DA with a total of 18 members.

1.3 Methodology

This section describes the various methods used to answer the evaluation questions.

1.3.1 Document Review and Literature Review

The project authority (PA) provided GGI with pertinent documents for the document review. Documents that were included were prepared by the GRDI Secretariat for the administration of Phase V, as well as various other GRDI internal documents. Please refer to Appendix B for a list of documents reviewed for the evaluation.

The document review also included findings from a collaboration study using bibliometric methods conducted by NRC Knowledge Management. GGI conducted a web search of key sources. In addition, literature produced by Genome Canada and the regional Genome centres (e.g. Genome Canada Five-Year Evaluation, 2014, annual reports, etc.) were considered. Once GGI identified sources for the literature review, we assessed each article for relevance and short listed 15 to 20 sources which were shared with and approved by the PA (see Appendix C for an annotated list of these sources). GGI then proceeded to review each article and extract relevant evidence to support the various evaluation questions identified for this work.

GGI prepared a technical report of the literature and documents reviewed, which was reviewed by the IEWG.

1.3.2 Administrative Data Review

GGI used a number of data sources for this analysis, including:

  • Three years of GRDI annual performance reports (APRs) for Phase V (2011-12, 2012-13 and 2013-14), which include a number of quantitative figures relating to resources, expenditures and performance;
  • GRDI funding amounts for SPP projects, by DA for 2011-12 to 2015-16; and
  • Performance data for indicators outlined in the performance measurement strategy (PMS), by DA for 2011-12 to 2013-14;

Based on this information, an analysis plan was developed that explained how the data would be used to answer the evaluation questions and indicators outlined in the evaluation matrix. Gaps and potential challenges were also identified.

The information was then organized into tables and presented in a technical report. Where possible, administrative data was presented against the targets identified in the program's PMS.

GGI prepared a technical report of the administrative data analysis, which was reviewed by the IEWG.

1.3.3 Key Informant Interviews

A list of 29 names was provided to GGI at the beginning of the evaluation. This list of potential key informant respondents was mainly composed of federal government employees across the eight participating DAs, including senior management, program managers and directors, and scientists. One external respondent was identified (from Genome Canada).

Four interview guides were developed, one for each key informant respondent type. The final guides (in English) are included in Appendix D. Once an interview was scheduled, respondents received the guide in advance and confirmation by email of the appointment. Interviews were conducted in-person or by phone and in the official language of choice of the respondent. The interviews lasted between 45 and 90 minutes each (depending on the respondent type and their availability). In some cases, a member of the IEWG attended the interview in order to hear the views of respondents first hand.

Of the 29 names provided to GGI, 26 interviews were conducted (covering 27 individuals), as follows:

  • Senior management (n=7);
  • Program managers and directors (n=14);
  • Scientists (n=4); and
  • External respondent (n=1).

Several names on the initial list were replaced due to unavailability and lack of awareness of the program. Of the three that were not interviewed, one individual declined to participate, one individual was unavailable and no replacement could be found and one individual was interviewed with another person on the list (thereby considered one interview rather than two).

GGI prepared a technical report of the interview results, which was reviewed by the IEWG.

1.3.4 Case Studies

A total of 13 case studies of mandated research projects funded by GRDI during Phase V (two projects from each of the seven DAs with mandated research projects with the exception of NRC where only one case study could be completed due to the availability of respondents) were conducted as part of the evaluation of the program. The case studies focused on the impacts/progress of the projects, including impacts on end-users, in addition to the identification of the facilitating/hindering factors to the transfer of knowledge to end-users.

The cases were selected by members of the IEWG in concert with program representatives. The primary focus of the selection was to identify projects that were either completed or sufficiently advanced to ensure that the evaluation could assess results and to ensure that they reflected the diversity of GRDI in terms of outcomes and long-term impactsFootnote 2.

In addition to case studies for mandated research projects, GGI conducted in-depth case studies for the two SPPs.

Each case study included:

  • A review of documents, including the project proposals, the APRs, and other documentation available (such as other studies conducted on the project, success story summaries conducted by the GRDI Secretariat).
  • Interviews with the principal investigator(s), scientists, partners and/or end-users. For the mandated projects, between 2 and 5 interviews were conducted whereas for the SPPs, 9 interviews were conducted for each case study. See Appendix E for the English versions of the interview guides.

In total, 45 interviews were conducted as part of the mandated project case studies, of which 26 could speak to the end-user perspective. For the SPP case studies, GGI conducted a total of 18 interviews, of which six had an end-user perspective.

GGI prepared a case study narrative for all 15 case studies conducted for the evaluation, and these were validated by the principal investigator(s) and reviewed by the IEWG.

1.4 Limitations to the Approach

As with all evaluations, there were some limitations experienced in the implementation of the methodology. Wherever possible, to alleviate the effects of the limitations and challenges on the evaluation findings, various mitigation strategies were used. For example, the evaluation featured a mix of four methodologies that produced both qualitative and quantitative evidence and used both primary and secondary data sources. With four methodologies, all evaluation questions benefited from multiple lines of evidence and the analysis featured the triangulation of evidence to arrive at overall findings. The main challenges experienced included:

  • Interviews as a line of evidence and potential response bias of interviewees: Interview evidence is based on personal perceptions of a select group of interviewees. As with all evaluations, the number of interviews conducted had to be limited, in accordance with the project scope and timeline. Further, time limitations meant that in some cases, not all interview questions were asked of every interviewee – in order to explore in more detail other questions. As a mitigation strategy, interview results were verified against findings from other lines of evidence.
  • Limitations relating to the documents reviewed: A significant portion of the documents received were dated 2009 (i.e., pre-Phase V). In addition, a number of the documents were generated by participating DAs which, because the documents were prepared by potential beneficiaries of the program, these documents may not present a completely unbiased picture when it comes to assessing GRDI's success and ability to achieve results. This limitation was offset in part by the administrative data review, which assessed progress against targets as well as by the case studies, which included some interviews with end-users outside of the program.
  • Performance information limitations: Potential limitations related to the validity and reliability of information presented in the APRs were identified, including:
    • Overall, the PMS and its companion document provide general guidance to departments and agencies, some definitions of indicators, as well as some examples of outputs. However, not all indicators are explicitly defined in these documents and there is a risk that different departments and agencies did not interpret indicators the same way. As a result there is a risk of inconsistent reporting by DAs, which could call into question the accuracy of the performance information. A similar situation could arise when identifying leveraged funds since the way in-kind contributions are calculated can be very different among DAs, for example.
    • The performance indicators related to staffing and collaborations were ongoing from year to year. Targets for these indicators were established at the Phase level as per the PMS and are reported as such in this report. Indicators differentiating between existing and new instances of collaboration with existing and new staff, for example (either by year or for the Phase overall), were not available, limiting what could be reported regarding these outputs.
    • There is no single place to get information about individual funded projects, such as a project database. Rather, information about individual funded projects was available in various sources at various levels of detail. While the indicators tracked for GRDI (such as numbers of collaborations, publications and so on) are available in an Excel spreadsheet, they could not be attributed to individual projects, DAs, funding amounts, sectors, etc.
  • Limitations relating to the bibliometric study: The bibliometric study that explored the degree of collaboration between participating DAs in SPPs had a few limitations.
    • First, the analysis only examined one indicator of the extent and breadth of DA collaborations (i.e., peer-reviewed co-publications created in the QIS and FWS projects). If considered in isolation the indicator likely underestimates the extent of DA collaborations.
    • Second, the nature of DAs' activities and operating environments affect the degree of scientific publications that they can produce. For example, NRC's collaborative work with industry is not appropriate for publication due to the proprietary nature of some these collaborations. Additionally, regulatory DAs conducting related scientific activities may not produce outputs that are amenable to scientific publication as is more likely with those DAs that are primarily conducting R&D.
    • Third, the scope of the bibliometric study did not include the full five-years of the SPPs since both projects ended in March 2016 and thus continued to produce publications. Therefore, the statistics for the SPPs underestimate the level of impact.

1.5 Development of the Final Report

As mentioned, technical reports were developed for each methodology. GGI then prepared a preliminary findings presentation, organized by evaluation question, and presented these preliminary results and recommendations to the IEWG. Following that meeting, GGI prepared this final report. Please note that where qualitative evidence is presented, the following scale has been used:

  • "Large majority" – findings reflect the views and opinions of at least 75% of respondents;
  • "Most" – findings reflect the views and opinions of at least 50% but less than 75% of respondents;
  • "Half" – findings reflect the views and opinions of 50% of respondents;
  • "Some" - findings reflect the views and opinions of at least 25% but less than 50% of respondents; and
  • "A few" - findings reflect the views and opinions of at least two respondents but less than 25% of respondents.

It should be noted that when a minority view is mentioned, it does not necessarily mean that a majority disagrees with the view. For example, the report may mention that "a few respondents said (…)" – this does not necessarily imply that the other respondents did not agree with the statement. They may have just not expressed a view about the issue. Generally, the report focuses on the majority of views expressed.

2.0 Findings

2.1 Relevance

EQ 1: How have the needs of federal departments / agencies (DAs) for GRDI funding evolved since its inception and to what extent is the Initiative still aligned with current needs? To what extent is there a continued need for interdepartmental research collaboration (i.e., shared priority projects)?

Summary: The needs of DAs have evolved away from initial capacity-building towards the application of genomics-based technologies and approaches. According to evidence from the interviews and case studies, some DAs still conduct projects to build capacity and most DAs are beginning to use genomics as an emerging tool in their day-to-day operations.

The evaluation evidence supports a continued need for the program and for genomics R&D. The program is consistent with the current needs of DAs and projects are aligned with the mandates of participating DAs. This is evidenced by how DAs position genomics R&D within their DA priorities, as well as by the extent of DA investments in genomics R&D for both GRDI and non-GRDI funded projects. The administrative data reveals that leveraging is occurring to a larger extent over time, likely driven by the increasing needs of DAs for genomics-based technologies and approaches within the context of a decreasing real value of the program, funding being shared among more participating DAs and increasing costs associated with the research.

In the absence of the program, DAs would most likely continue to deliver upon their mandates using existing technologies and approaches that, without genomics-based innovations, will quickly become slower, more costly and inefficient. Thus, without GRDI, DAs' ability to deliver on their mandate in the future might be limited. While there is a high degree of leveraging, it is unclear (since DAs were not asked to provide this information) how much DA funding for genomics R&D would be available given other departmental priorities.

There is evidence that there is a continued need for interdepartmental collaboration, including as part of the governance model, as well as through the funding of SPPs.

 
Continuing need for genomics R&D

GRDI was established in 1999 with the objective of building and maintaining a foundation for genomics R&D capacity in federal DAs. Research supported by the initiative aims to uphold regulatory, public policy, and operational mandates in areas that are important to Canadians, focusing on the roles of federal government research in important areas such as health care, food safety, sustainable management of natural resources, a sustainable and competitive agriculture sector, and environmental protection, with strong collaborations with universities and the private sectorFootnote 3.

The 2012 audit conducted by the Office of the Comptroller General confirmed the program's relevance and the continued need for a horizontally managed initiative supporting genomics R&D across participating federal DAsFootnote 4. The program also aims to benefit selected stakeholders outside the federal government (such as industry stakeholders in the development of vaccines, suppliers of food, agricultural and forestry products and those engaged in international trade). The need for genomics research (and applications) exists outside of the federal government as well. The literature review found that Canadian businesses (both private and public) are looking to genomics as a means to address some of their most demanding challenges and are recognizing that genomics can provide them with a universally competitive edgeFootnote 5. For example, as technology continues to rapidly advance, businesses are looking for "the tools of genomics to allow data to be generated faster and cheaper by orders of magnitude."Footnote 6

Evolution of needs

Interview respondents of all types generally agreed that their DAs' needs have evolved since the program's inception. The evolution of the program was most often characterized to be moving from R&D capacity building to application of new knowledge and new technologies. Since the program's implementation, many participating DAs have developed high-impact genomics R&D capacity. Through the funding and conduct of projects, GRDI has played an important role in positioning genomics researchers and contributing to areas that are important to the well-being of Canadians. GRDI's APRs indicated that GRDI's and Canada's capacities and expertise in genomics are supporting funded DAs to address the key needs of their sector and apply high quality, genomics-based R&D solutionsFootnote 7. For example, at NRC, GRDI investments support "programs requiring genomics-related activities to help industry and government tackle strategic national priorities through mission-oriented research and technology deployment."Footnote 8 At PHAC, GRDI projects use "genomics approaches to generate leading-edge knowledge to inform public health decisions and to develop innovative tools in response to the public health needs of the federal government and of our provincial partners."Footnote 9

In some cases, capacity is still being developed. Capacity can vary between and within DAs and is often driven by the fact that genomics R&D is constantly evolving and changing based on emerging approaches and needs. For example, while Health Canada has built a strong genomics R&D capacity, one of the goals of a mandated case study project was to build capacity in the areas of next generation sequencing and establishing core bioinformatics capacity. But in most instances, the ongoing need for GRDI funding was described as applying newly-developed technologies and developing new technologies. The application of genomics technologies is often occurring in parallel with existing traditional approaches until the new genomics-based approaches are beyond the proof-of-concept phase and fully verified (e.g., requiring the deployment of two systems while genomics is in the pilot stage). However, according to the majority of respondents and documents reviewed for the evaluation, there is a great deal of promise that these genomics-based approaches will help DAs better meet their mandates with more cost-effective, timely and/or accurate information and data.

Alignment with current DA needs

The results from all lines of evidence confirm that the program is aligned with the current needs of DAs. Participating DAs frame their GRDI program within their existing program areas which are aligned with their respective strategic outcomes, programs, and sub-programs defined in the Program Alignment Architecture of each involved organization. GRDI developed business case documents specifying their proposed areas of focus. As a result, the calls for proposals for the mandated research projects were focused on areas that reflect corporate mandates, priorities, research strategies, as well as balanced research portfolio approachesFootnote 10. Additionally, many of the participating DA Departmental Performance Reports (DPRs) make specific reference to genomics R&D being undertaken by the organization. Most interview respondents indicated that the program is still aligned with their DA's and the federal government's current needs. As well, there was general agreement among those consulted for interviews and case studies (including senior management, program managers and directors, and scientists) that there is still a need for genomics R&D funding for federal scientists.

Degree of leveraging

The degree of leveraging is another indicator of need for funding for genomics R&D since, if a DA is willing to invest their own funds in genomic R&D, this suggests they see a need for this type of investment. The evaluation found that there has been a high degree of leveraging for GRDI-funded projects, as well as DA-specific investments in genomics R&D.

The total leveraged funds, $89,364,000 exceeded the funding allocation for the program by a factor of 1.5 (see Table 3). Leveraged funding includes A-base funds and funds from other external sources in the form of grants, fee for service contracts, leveraged funding from collaborators etc. Overall, leveraged funds accounted for 60% of total investment under Phase V. Moreover, leveraged funding has been increasing over time, accounting for 50% of the total DA and GRDI investments in 2011-12, to 62% in 2012-13, to 65% in 2013-14.

Table 3: Estimated leveraged funding (A-base and other sources, $000 )
  FY 2011-12 FY 2012-13 FY 2013-14 Total GRDI investment Total DA investment ( incl. GRDI) % of GRDI funds to total investment
AAFC 1,373 3,223 9,249 13,845 15,300 90.5%
DFO 1,493 1,626 1,273 4,392 2,295 191.4%
ECCC 1,360 2,090 1,867 5,317 2,550 208.5%
HC 1,805 1,409 1,433 4,647 5,100 91.1%
NRC 6,241 10,089 9,798 26,128 15,566 167.9%
NRCan 2,815 2,815 2,853 8,483 5,100 166.3%
PHAC 3,914 3,954 4,165 12,033 5,100 235.9%
QIS 626 3,037 3,062 6,725 4,115 163.3%
FWS 39 3,613 4,015 7,667 3,945 194.3%
Coordination and common functions N/A 87 40 127 627 20.3%
Total 19,666 31,943 37,755 89,364 59,700 149.7%

Source: GRDI Administrative Data – Fiscal Years 2011-12, 2012-13, 2013-14

While GRDI has been instrumental in establishing genomics programs across DAs, in all cases the demand for funding has exceeded the level of available funds. In 2013 an ad hoc report was developed as part of the Management Response and Action Plan to address recommendations stemming from the 2010 evaluation. The report was prepared for the GRDI ADM CC with the objective of determining the materiality of GRDI and the relative importance of genomics in DA R&D activities. Overall, the ad hoc report found that the proportion of genomics R&D investments between GRDI and non-GRDI projects varied greatly between DAs. Some DAs, such as ECCC, only conducted genomics R&D under GRDI. In comparison, NRC invested $29.2 million in genomics R&D, the highest of all DAs.Footnote 11

Further, the report reveals that in 2011-12, "GRDI funds represented 19.7% of the total funding allocated to genomics R&D from all sources (A-base and other sources). However, because these funds were used to leverage A-base funds and other resources, GRDI projects represented about half (46.3%) of the total genomics R&D projects conducted in departments/agencies."Footnote 12 In terms of other investments since the implementation of GRDI, the demand for genomics R&D over the past few years has increased, as evidenced by the number of projects funded by external sources including matching investments from external stakeholders.Footnote 13

Although GRDI remains an important source of funding for participating DAs, the document review found that other sources of funding (e.g., DA A-base funds) are playing an increasingly important role in supporting DAs to achieve their mandates (where GRDI funds were insufficient to do so)Footnote 14. Thus, GRDI funds for genomics R&D is increasingly not meeting the needs of DAs, which are compensating with A-base funding.

These leveraged and additional investments are being made within a program context whereby GRDI funding has remained constant since the program's inception. Using the annual inflation rate since 2000Footnote 15, the real value of the $19.9 million annual investment has declined to $13.36 million in 2015. Moreover, there is less GRDI funding available for mandated projects for participating DAs due to a few factors, including: the use of funds for SPPs; the inclusion of a new partner (CFIA) as part of Phase VI (which affected two DAs); and the fact that more GRDI-funded projects focus on the application of new technologies and take-up by end-users which, according to respondents, is often more expensive than basic research since there is a greater involvement of end-users (involving meetings, consultations and the development of documentation targeted to end-users) and the nature of the work (involving prototype development, field tests, etc.).

Impact of the absence of the program

In the absence of GRDI funding, most respondents of all types felt that their DA could still achieve its core mandate related to the day-to-day delivery of services and regulatory work. However, while some genomics R&D would continue to occur in most cases, it would be on a smaller scale and there would be less innovative research conducted and much less interdepartmental collaboration. Some respondents said that their DA would continue to achieve its mandate by using the existing (described as older, slower, more costly, less effective and less efficient) methods. Also, some interview and case study respondents felt that their DA would be less prepared for future requirements dictated by their mandate (such as new policies based on emerging issues such as new and/or evolving species of pests). Finally, a few managers mentioned that Canada's participation on the world stage and the country's ability to stay current with international requirements/trends (including international regulations) would suffer if GRDI funding were to end (if, for example, Canada was unable to use the proper technology/tools).

An analysis of the 2009 DA business cases for Phase V projects found that many noted that not moving forward or failure to renew GRDI would decrease participating DAs' capacities and abilities to meet the emerging needs of Canadians in key areas of their mandates. This message was relayed again by most interview and case study respondents seven years later as part of this evaluation. According to these business cases, all participating DAs associated potential foregone costs with not receiving funding under the program. For instance, NRC felt that a lack of support for genomics would reduce, and in some cases, even eliminate the ability to build large-scale research teams that have been critical in creating technologies and supporting Canadian industryFootnote 16. For ECCC, without the funding received by GRDI, the Strategic Technology Applications of Genomics in the Environment (STAGE) program would cease to exist, meaning they would have to rely on outdated technologies that would no longer be considered cutting edge.Footnote 17

Many participating DAs make significant investments in genomics R&D in addition to the funding they receive from GRDI. Thus, for these DAs, the impact of having less or no GRDI funding is less obvious. Having said that, there is no evidence to suggest that DAs would continue or stop making genomics R&D investments in the absence of the program. Similarly, the evidence does not indicate the extent to which GRDI acted as a catalyst for these DA investments. In terms of overall funding over time, the APRs indicate that while GRDI funding for Phase V remained constant at $19.9 million, non-GRDI funding has increased from $19.6 million in 2011-12, to $31.9 million in 2012-13, and $37.7 million in 2013-14. Funding and in-kind contributions in support of GRDI projects have been leveraged from various sources such as A-base, B-base or C-base funds from GRDI funded departments, financial contributions from granting agencies (i.e. Natural Sciences and Engineering Research Council of Canada (NSERC), the Canadian Foundation for Innovation (CFI), the Canadian Institutes of Health Research (CIHR)), and collaborations with external partners, etc.Footnote 18

Continued need for interdepartmental collaboration

Collaboration is a cross-cutting theme in all 2016 Ministerial mandate letters, especially those responsible for science. According to an analysis of the mandate letters for participating DAs, collaboration is necessary to provide the evidence base for today's decision making because modern problems require multi-disciplinary, multi-sectoral and/or multi-jurisdictional approaches.

The concept of interdepartmental collaboration in the context of this evaluation question focused on the extent to which there is a continued need for SPPs. However, the evaluation also considered the continued need for interdepartmental collaboration in the context of the overall governance of this horizontal program (described above in Section 1.1.3). The evaluation found that these mechanisms are appropriate and working effectively (see Evaluation Question 8 in Section 2.3 for more details). While the degree to which there is a continuing need for these governance bodies was not directly assessed in the evaluation, none of those consulted for the evaluation identified any significant opportunities for improvement or desired changes to the status quo.

With respect to whether there is a continued need for the interdepartmental collaboration characterized by the SPPs, the evaluation found strong evidence that there is a continued need. The two SPPs funded in Phase V (the FWS and QIS projects) were based on common issues of importance to Canadians (per the selection criteria). Engaging researchers in cross-department collaborations has enabled DAs to address SPP themes from multiple perspectives (i.e., from the perspectives brought by multiple researchers in multiple DAs). According to the document review, multidisciplinary teams are becoming increasingly important to generating approaches for addressing issues of importance to Canadians.Footnote 19 The importance of multidisciplinarity was echoed by all interview respondents who felt that interdepartmental collaboration is still needed and relevant.

The main scientific emphasis of GRDI is towards downstream activities, such as the application of genomic information to deliver on federal mandates, and for the development of new products, processes, and services.Footnote 20 According to this same source, genomics research is increasingly being used with other technology areas (i.e. information technology, nanotechnology, physical and engineering sciences, new plant breeding techniques) and is creating new science disciplines (such as bioinformatics, nanobiotechnology, and biophotonics). Moving forward, integrating researchers from a variety of disciplines will be required to remain innovative and competitive.

Respondents indicated that the main needs addressed by the SPPs are the larger, most widespread problems facing the Government of Canada. According to some federal researchers conducting GRDI-funded research, GRDI's SPPs have also instilled a national focus on a number of topics important to the well-being and health of CanadiansFootnote 21, reinforcing their continued need. For example, the FWS SPP tracks the source of Verotoxic Escherichia coli (VTEC) such as E. coli 0157:H7, that can be present in food, and can reduce risks of serious and even fatal illness.Footnote 22

EQ 2: To what extent do the activities and outcomes of GRDI align with the priorities of the federal government and the priorities of partner departments and agencies?

Summary: Overall, the evaluation found a strong alignment of GRDI activities and outcomes with both federal government and DA priorities including alignment with the federal Science and Technology (S&T) Strategy and consistency with the mandate letter of the Minister of Innovation, Science and Economic Development (ISED). Not only are GRDI projects strategically aligned to meet the objectives of their respective DAs, they are also aimed at informing policy, program and regulatory decisions. The evaluation also revealed that mechanisms exist for GRDI as well as for all DAs for the selection of mandated projects/SPPs that ensure alignment to federal/DA priorities.

 

The evaluation found a strong alignment between GRDI objectives and both federal government and DA priorities. All interview respondents, as well as those others consulted for the case studies, felt that GRDI was aligned with the priorities of the federal government and their DA. This was supported through a review of the documents, which state that the projects funded under GRDI are focused on DA mandates as well as wider government priorities.Footnote 23 Further, an analysis of the 2013-14 Annual ReportFootnote 24 found that GRDI projects have been strategically aligned to meet the objectives of their respective DAs, and are aimed at informing policy, program and regulatory decision-makers. In particular:

  • The NRC went through an update of its Program Alignment Architecture during 2013-14. The renewed alignment showcased the department's new industry-focus, the Government of Canada's Strategic Outcomes and federal priorities as well as the NRC's business processes. Together, GRDI and NRC support NRC's Strategic Outcome Canadian businesses prosper from innovative technologies, the Program Technology Development and Advancement, and the Sub-Programs Aquatic and Crop Resource Development and Human Health Therapeutics.Footnote 25
  • Within AAFC, research conducted and innovation activities supported their goals of increasing and improving the competitiveness and sustainability of the country's agriculture sector. The funding they have received has been critical to enabling the department to move forward on this front. With the funding received through GRDI, AAFC has worked towards its targets and developed and strengthened the Canadian Crop Genomics Initiative. This allocation has allowed for the formation of various multi-disciplinary teams across the country that focuses specifically on improving sustainability and competitiveness on the topic.Footnote 26
  • The CFIA was a participant in both of GRDI's SPPs as well as the program's overall governance for Phase V.Footnote 27 In Phase VI, funds through a reallocation from NRC and AAFC genomics will play a large role in CFIA providing innovative, faster and more accurate testing methodologies through state of the art approaches. This will help to minimize public health risks associated with the food supply and transmission of animal diseases to humans, applying science and standards to market access, and ensuring safe and sustainable plant and animal resource bases. This works to increase the safety and security of the Canadian food supply and strength of the country's economy.Footnote 28
  • The GRDI-funded genomics research at DFO builds on the scientific knowledge base and expertise necessary to support sustainable commercial, subsistence and recreational fisheries management, ecosystems and oceans management priorities which include aquatic invasive species, species at risk, sustainable marine mammal populations, aquatic animal health, aquaculture, and understanding the adaptability and response of aquatic organisms to a changing climate and other stressors. This focus on aquatic species and ecosystems is strategically aligned with federal and Departmental responsibilities for environmental sustainability, scientific support for regulatory and policy decisions. National coordination of the program at DFO is done through the Biotechnology and Genomics Program. The program's goal is to support genomics research based on two of the three Strategic Outcomes of the department's Program Alignment Architecture.: Economically Prosperous Maritime Sectors and Fisheries as well as Sustainable Aquatic Ecosystems.Footnote 29
  • All GRDI funded research and development activities at ECCC are also aligned with departmental strategic outcomes. These outcomes include conserving and restoring Canada's natural environment for current and future generations, and the minimization of threats from pollution to Canadians and their environment. GRDI-funded activities that the department undertakes contribute to the monitoring and understanding of the country's ecosystem; help assess the risks created by chemical pollutants to wildlife and migratory birds; and deliver practical applications that support regulatory compliance and evidence-based decision making related to risk mitigation and conservation efforts.Footnote 30
  • At the Canadian Forest Service (CFS) of NRCan, genomics research is used to address Canada's natural resources and the competitiveness of the products created. Through GRDI, the CFS has developed the foundation for contributing to the Strategic Outcome Economic Competitiveness specific to the Program Economic Opportunities for Natural Resources. Additionally, the research has contributed to the CFS Intended Outcome Advancing Forest Product Innovation. These foundational activities for NRCan-CFS have resulted in the creation of data, different infrastructures and strong collaborations to deliver efficient applications.Footnote 31
  • GRDI at HC contributes to achieving the department's priorities through the creation of knowledge that is required for the effective regulation of health and food related technologies. Genomics research contributes towards policy development and regulations, informing and engaging the public on emerging technologies and supporting HC's efforts in harmonizing policies nationally and internationally. Additionally, during Phase V, GRDI addressed a number of strategic objectives under the sub-activity of Emergent Health Issues.Footnote 32 This sub-activity identifies emergent issues, provides strategic policy advice on how best to address them and develops appropriate responses, such as policy, new legislative or regulatory frameworks, tools or other approaches. The goal is to develop a responsive regulatory regime.
  • At PHAC, projects funded by GRDI support the Strategic Outcomes surrounding the public health infrastructure, the promotion of health and disease prevention and health security. Projects directly align with the Program Public Health Infrastructure to develop and apply leading-edge public health science and related tools to improve responses to emerging health risks and contribute to better public health. One way in which this is done is through the creation of innovative tools to apply genomic and bioinformatics technologies to have more effective public health interventions of infectious and chronic diseases. GRDI also helps support decision making and program development through leading edge scientific knowledge. In its work, there are various collaborations and exchanges of knowledge among public health professionals from federal, provincial, territorial and municipal levels of government and non-government organizations that GRDI helps facilitate.Footnote 33

The evaluation explored whether there are mechanisms in place to select mandated projects/SPPs that ensure alignment to federal/DA priorities. The evidence reveals that such mechanisms exist for GRDI as well as for all DAs for the selection of mandated projects. In particular, according to interview respondents and program documentation reviewed for the evaluation, GRDI is overseen and managed by an interdepartmental ADM CC, which is chaired by NRC. According to interview respondents, committee structures such as this help ensure alignment with the priorities of the government and the mandate of GRDI participating DAs. Additionally, as noted by interview respondents, since 80% of GRDI resources are allocated to fund DA-specific (mandated) projects and 20% of GRDI resources fund projects aimed at broader government-wide issues, the program ensures alignment to DA and federal government priorities.

Additionally, some management interview respondents explained that each DA has its own priority-setting mechanism to ensure GRDI mandated projects are aligned with their DA's mandate. For example, some use a process whereby a Letter of Intent (LOI) is solicited from scientists, followed by a proposal if the LOI is approved; others do not use an LOI/full proposal process, but still employ a competitive process to assess projects against each other; most DAs use peer review as part of their selection process to ensure the quality of the science, and lastly, most DAs rely on senior management, during their review of the projects, to ensure alignment with their DA's priorities.

The literature review also highlighted clear linkages of GRDI activities and outcomes to Government of Canada priorities. For example, genomics, as applied by the DAs within GRDI, is a key component of the federal S&T Strategy, which aims to develop a sustainable national competitive advantage by partly building on research strengths, generating new ideas and innovations, as well as achieving excellence. Looking forward, the Minister of Innovation, Science and Economic Development (ISED, the department responsible for the S&T Strategy) is tasked with the responsibility of supporting business growth, innovation and exports to ultimately create good quality jobs and wealth for Canadians (all of which are goals of many GRDI projects). Priorities identified in the Minister of ISED's mandate letter relevant to this program include:Footnote 34

  • "Support your Ministerial colleagues as they re-insert scientific considerations into the heart of our decision-making and investment choices."
  • "Examine options to strengthen the recognition of, and support for, fundamental research to support new discoveries."
  • Additionally, the Minister of Science has been tasked to pursue goals with a renewed sense of collaboration.

EQ 3: To what extent is GRDI consistent with federal roles and responsibilities?

Summary: The evidence confirms that GRDI-funded projects are consistent with federal roles and responsibilities. All lines of evidence demonstrate that it is appropriate for the federal government to be involved in genomics R&D. The federal government has legislated mandates related to the health and safety of Canadians (via regulatory activities) and support for industry (via economic development activities) and according to a few respondents, GRDI is an avenue for DAs to meet these mandates. Moreover, several interview respondents indicated that researchers in the academic and private sectors would not be conducting this type of work, since it is not part of their mandates or areas of interest. The evidence also illustrates that there is minimal duplication with other similar organizations or federal programs (such as Genome Canada).

 

The evaluation found that GRDI-funded projects are consistent with DAs' roles and responsibilities. GRDI aims to support the government's ability to anticipate and respond to the needs of Canadians in relation to areas of government responsibility for public health, sustainable resource management and ecosystems, the economy, agriculture and the environment.

As illustrated below, evidence from all lines of evidence demonstrated that it is appropriate for the federal government to be involved in genomics R&D. Together, the GRDI participating DAs have legislated mandates related to the health and safety of Canadians (via regulatory activities) and/or support for industry (via economic development activities). According to a few respondents, GRDI is an avenue for DAs to meet these mandates since most GRDI projects aim to increase the federal government's capacity to delivery on both of these broad mandates. Moreover, a few management respondents and scientists interviewed for the evaluation suggested that other researchers (such as those in academia or the private sector) would not be conducting this type of work since it is not part of their mandate or areas of interest.

In terms of the degree of duplication with other organizations or federal programs, the evaluation found that duplication with Genome Canada funded research is minimal and that the research funded by the two entities is largely complementary. Duplication is limited, in part, because of the funding barriers that exist between Genome Canada and GRDI, which inhibits the possibility of overlap between the work of each (i.e., Genome Canada funds researchers located in academia, whereas GRDI funds researchers located within federal DAs). Thus, it is not possible for federal researchers to access Genome Canada funding. Moreover, a few respondents indicated that academics are unlikely to pursue similar project objectives since they do not share federal DA mandates. However, federal scientists can and do leverage academic resources by collaborating with academics, supported by the Government of Canada, who are working in the same theme areas.

In addition, according to interviews and a review of documents, participating DAs typically institute a competitive selection process where all projects undergo review by individuals both internal and external to the DA, including experts and senior managers who are knowledgeable of research being conducted in other DAs and countries.Footnote 35 This selection process has not changed significantly since the last evaluation, which found that the scientific peer review component to select GRDI-funded mandated research projects was an effective method of limiting duplication of efforts.Footnote 36 The ADM coordinates GRDI activities with other federally funded research programs (e.g., Genome Canada, CIHR, NSERC) to ensure that opportunities for collaborations are pursued, especially in terms of strategic planning.Footnote 37 The committee includes members from each funded organization and guest representatives from ISED and Genome Canada.Footnote 38

The literature review found that, although some Genome Canada and GRDI mandated projects appear to be similar in nature, the outcomes of these projects will be very different (e.g., the EPIC4 Genome Canada project where the whole Coho genome was sequenced versus Single Nucleotide Polymorphism (SNP) research work done on Atlantic Salmon in a GRDI project). Genome Canada and GRDI are both interested in priorities for Canada that are driving academic research and apply to areas with application for DAs. At the same time, it is recognized that academic research in conjunction with research from the Government of Canada is necessary to drive and foster innovation. Genome Canada's 2012-17 Strategic Plan highlights the need to further establish strategic partnerships with GRDI by working collaboratively in areas consistent with participating GRDI DAs' mandates where genomics can have a significant impact.

2.2 Performance – Effectiveness

EQ 4: To what extent have the research projects funded as part of Phase V contributed to the development of evidence based public policy and innovative knowledge / technologies? To what extent and under what conditions were this knowledge and these technologies transferred to end-users inside and outside of the federal government?

Summary: Multiple lines of evidence confirm that research projects funded as part of Phase V have contributed to the development of innovative knowledge/technologies as demonstrated by: the total number of scientific contributions, well exceeding the target; and the total number of research tools and processes, also greatly exceeding the target. Additionally, the data indicate that this knowledge and these technologies were transferred to end-users inside and outside of government as evidenced by: the total number of knowledge and technology transfer activities, exceeding the target; the total number of communication products, exceeding the target; and the total number of research and technical personnel involved in projects, exceeding the target, the breadth of the type of personnel involved; and public policy decisions.

Targets were established based on the performance reported in Phase IV. The data from the current evaluation indicate that more scientific outputs (62%), and, importantly, more research tools and processes (940% more) were developed as well as more knowledge/technology transfer and outreach activities conducted (40% more) under this phase of GRDI. Overall, based on these data and interviews the evidence reveals that projects funded in Phase V produced more tools and processes to specifically address research and technical challenges faced by DAs and their stakeholders to strengthen their programs, policies and activities.

All interview respondents felt that innovative knowledge and technologies have been developed and that these have been transferred to end-users, influencing public policy in many cases. This is also confirmed by other lines of evidence, including case studies and document review. While progress was noted since the last evaluation, the evidence from the document review and interviews indicate that efforts to engage end-users should continue. The GRDI Innovation Management Strategy (IMS) developed in 2015 addresses, in part, concerns raised regarding timelines, clarity of pathways for technology transfer and ensuring that stakeholders who would use the knowledge and tools generated by GRDI are fully engaged.

End-users include those within individual DAs as well as those outside the funded DAs (including other DAs, the private sector, provinces and territories and other countries). Many different mechanisms for knowledge/technology transfer to internal and external users were mentioned, most commonly: peer-reviewed papers; presentations at conferences and workshops; and direct engagement/interactions, and collaborations with end-users.

Some commonly mentioned facilitating factors for knowledge/technology transfer to end-users included: knowledge/technology transfer must be proactive and planned (e.g., built into the project milestones); having end-users at the table when the research program is being designed; and having an incentive (either regulatory or financial) for industry in order for them to adopt the new technology.

 
GRDI performance: Knowledge and technology outputs

Research funded by GRDI has produced and disseminated a large number of scientific publications. The evaluation also found strong evidence of the development of knowledge and/or technology. The document and administrative data review provided an extensive overview of the scientific contributions produced in Phase V of GRDI. Overall, the number of scientific contributions exceeded targets by 62% (see Table 4 on the following page). In terms of the development of research tools and processes, Phase V reported a very high number of research tools and processes (exceeded targets by 940%). Note that targets for Phase V were based on the actual output levels reported in Phase IV (2008 and 2011).

In terms of transfer to end-users, all targets were also exceeded by more than 40% for knowledge/technology transfer activities, communications products and the number of highly qualified personnel (HQP) involved in GRDI-funded projects (Table 4). Additional details on these outputs, presented by participating DA are provided in Appendix F.

Under Phase V of the GRDI, more than 500 Knowledge and Technology Transfer activities/outputs were reported. The number of knowledge and technology transfer activities and outputs increased from 90 in 2011-12 to 252 in 2013-14. The production of knowledge and technology transfer activities and outputs under Phase V of the initiative is well above the range recorded for Phase IV of the initiative. For instance, AAFC has developed genomic tools to help refine promising new industrial crops, identified a leaf rust resistance gene leading to improved understanding of resistance mechanisms, and developed new tools to accelerate the breeding process and get improved crop varieties to market faster.Footnote 39 ECCC has developed a unique bacterial DNA marker which detects fecal contamination from seagulls in waters around the Great Lakes that has resulted in the discovery of new tools to detect and measure seagull fecal contamination in water samples. ECCC has also developed toxicogenomics methods to identify endocrine disruptors in salmonids and to assess the effectiveness of municipal wastewater treatment systems. The results showed the relative effectiveness of the various systems, and are used to guide municipalities that are wanting to introduce new treatment systems.Footnote 40Footnote 41

Table 4: Targeted and reported number of scientific contributions and of research tools/processes (2011-12 to 2013-14, includes SPP output for this periodTable note *)
Outputs Targets for Phase V (2011-12 to 2013-14)Table note ** Reported (2011-12 to 2013-14) % Above target
Scientific contributionsFootnote 42 1871 3036 62%
Research tools/processesFootnote 43 30 283 940%
Knowledge/technology transfer activities and outreach activities for disseminating results to end-usersFootnote 44 366 513 40%
Communications productsFootnote 45 151 241 60%
Research & technical personnel involvedFootnote 46 1690 2410 43%
 

Source: GRDI Administrative Data – Fiscal Years 2011-12, 2012-13, 2013-14

Table 4 notes

Table note *

SSP were funded for two additional years (2014-2015 and 2015-2016). SSP data for these two years are not presented in this table. SPP data for the fiscal years 2014-15 were available, but not for 2015-2016.

Return to footnote * referrer

Table note **

Quantitative targets were established based on GRDI Phase IV results between 2008 and 2011 and described in the GRDI Performance Measurement Strategy.

Return to footnote ** referrer

 

GRDI's APRs for Phase V also highlight a number of innovative tools produced by GRDI-funded projects. Table 5 depicts the number of research tools and processes for mandated projects undertaken from 2011-12 to 2013-14 and Table 6 depicts the results for SSPs for the five-year period (2011-12 to 2015-16). While data for the final year of SPPs (2015-16) was not available, SPP case study interview respondents confirmed that a large number of outputs will be finalized and reported in the final year of funding. Some examples of research tools and processes developed include:Footnote 47

  • Microbial In Silico Typer (MIST), a bioinformatic tool to simulate/predict the results of multiple molecular subtyping methods from draft-genome sequence data (FWS);Footnote 48
  • Protocols for the extraction of nucleic acids from marine vertebrates and invertebrates (QIS);
  • Fungi microarray (Fusarium) data deposition (#GSE56340) in Gene Expression Omnibus (GEO) databaseFootnote 49
  • DNA markers for genotyping wild and farmed Atlantic salmon (Salmo salar) (DFO);
  • Chicken Microarray (Agilent 44K ) for screening for potential toxic effects of priority environmental contaminants in birds (ECCC);
  • Assay for regulators to screen chemical food additives and contaminants for their ability to activate the immune system and increase the risk of food allergies (HC);
  • Orthology & gene order analytics: standalone scripts/programs (NRC);
  • Bioinformatics pipeline for identification of secreted proteins (NRCan); and
  • A new SNP-based tool for subtyping of S. Enteritidis (PHAC).
Table 5: Mandated projects: Number of research tools and processes developed, 2011-12 to 2013-14
Research Tools
  2011-12 2012-13 2013-14 Total
AAFC 19 16 20 55
DFO 3 0 5 8
ECCC 10 13 15 38
HC 4 9 11 24
NRC 4 2 6 12
NRCan 9 4 4 17
PHAC 4 5 6 15
Total 53 49 67 169
Research processes
  2011-12 2012-13 2013-14 Total
AAFC 15 8 7 30
DFO 1 0 0 1
ECCC 2 10 2 14
HC 0 3 0 3
NRC 4 5 9 18
NRCan 4 3 2 9
PHAC 2 7 8 17
Total 28 36 28 92
 

Source: GRDI – 3 Years Performance Tables 2011-2014

All lines of evidence found extensive evidence : Number of research tools and processes developed, 2011-12 to 2013-14
Research Tools
  2011-12
Table note *
2012-13 2013-14 2014-15 2015-16
Table note **
QIS 4 2 7 8 N/A
FWS N/A 4 5 3 N/A
Total N/A 6 12 11 N/A
Research Processed
  2011-12
Table note *
2012-13 2013-14 2014-15 2015-16
Table note **
QIS 7 2 3 2 N/A
FWS N/A 1 1 0 N/A
Total N/A 3 4 2 N/A

Source: GRDI – 3 Years Performance Tables 2011-2014 and QIS Project Partial Report, November, 2012

Table notes

Table note *

Performance tables for 2011-12 do not provide data for SPPs spAll lines of evidence ecifically. QIS numbers in this table were calculated using the narrative in the QIS Project Partial Report, November, 2012. FWS partial report does not include details on the research tools and processes produced.

Return to footnote * referrer

Table note **

Performance tables for 2015-16 were not available at the time of the evaluation.

Return to footnote ** referrer

 

All lines of evidence found extensive evidence of knowledge generated as a result of GRDI-funded projects. The following represent some, among the many, examples found:

  • Genomics knowledge to strengthen public health programs, policies and activities related to infectious and chronic disease, including: development of rapid methods for molecular typing of Salmonella and genomic tools and methods to more accurately identify disease-associated E. Coli strains;
  • Genomic knowledge for the Canadian health regulatory system, including genomic assessment of chemical food contaminants leading to food allergy and the development of short term cancer bioassays using transgenic mice exposed to carcinogens; and
  • Using genomics to improve the value of cereal, canola and legume crops, including biodiversity, gene mining and functional analysis for the identification and extraction of genes for desirable traits.

The documents reviewed provided evidence of changes in priorities resulting from the research conducted in GRDI-funded projects. The APRs highlighted some examples of how project results have been used thus far to inform decision-making leading to changes (including changes to public policy) that will ultimately benefit Canadians:

  • Food and water safety: The FWS SPP results are being shared with a broad range of end-users and stakeholders which will enable researchers in both government and academia to build and refine the knowledge created during this project. Regulatory bodies and the food industry will also be able to use this information and advanced technology to improve water and food safety. Lastly, the research produced through the FWS project is also being shared with private sector organizations that may be interested in the commercial potential of these technologies.
  • Quarantine and invasive species: Based on case study evidence, GRDI genomics data on the Downy Mildew pathogens, a fungal plant pathogen of soybean, have allowed Canada of soybean helped to keep soybean exports in Malaysia. Downy Mildew (P. manshurica) was introduced to Canada decades ago and has had a marginal economic impact on Canadian crops except when quarantine issues are raised. Downy Mildew, although found around the world, remains a quarantine species for some countries such as Malaysia. In March 2014, Malaysia proposed new import conditions for soybeans from Canada, including imported soybean seeds, stating they must be free of these pathogens. By linking newly acquired GRDI reference data with other scientific and historical data at AAFC and other international databases, AAFC, the CFIA and the Canadian Grain Commission rapidly produced scientific results to challenge the Malaysian request. The scientific team demonstrated that this species was common in Canada, therefore, it would be nearly impossible to certify soybeans as being free of the Downy Mildew. Furthermore, the GRDI data were used to challenge Malaysia to demonstrate, in turn, that this species was not native in their land. Given that Malaysia is surrounded by countries that have reported P. manshurica, it is very unlikely that it is completely absent from Malaysia, even if there is no commercial soybean production in Malaysia. As a result of the GRDI data, trade borders remained open and Canada continues to export soybeans to Malaysia. This trading partner was forced to change its regulations on the basis of the evidence provided.
    The GRDI Downy Mildew data also helped to avoid a false positive of Canadian commodities with the Brown Stripe Downy Mildew of Corn (Sclerophthora rayssiae var. zeae), a US Select Agent. If this pathogen were to come into North America, it is estimated 90% of all North American corn crops would be lost. USDA - APHIS needed GRDI data to sort out some taxonomic issues following the sequencing of the select agent. It turned out from its data analysis that Sclerophthora cryophila from Canada is the closest relative to the select agent which has no published sequences yet. This Canadian species is a somewhat benign plant pathogen of little economic value that is specific to orchard grass highly common in BC. Without this knowledge on the innocuous orchard grass pathogen, metagenomic data from BC would likely test positive for the US select agent, potentially causing trade disruption. AAFC and USDA are now working together to generate reference data, including the genomes of the orchard grass pathogen and the select agent to avoid false positive detections.
  • Municipal resource management decisions: "Using toxicogenomics tools, GRDI scientists [ECCC] assisted the City of Montréal (…) to select a new wastewater treatment system. Results showed that ozonation effectively reduced target levels of bacteria while breaking down many other contaminants, and the city invested in an ozonator facility. Another GRDI project showed the relative effectiveness of wastewater treatment systems of twelve other municipalities, to be used as a guide by municipal decision-makers when new treatment systems need to be introduced."Footnote 50
  • Plant protection regulations (with applications for agriculture, forestry and natural environments): "Even with import requirements in place aimed at keeping it out, soybean cyst nematode was detected in Ontario in 1987. At that time, soybean producers in affected areas became subject to regulations to prevent the movement of soil and plants that could be infested. With early detection in mind, GRDI scientists developed a test for the presence of this nematode in soil that is faster and more accurate than anything available in the past. In 2013, the test was used to confirm the presence of soybean cyst nematodes in Quebec for the first time, a finding that played a part in the CFIA decision to lift the regulations that had been imposed on soybean growers in Ontario and Manitoba."Footnote 51
  • Fisheries management and conservation: "Conservation of threatened Atlantic salmon populations around Newfoundland and Labrador is a major goal for DFO. Through sampling of wild Atlantic salmon from conservation units around Newfoundland and from aquaculture samples representative of all strains cultured on the South Coast of Newfoundland, researchers developed microsatellite based marker systems for identifying aquaculture escapes in wild populations. Genetic data on wild Atlantic salmon in Newfoundland and Labrador contributed to science-based recommendations provided to protect Canada's wild populations of Atlantic salmon and became instrumental in enabling government enforcement officers in forensic analysis to identify confiscated fish products."Footnote 52
  • Environmental risk mitigation: ECCC's scientists developed various innovative methods and tools to support the management of environmental risks posed by chemical, biological, and physical pollutants. One method entailed exposing a crustacean microarray to varying levels of contaminants and examining their effects on specific genes. By using this method, scientists were able to demonstrate significant changes in gene expression in certain species due to contaminant exposure. These methods support regulations such as the 1999 Disposal at Sea Regulations under the Canadian Environmental Protection Act as well as the department role under the Fisheries Act.Footnote 53 For example, "A genomics method that uses biomarkers associated with chronic stress to investigate the effects of environmental disturbances on wildlife health was developed and validated by GRDI scientists. So far, this technique has been used to examine the effect of large scale environmental disturbances on the immune functions, reproductive activities, and survival of wild birds. Information collected through this work was disseminated to provincial decision makers to support related risk mitigation efforts."Footnote 54
Knowledge and technology transfer to end-users

All interview respondents who were in a position to comment on this performance aspect felt that there had been knowledge transfer to end-users (including those within and outside their DA). Management respondents indicated that the new technologies and approaches developed with GRDI funding have become routine parts of their business. It was also noted by a federal scientist, while some parallel work is being done with older technologies (to ensure reliable and valid results are produced with new approaches), end-users are increasingly using GRDI-developed approaches in the field.

The following knowledge and technology transfer mechanisms were identified by interviewees in the context of stakeholder and case study interviews and confirmed by the review of GRDI administrative data (i.e., GRDI annual performance reports):

For transferring knowledge to those within DA, mechanisms included: creation of communities of practice, internal symposia or workshops, peer-to-peer interactions, established communication tools to facilitate the transfer of information from laboratories, to risk assessment to the policy group, and open-source software. One respondent indicated that the extent to which internal end-users use GRDI-funded tools is a work in progress and "won't change the way the department works overnight." That respondent's comment speaks to the finding that there is a continuing need for ongoing knowledge transfer of GRDI outputs to the federal programs that benefit from their application.

For transferring knowledge outside the DA, main mechanisms included: presentations and/or publications, attending conferences and workshops and direct interaction with end-users such as having them on the project team, engaging them as collaborators, or sitting down with them during a workshop; collaborating with potential end-users such as asking them to become pilot testers/early adopters, producing popular articles and web content to reach a broader audience; and contributing to international organizations.

Success and hindrance factors that explain GRDI's performance in terms of knowledge / technology transfer to end-users.

The transfer of knowledge and technology to end-users was also identified as a challenge in the last evaluation (in 2011). For instance, Recommendation 5 in the last GRDI evaluation recommended that the GRDI "develop mechanisms that further integrate users of R&D in all stages of genomics R&D project's life cycles in order to ensure proper alignment of scientific progress with targeted potential uses and expected impacts."Footnote 55

In fact, the 2011 evaluation found that most end-users were not systematically identified or integrated into the research process.Footnote 56 Rather, most collaborators and end-users consulted were only engaged after its completion or were not engaged in the dissemination and transfer of R&D results stage of Phase III projects. This was considered inadequate as the program requires the adoption and application of genomics R&D results, meaning it is critical that there are better mechanisms to integrate users of R&D results in all stages of the project's life cycles and improve continuity of user involvement in the context of long-term R&D projects. According to the 2013-2014 APR, the APRs now require a list of available stakeholders and end-users for each project (including contact information).Footnote 57 Those consulted for the current evaluation reported some improvements on this front and actions have been taken to mitigate these risks for Phase VI projects, such as the Innovation Management Strategy (IMS) in 2015 and the conduct of a workshop for the FWS SPP in 2016.

While the current APR template for the mandated projects and SPPs requires projects to identify the potential end-users and engage them in the projects, the GRDI IMS and the SPP mid-year report identified opportunities for improvements. In this regard, end-user engagement and the uptake of project deliverables were areas identified as needing improvement in the SPPs funded in Phase V.Footnote 58 The SPP mid-year report completed in 2013 recommended that more efforts are needed to fully engage in technology transfer activities to ensure that the needs and interests of the end-users are understood.Footnote 59 More feedback from collaborators, stakeholders and other end users to ensure end-user approval was also suggested.Footnote 60

These issues were also echoed in the program's Strategic Planning Meeting (2014). In this meeting, representatives for both SPPs reported having issues fully engaging early adopters (both, public and private) to define key success factors (rather than traditional methods) to maximize the impacts of their research results. Mitigation efforts employed by the QIS team included ensuring that potential end-users were included in both the planning and execution of the project and received regular feedback.Footnote 61

The QIS team also found it difficult to engage end-users to validate the value of project research results. Given that these projects were new to the participating DAs, there was little time to develop relationships with end-users. Further, they did not have access to any public outlet to allow the information to be accessed at the beginning of the project. Mitigation strategies extended to building relationships with collaborators and end-users steadily throughout the project. A portal is also being developed to disseminate the data and information generated through the project.Footnote 62

Interview respondents identified a number of common facilitating and hindering factors for the successful transfer of GRDI knowledge and technology to end-users. Facilitating factors highlighted by respondents included:

  • Knowledge/technology transfer being proactive and planned (e.g., built into the project milestones);
  • Having guidelines for DAs to link end-users to research objectives;
  • Having a requirement for DAs to report on knowledge transfer activities;
  • As the regulator, when the federal government says they are using a certain approach, this is an incentive for industry to also use the same approach to ensure they are in compliance;
  • Engaging end-users in the research project from the beginning (i.e., at the planning and design phase);
  • Having personnel with the necessary capacity and expertise to challenge industry;
  • Developing a training program for end-users;
  • Packaging information in a user-friendly manner; and
  • Ensuring there is a cost-advantage for industry in order for them to adopt a technology.

Hindering factors included:

  • Lacking funds to focus on knowledge transfer or lack of a dedicated knowledge transfer unit within DAs;
  • Having limited ability of the scientists to attend conferences to share information about the technology and meet with end-users directly due to funding challenges;
  • Having end-users who are more comfortable with traditional approaches or who do not have confidence in the science;
  • Having a long lag time between bench research, technology development and adoption/implementation (which can be as much as 10-15 years, for example in bringing a new crop variety to market) when there may not be continuous funding to bring a research result completely through the innovation pipeline.

In February 2016, a 2-day knowledge transfer workshop for the FWS SPP was held at the National Library and Archives with the purpose of showcasing some of the innovations of the FWS SPP, including new processes and tools that have the potential to significantly improve food safety in Canada. This workshop helped identify some of the key success and hindrance factors that influence the transfer to end-users.Footnote 63

"The overall goal was to foster an environment of exchange and collaboration between researchers and end-users to boost the commercial uptake of publicly-funded R&D throughout Canada. The workshop also represented a unique opportunity for those involved in food and water safety from industry, academia and government to explore innovative approaches to some of the challenges facing the food safety community."Footnote 64

The workshop reiterated some of the gaps in science, research, technology and tools, including the need for more collaboration between producers and users of scientific evidence. Important questions were raised on how to continue communicating and collaborating moving forward.Footnote 65

Overall, the cross-sectional nature of the participants in the workshop was considered to be an important strength and there is much value in ongoing networking and developing communities of practice across the participating sectors and organizations. It was noted that the players themselves are responsible for making collaboration happen. Workshop participants noted the following top suggestions to ensure continued dialogue and guarantee a legacy of outcomes and actions:Footnote 66

  • Governments, universities, publicly funded research institutions and companies, need to enable collaboration and avoid duplication of efforts.
  • Workshop participants underscored the importance of face-to-face meetings, workshops and events to achieve this goal. They also stressed the need to "broaden the circle" to ensure the right people are at the table for these discussions (including academics and end-users).
  • Create platforms for funding, collaboration and extending innovation.
  • Facilitate more opportunities for interaction with the end-users (regulatory and industry end-users) of these new tools and technologies.
  • Create bridges between research being done within specific disciplines to allow sharing of best practices, ideas and lessons learned.
  • Communicate and disseminate research outputs and outcomes upstream to senior management, as well as DA policy managers, to support knowledge translation.

EQ 5: What impacts have been generated from the use of GRDI's research results by end-users and external stakeholders? What success / hindrance factors have influenced the use of knowledge and technologies by end-users and external collaborators? To what extent would those research results have occurred without the funding from GRDI?

Summary: Multiple direct and indirect end-users apply, or are highly likely to apply the knowledge and tools developed as a result of GRDI projects, including researchers and lab staff, surveillance agencies, breeders, producers, importers/exporters, regulators and various decision-makers in government.

Evidence from the case studies and the interviews provide details of the most significant accomplishments and impacts of the program. These include impacts in the areas of: improved public health and wellness; avoided health system costs; efficiencies and avoided losses for the public and private sectors, environmental sustainability; improved detection of invasive species; improved varieties/traits of plants, trees and animals of commercial value; and improved fisheries tracking and management. Evidence indicates that the benefits will likely exceed program costs within the next 10 years. According to the evidence, while many factors contributed to these impacts, most of the research projects would not have happened or would have been delayed in the absence of GRDI.

Evidence indicates that some of these benefits have been facilitated by early involvement of end-users in the research projects, and/or when actions and technologies are deployed to transfer knowledge and tools to users outside the federal government.

Impacts for end-users

Interview and case study respondents provided multiple examples of GRDI project outputs that are being used by internal and external end-users. They include genomics information, databases, platforms and tools used by direct users such as laboratory personnel in federal government, provincial and university labs. These users utilize these tools for various purposes, including the characterization and/or detection of bacteria, viruses, fungi, plants, trees, fish, agricultural products, etc. These tools serve the needs of other users and stakeholders, such as surveillance programs, breeders, producers, importers/exporters, regulators and various decision-makers in government (including policy makers). These indirect users benefit from genomics in a wide variety of areas, including the protection of the environment and sustainable development; improved public health and wellness; increased production/economic benefits and various avoided costs and improved efficiencies.

Figures 1 and 2 present a summary of the estimated and partial health-related and resource-related impacts of some of the GRDI-funded projects. The impacts are estimated in the sense that the concrete impacts of the innovations are expected to be realized in the next 10 years (longer in some cases). Estimates are based on case study evidence and conservative assumptions. These impacts are deemed partial because, 1) they are based on an analysis of the impacts of a sample of projects (case studies); and 2) the analyses did not include a complete analysis of the secondary impacts (e.g., fiscal revenues). Also, it does not involve an input-output analysis of the benefits.

The models illustrate how the innovations achieved in projects result in outcomes and longer-term impacts. Where possible, the range of benefits has been articulated (e.g., avoided costs). The benefits shown in the model are based on an analysis of multiple sources of evidence reviewed as part of the case studies, including interviews, documentation and literature. In some cases, conservative assumptions were used based on literature. As shown, the estimated benefits (likely in the range of hundreds of millions of dollars) will exceed the costs of the program.

Figure 1: GRDI impact summary – Health related outcomes (estimated)
Long description of image follows.
Long description of Figure 1.

Figure 1 is a visual depiction of the GRDI impact summary for health-related outcomes (estimated) – showing the progression through innovation achieved in projects, to applications, to outcomes and ultimately to health impacts. In addition, examples of impacts from case studies are provided, relating to these health impacts.

Demonstrated innovation
Highly-likely or demonstrated applications and outcomes:
Innovation Applications Outcomes Health Impacts
  • Innovative knowledge
  • Improved genome database information
  • Improved bioinformatics, tools, gen9omic technologies & processes
  • Improved/reliable characterization/ identification
  • Improved detection/ diagnostics & monitoring tools
  • Improved policies & regulations/ evidence-based decisions making
  • Improved health treatment
  • Improved safety and security
  • More timely/ effective interventions
  • Improved health outcomes
  • Avoided harm/ costs
  • Improved economic benefits
 
Examples of Impacts from Case Studies
Partial impact analysis - Highly-likely impacts within 5-10 years
Improved technology for the classification of Salmonella Serovars(PHAC)

Avoided costs associated with outbreaks and associated disease

SBE: Cost reduction of 1% per year = $370M

Improved technology for HIV drug resistance testing(PHAC)

Reduced HIV testing cost as compared to traditional/commercial method

SBE: By 40% and 90%

Improved detection & risk assessment methods against foodborne pathogens(HC)

Avoided costs due to incidence reduction of campylobacteriosis & Listeriosis cases

SBE: 20-75% less cases per year = $124M-$287M

Improved detection and diagnostic methods against water-borne pathogens and other toxic agents (NRC, PHAC, HC, CFIA, EEEC)
Shared Priority Project
  • Avoided costs due to major bacteria outbreack, recalls & trade issues avoided
    SBE: $5M to $15M, once every 10 years
  • Avoided hospitalization costs due to bacteria outbreaks in food and water
    SBE: $1.25M per year = $12.5M /10 years
Improved water-usage management resulting from improved knowledge about harmful algae blooms (ECCC)
  • Reduced risk of public exposure to contaminated water supplies
    Benefit estimate N/A
  • Reduced costs to agriculture and fisheries
    Benefit estimate N/A
 
 

Note: Figure is based on scenario-based estimates (SBE) derived from case studies. Methods and sources of estimates are multiple and different for each case; N/A: not available or not calculated in the context of this particular case. Source: Goss Gilroy, Inc.

Figure 2: GRDI impact summary – Estimated resource-related outcomes
Long description of image follows.
Long description of Figure 2.

Figure 2 is a visual depiction of the GRDI impact summary for (estimated) resource-related outcomes – showing the progression through innovation achieved in projects, to applications, to outcomes and ultimately to resource impacts. In addition, examples of impacts from case studies are provided, relating to these resource impacts

Demonstrated innovation
Highly-likely or demonstrated applications and outcomes:
Innovation Applications Outcomes Health Impacts
  • Innovative knowledge
  • Improved genome database information
  • Improved bioinformatics, tools, gen9omic technologies & processes
  • Improved/reliable characterization/ identification
  • Improved detection/diagnostics & monitoring tools
  • Improved policies & regulations/ evidence-based decision making
  • Improved resources/ stock management
  • Improved safety and security
  • More timely/ effective interventions
  • Improved breed/ traits
  • Enhanced protection & management of Canada's environment & resource sectors
  • Improved market access
  • Enhanced sustainability of resources/ biodiversity
  • Avoided harm/ costs
  • Improved economic benefits
 
Examples of Impacts from Case Studies
Partial impact analysis - Highly-likely impacts within 5-10 years
Improved insect resistant & yield of Canadian spruce (NRCan)
  • Gains in wood yield/volume
    SBE: Gain 5% = $300M/year (in very long term)
  • Avoided costs due to plantation losses are associated to insect damage
    SBE: Loss reduction 1-5% per year = $300M
Improved detection and diagnostic methods against Quarantine and Invasive Species (AAFC, CFIA, ECCC, DFO, NRC, NRCan)
Shared Priority Project
  • Avoided costs due to faster/accurate detection methods of quarantine vine viruses
    SBE: 345% less expensive ($110k vs $2.6M: 1800 vines)
  • Avoided costs due to removal of restrictions to Canadian yellow peas exports to India
    Benefit estimate N/A; Market of $350M per year
  • Avoided costs due to removal of restrictions to Canadian soybeans exports to Malaysia
    Benefit estimate N/A; Market of $85M per year
Improved fish management from improved knowledge about Salmon, Scallop & Redfish populations (DFO)
  • Improved management of Atlantic Salmon sub-populations (protection vs recreational fishing)
    SBE: Re-opening of designated areas to recreational fishing = +$10M per year
  • Improved processes on redfish and sea scallop stock assessment and recovery potential
    Benefit estimate N/A
Improved disease resistant cultivar of Canadian wheat (NRC) Avoided costs due to crop losses associated with fusarium head blight (FHB)
SBE: Loss reduction of 1% per year = $15M
Improved technology for oat improvement (AAFC) Avoided costs due to crop losses associated with crown rust
SBE: SBE: Loss reduction 1-5% per year = $6M-$30M
 
 

Note: Figure is based on scenario-based estimates (SBE) derived from case studies. Methods and sources of estimates are multiple and different for each case; N/A: not available or not calculated in the context of this particular case. Source: Goss Gilroy, Inc.

In terms of long-term impacts, Table 7 identifies the impact areas on which the DAs are focusing. The table contains results for 68 mandated projects and the two SPPs. Data is based on the 2012-13 and 2013-14 APRs and case study evidence. Some projects reflect multiple impact areas.

Table 7: GRDI-funded projects Linked with Areas of Intended Long-term Impact
  Number of Mandated Projects Number of Shared Priority Projects
Improved Public Health and Wellness 17 2
Towards Environmental Sustainability 27 1
Evidence-Based Decision-Making 21 2
Formulation of Policies, Standards and Regulations 9 2
Support for Canadian Commercial Enterprises 30 1

Evidence from the case studies and the interviews provide details of the most significant accomplishments and future estimated impacts of the program. These include: impacts in the areas of improved public health and wellness; environmental sustainability; improved detection of invasive species; improved varieties/traits of plants, trees and animals of commercial value; and improved fisheries tracking and management. Each of these is described in more detail, below.

  • Improved identification, detection, diagnostics and monitoring of harmful bacteria and viruses. Many projects have led to the development of tools and resources for improved identification, detection, diagnostics and monitoring of harmful bacteria and viruses. This was achieved by improved bioinformatics platforms and the development of improved tests to detect and identify water and food-borne bacteria, including E. coli, Salmonella, Campylobacter and Listeria. For example, the FWS project led to the development of bioinformatics platforms that store and make available markers and other information about E. coli, and eventually about other key bacteria monitored by the Canadian surveillance system. It also led to the development of rapid lab tests based on genomic marker information. While in advanced validation stages, results to date show that the tests will likely become the main procedure to identify E. coli at all stages of the chain (from food source to market). Other similar projects were conducted on Salmonella, Campylobacter and Listeria as part of mandated research projects. Extensive work has also led to successes in the area of the HIV drug resistance testing that may eventually lead to more efficient HIV tests and improved treatment for HIV patients.

    Ultimately, the analysis of evidence gathered through the case studies suggests that it is highly likely that these improvements to the surveillance system will reduce the costs and harm associated with these bacteria and the HIV, including costs to the health system and to producers, and costs in terms of harm to human health. These are summarized in the following points. As shown, likely impacts are in the range of billions of dollars in the next 10 -20 years (for GRDI investments made in 2011-2014, that is, $59.7 million):

    • Genomic characterization of clinically important foodborne and environmental isolates of Campylobacter and Listeria: Knowledge has been transferred to end-users at HC and CFIA. According to documentation, projected annual benefits range from a low of approximately $120 million to a high of approximately $240 million. Over a 20-year period, the present value of these benefits ranges from approximately $1 billion to $5 billion.
    • FWS: Characterization and rapid detection of E. coli is likely to lead to health care and private sector savings and avoided costs in the range of $22M to $32M over 10 years (total).
    • New technology for HIV drug resistance testing: This research will contribute to improved HIV resistant drug results. It will improve treatment for HIV patients. The new platform can also reduce HIV testing cost by about 40% as compared to traditional approaches.
    • A Rapid Geno-Serotyping Tool for the Classification of Salmonella serovars: This project will improve the response capacity of the public health system and support science-based decision-making related to foodborne public health issues.
  • Improved characterization and detection to monitor and protect the environment. Case study evidence provides two examples of such projects. The ECCC project that developed the Rapid Assessment Indicator of Algal and Bacterial Community Composition provided a cost-effective sampling and monitoring method to monitor harmful algal blooms, which affect the water quality of multiple Canadian lakes. These improved methods will contribute to an improved management of fisheries, municipal water systems and cyanotoxin-related beach postings (quality of swimming water). Another example is the Toxicogenomics project, which developed genomics-based methods to assess the toxicity of various chemicals that can harm the environment as well as human health. The project also led to more efficient toxicity tests.
  • The QIS SPP focused on building targeted reference libraries of genetic data (DNA barcodes) and associated protocols for a number of aquatic and terrestrial organisms of economic and biodiversity importance to Canada. The genomics bioinformatics libraries and detection tools led to many impacts (achieved and likely) in terms of environmental and economic impacts. The genomics detection tools are more efficient than traditional detection techniques, including reduced lab time to confirm the absence of pests in products to be imported in Canada or exported abroad.

    The detection tools also allow Canada to protect its access to foreign markets when Canadian products are suspected of carrying pests. Genomics information allowed Canada to protect its soybean export market in Malaysia ($90M annually, based on case study evidence), and to prevent a closure of its corn export market in the U.S. ($134M annually, based on case study evidence). Similar actions also protected the pea export market in India

  • Improved varieties/traits. A number of projects led to the successful development of improved varieties (wheat and oat) of commercial value. These projects essentially involve the identification and characterization of strains that demonstrate resistance to pests that result in significant costs to industry, either through direct losses due to pests, or indirectly through increased barriers to exports. Three case studies conducted for the evaluation demonstrate the successes of these projects. In particular, improved crop production is expected from the characterization of pest-resistant wheat and oat. Although further research and validation is yet to be conducted, case study evidence indicates that it is highly likely that these will lead to major savings for industry (by avoiding crop losses). In another case, research on tree characterization for superior wood fibre and pest resistance will lead to the outplanting of trees with these desirable traits. In the long term, the higher wood volume is expected to increase harvest revenues by $300M per year once the trees are harvested. Other projects led to the identification of genes involved in fungal pathogen-host interactions of white pine and poplar, which will lead towards the development of diagnostic tests. Genetic markers have been developed for these species. Genomic tests for pest resistance pines (against fungi) were transferred to breeding programs in Canada (in British Columbia and Quebec) and will eventually be used by forest managers in Canada and in the U.S.
  • Improved fisheries tracking and management. Also in the area of natural resources, GRDI-funded research has improved fisheries management. For example, genomic characterization work on Redfish has enabled a better mapping of sub-populations. This information can be used for not only current stock assessments but predicting future directions of the stocks, setting conservation priorities and recovery strategies as well as enabling better long-term management practices. Genomics mapping tool development will also improve scallop stock management, providing new and important information that would assist DFO managers in moderating risks to scallop stocks, including decisions about whether to restrict harvest levels. Finally, the identification and application of targeted groups of genetic markers to quantify the genetic impacts of farmed Atlantic salmon on wild populations and the frequency of interbreeding in the wild directly targets a client need and is a first step towards identifying impacts and strategies for alleviating those impacts that result from interactions among wild and escaped farmed salmon in Atlantic Canada.

In terms of whether these impacts would have occurred in the absence of GRDI, the evaluation found that most of the research projects would not have happened or would have been delayed in the absence of GRDI. In particular, case study evidence suggests that both SPPs would not have happened without GRDI. As these SPP are addressing areas where federal science-based DAs have complementary mandates, the absence of GRDI would not achieve the same synergistic outcomes that the SPP have achieved to date. Case study evidence also indicated that half of the mandated projects covered by the case studies would not have taken place, three would have been delayed and one would have been at a lower scale. Supporting the case study evidence, some of the interview respondents supported the view that some of the projects would have been conducted without GRDI, albeit at a slower pace or later in time.

Most respondents felt that some genomics R&D would continue in the absence of GRDI. Some said it could not be sustained at the same level and that trade-offs with other research would have to be made and/or work would happen more slowly. A few respondents indicated that their DA would stop doing genomics R&D altogether and that their scientists would leave.

Success and hindrance factors that influenced GRDI's ability to generate impacts for end-users

The GRDI IMS (2015) and the SPP mid-term review (2013) identified opportunities for improvements. Early end-user engagement was identified in these reports as being key to the successful implementation and take-up of research results.Footnote 67 More feedback from collaborators, stakeholders and other end users to ensure end-user approval was suggested.Footnote 68 There is evidence that improvements have been made since these reports were released, according to case study evidence, which indicates that many projects (including shared-priority projects) involved end-users later in funding cycle.

These issues were also echoed in the program's Strategic Planning Meeting (2014). Representatives for both SPPs reported having issues fully engaging early adopters to define key success factors to maximize the impacts of their research results. Mitigation efforts employed by the QIS team included ensuring that potential end-users were included in both the planning and execution of the project and received regular feedback.Footnote 69

The IMS was developed in 2015 in part to address concerns around timelines and clarity of pathways for technology transfer and to ensure that stakeholders who would use the knowledge and tools generated by GRDI were fully engaged. The IMS provided guidance "on designing GRDI SPPs and preparing Project Management Plans to improve end-user engagement and the uptake of project deliverables."Footnote 70 The IMS articulates that the project impacts could be enhanced by increased end-user engagement and seeks to improve end-user engagement in Phase VI.Footnote 71

The recommendations in the IMS around enhancing end-user engagements are based on the SPP guiding principle that "GRDI Projects will be designed with a view to knowledge translation and technology transfer so as to ensure the creation of value for the intended end-users. These end-users will influence the work, from the design of the project through to the rollout of the final product(s). The intended impact and how it may be measured will be outlined at the project design phase."Footnote 72 Some of the proposed solutions include:

  • Designing an outcome-driven project: Project Charters should clearly illustrate how the project intends to create, deliver and capture value – "It can "tell the story" of your project, with every piece fitting together in a unified vision/picture. It should map out all the components necessary to optimize success."Footnote 73 For example, using something similar to a business model canvasFootnote 74 can help visualize how one could design their project.Footnote 75
  • Develop a "Value Proposition": One should be able to clearly articulate "what is it you're doing/offering, for whom and how is it useful?"Footnote 76
  • Explore the best ways to engage end-users: The IMS strongly suggests that the Project Team, with the assistance of designated business / commercial advisors from as many participating GRDI DAs as want to be involved, should create a potential end-users' list/database. "Initially the list would serve the purpose of identifying organizations/individuals from whom input would be solicited on project design, but could evolve over the course of the project to be used to help develop plans from future consultation or feedback on progress, to identify potential collaborations or licenses, or for any other engagement purpose."Footnote 77 Some questions to consider at this phase include:
    • How can the end-users help you design your project to ensure it meets their needs?
    • What relationship do you need to build with those end-users to ensure they will actually use your end product/solution?

EQ 6: To what extent have the GRDI partners been successful at selecting, managing, collaborating, and achieving results as part of interdepartmental research projects?

Summary: The selection and management of the SPPs were highly praised by most of those consulted for the evaluation. However, a few respondents (including interview and case study respondents) from two DAs had concerns about the selection of the SPPs. While the process for developing and approving the Phase V SPPs is well articulated, there is a lack of clarity regarding the selection of sub-projects. The lack of clarity regarding how sub-projects are selected and the potential for the perception of conflict of interest (with respect to Phase VI) were both raised by these few respondents as problematic in the selection of SPP sub-projects. . The evaluation could not definitely determine how the criteria in the GRDI Framework were applied to the selection of sub-projects.

Management of the SPPs was positively received, including the overall governance for SPPs, the role of Management Advisory Committees (MACs), the empowerment of scientists in the management of their projects, and project management approach and tools (such as the project charter). Mid-year reporting was seen by QIS case study respondents to be overly burdensome.

Because SPPs are only funded for 5 years but address large, complex, cross-discipline issues across DAs, some SPP case study respondents expected some work to continue on these topics when GRDI funding ends. However, the case studies revealed there are no formal plans for follow up identified related to what happens now that the funding has ended. This introduces a risk that the results achieved to date and the potential for further impacts will not be maximized given the large investments of time and money made during Phase V and the two subsequent years the SPPs were funded.

The evidence from the administrative data and the bibliometric study illustrates that the SPPs had a high degree of collaboration. The number of collaborations with other government departments exceeded targets and the bibliometric study found there had been a large number of interdepartmental collaborations (including 69% of QIS publications and 49% of FWS publications showing co-authorship with other SPP participating DAs, respectively). Interview respondents and other sources (such as success stories prepared by the Secretariat and the case studies conducted for the evaluation) indicate that the SPPs have greatly increased interdepartmental collaborations and that the impacts of the SPPs likely would not have occurred without the interdepartmental nature of the projects.

Selection of SPPs

According to the ADM CC Terms of ReferenceFootnote 78, decisions regarding project selection (among other activities) are to be guided by defined criteria. The Governance Framework for Phase V outlines decision-making criteria to advance GRDI objectives, for the shared priority areas and complementary criteria to select programs/projects, including for mandated research projects. Table 8 presents these three sets of criteria.Footnote 79

Table 8: Decision-making criteria
Criteria to advance GRDI objectives Additional criteria to guide the selection of shared priority areas Additional complementary criteria to select programs/projects
  • Strategic importance or urgency of the opportunity/issue
  • Alignment with Government priorities
  • Relevance to role of federal S&T
  • Impact/benefits for Canadians (economic, social, and environmental)
  • Area of scientific strength, competitive edge in Canada and internationally
  • Importance of an integrated federal genomics R&D approach
  • Capacity in more than one GRDI department/agency
  • Appropriate rationale, goals and approaches
  • Originality and innovation
  • Scientific leadership
  • Strong multidisciplinary integration with GRDI and outside partnerships
  • Capacity of the team, likelihood of success
  • Quality of proposed management
  • Leverage of funds
 

According to the Governance Framework, the ADM CC agreed upon the two themes for Phase V SPPs. The WG, with support of the Secretariat, organized "workshops and meetings with research directors and senior research staff that have appropriate expertise in the identified themes for the purpose of identifying the specific research focus and the key research staff from the participating departments to lead the development of research project objectives and to build comprehensive project proposals, including work and expenditure plans."Footnote 80 As stipulated in the Flow Chart (Annex 6 to the Governance Framework), in addition to approving the themes, the ADM CC reviews and approves the research focus and high-level objectives; the project proposal; and the project charters of each SPP. The detailed project proposals, which include the scientific details of all the sub-projects that will deliver on the project goals, are "sent out for external scientific peer review to validate the proposed scientific approaches and for quality control and enhancement purposes. Input from the peer review is integrated into the … project charters that must be approved by the ADM CC prior to implementation."Footnote 81

It is not clear from program documentation provided for the evaluation what criteria are applied for the selection of sub-projects, although the Governance Framework does say that the "distribution of funds [for each SPP] reflects existing strengths and activities, and supports specific departmental contributions to the selected shared projects. The first year is devoted to planning and initiating the projects through workshops and meetings, organizing external peer reviews, hiring qualified personnel, and finalizing formal project charters."Footnote 82

In addition to the Governance Framework, the SPPs are provided with regular guidance by the GRDI WG and specific feedback from the ADM CC at the various stages of the SPP development. Ultimately, as per the GRDI SPP model, the proposals are directed by a Scientific Coordinator identified from a lead department, with support from Sub-Project Leads. The GRDI Governance Framework for Phase VI specifies that the Scientific Coordinator has the overall responsibility for the project.

Most respondents of all types felt that the selection of interdepartmental research projects (i.e., SPPs) was appropriate and effective. While the approach was complex and required the input of many, most respondents felt it was the most appropriate approach and was ultimately effective at identifying the topics and narrowing them down to projects that could be worked on interdepartmentally for maximum impact. Once the SPP themes were selected, most case study respondents felt that the bottom-up model with strong support from management was particularly effective. That is, SPPs were largely developed by scientists (which ensured that milestones and deliverables were realistic and leveraged interdepartmental collaborations) and then backed by managers and senior management (including linkages to DA priorities and end-user objectives, as well as funding and other support from the DA).

Despite this, a few respondents from two DAs were not satisfied with the selection approach. Specifically, these few interview respondents as well as some scientists interviewed for the QIS case study raised concerns about the transparency and level of input by scientists regarding the selection process for SPPs and sub-projects for Phase VI. For example, while scientists were invited to consultation meetings for the definition of Phase VI SPPs, these few scientists consulted for the QIS case study felt that their input was limited (this is in contrast to Phase V, where they felt their involvement in sub-project selection was more meaningful and contributed to better sub-project selection/design).

As well, a few respondents from two DAs felt that the selection of sub-projects lacked an objective review mechanism. As well, a few respondents raised the possibility that there could be a perception of a conflict of interest because the persons developing and selecting sub-projects (i.e., the Scientific Coordinator with input from Project Leads) could be seen to be acting in their own interest and/or the interest of their DA(s). Finally, a few respondents were concerned about the fact that the timing of the identification and development of sub-projects, which coincided with some scientists travelling for fieldwork, meant their input was not fully considered. These concerns and perceptions voiced by these respondents are consistent with the lack of clarity from the GRDI Governance Framework (outlined above) regarding how the criteria are to be applied in the selection of the sub-projects that contribute to the integrated SPPs.

Recommendation #1: The ADM CC should consider exploring and formally defining how SPP sub-projects are selected (including how and when the input of scientists is considered). Part of this selection process should consider and evaluate the potential for conflict of interest during the approval process as well as the input of scientists.

Management of SPPs

The management of both Phase V SPPs was highly praised by the large majority of those consulted by the evaluation. The document review found that the interdepartmental projects have a strong governance structure guiding and supporting their implementation. According to the 2013-14 Annual Report "both shared priority projects have detailed governance structures in their Project Charters to ensure seamless integration and clear roles and responsibilities."Footnote 83 These structures consist of Management Advisory Committees (MACs), including senior managers from each of the participating DAs, a Science Advisory Board with members representing academia, government and industry, theme leaders, dedicated project managers, and overall leadership by Scientific Project Coordinators. During the interviews, a few management respondents highlighted the MAC as being an effective mechanism for the management of the SPPs and a few suggested the body should play a larger role and/or meet more frequently.

Moreover, respondents to the QIS case study also provided positive feedback regarding the project's governance. The commitment at the ADM level to the project in particular was seen by respondents to be instrumental and considered a component of project success because it brought attention and focus to the common goals of the project. The MAC was considered complementary and helpful to ensuring all DAs had the same vision, took ownership, and committed to deliver on the project and support its outcomes. However, respondents suggested SPP project managers and coordinators should sit on the Interdepartmental WG because the MAC does not meet often. Most FWS case study respondents also provided positive feedback regarding the project's governance, but did not elaborate.

Further, ongoing communication is sustained through conference calls, emails, presentations, and regularly scheduled meetings, to share updates and provide decision-making fora. Both projects also use web-based SharePoint sites to store project-related documents for access by all project participants and advisory boards.Footnote 84

Key to the success of the projects was the empowerment of scientists in the management of projects and their willingness to collaborate and to share data/information with each other. Given the focus on shared outcomes and that each project depended on deliverables developed by several participating DAs, there was a mutual understanding that project teams needed to deliver individually for the overall project to be successful.

QIS case study respondents also felt that the project management approach and tools (e.g., project charter, guidelines, practice notes) were useful and contributed positively to the success of the QIS project by setting clear expectations, timelines, and deliverables, and facilitating the sharing of Standard Operating Procedures (SOPs) and other materials. FWS case study respondents did not comment specifically about the project management approach and tools.

One area of concern highlighted by some QIS case study respondents was that mid-year reports were found to be a heavy burden to complete. Instead, it was suggested that a full annual report and a brief mid-year report would be sufficient to track activities and progress towards outcomes.

Also, interviews conducted with principal investigators and scientists for both SPPs revealed inconsistent and unclear views of what will happen once the GRDI funding for the projects ends. While most said some work would continue and that efforts would be made to collaborate to the extent possible, the degree of these efforts and collaboration would be necessarily less than when the SPPs were funded. Thus, there is a risk (identified by a few respondents and inferred from the evidence) that the work done to date on these projects will not continue to occur in a timely manner and that work will be less efficient or effective. The result would be that the funding spent on the SPPs would be wasted when the projects do not result in any impact.

Recommendation #2: The ADM CC should consider requiring SPPs funded in Phase VI and beyond to develop a plan exploring how the technology/knowledge developed during the project will be transferred to, and used by, end-users outlining follow-on work that should be conducted by the partners to ensure such transfer and identifying potential funding sources to maximize uptake.

Degree of collaboration within SPPs

According to administrative data supplemented with case study evidence, the SPPs involved more than 80 different federal research scientists and resulted in over 430 collaborations, over 270 peer reviewed publications, and fostered partnerships with federal DAs, external academic institutions and international trading partners for standard setting.

As part of the evaluation, the NRC Knowledge Management unit conducted a bibliometric study to examine the degree of collaboration within and outside the networks of GRDI's two SPPs. This study revealed that both SPPs resulted in a large number of interdepartmental collaborations.

For the QIS project, there were six GRDI-participating DAs involved in the QIS publications: AAFC, CFIA, DFO, ECCC, NRCan and NRC. The bibliometric study found that some DAs collaborate more often than others. For example, while CFIA and AAFC are seen to collaborate frequently, DFO and ECCC seldom collaborate with other federal organizations. CFIA collaborates frequently with NRCan (also with AAFC and NRC), while AAFC collaborates frequently with CFIA and NRCan.Footnote 85

Using a sample of 65 sub-projects, the analysis of QIS found that 69% of publications showed collaboration with at least one other institution (including other government DAs, academic institutions, provincial governments or private sector companies), with approximately a quarter of these with three or more affiliations with institutions. On average, each project publication had eight institution affiliations and involved 19 scientists.Footnote 86 Although QIS has a high number of publications, the number of frequent co-publications (more than 3 co-publications) with other organizations was low.Footnote 87

For the FWS project, there were five GRDI-participating DAs associated with the publications: PHAC, NRC, CFIA, HC and AAFC. The bibliometric study found that PHAC is the most active DA followed by NRC and HC. PHAC collaborates frequently with other DAs, but works the most with a network of seven non-federal government institutions (namely University of Lethbridge, University of Lisbon, Dalhousie University, Simon Fraser University, University of Maryland, University of British Columbia (BC) and the BC Public Microbiology Reference Laboratory). NRC and HC are bridges between collaborating DAs but are bridging to other collaborating DAs through a low number of co-publications (usually 2-3 co-publications). NRC's scientists co-published more with other NRC's scientists than with scientists from other DAs.Footnote 88

The analysis of the FWS SPP found that 47% of publications consisted of collaboration with at least one other institution, and approximately a quarter of publications had three or more institution affiliations. On average each project publication had seven DA affiliations and 18 scientists (authors). This indicates that when publications are co-authored by at least two scientists, there is generally cross-DA collaboration.Footnote 89

It was the opinion of most of those consulted for the evaluation, through interviews and SPP case studies, that the SPPs greatly increased interdepartmental collaboration by funding projects that included the participation of scientists from multiple DAs. Moreover, respondents agreed that the impacts realized by the SPPs could not have been achieved by one DA working on its own due to the nature of the work conducted through the SPP sub-projects.

The success stories prepared by the GRDI Secretariat highlight the SPPs' effectiveness in enabling cross-government collaborations. The SPPs funded under GRDI demonstrate strong collaborative efforts involving researchers from a number of federal DAs. The collaborations fostered under GRDI shared projects have played a role in linking and sharing expertise among scientists and accelerating progress. This is reflected in the following quote from one of the lead researchers involved in GRDI FWS project:

"The GRDI enabled researchers in different departments to reach out to one another and combine their expertise. For example, [a] biologist …and her team at Health Canada played a key part in developing the existing test, and [a co-investigator] and I have also relied on her expertise to get us where we are today. I don't think that kind of collaboration would have been possible without the GRDI."Footnote 90

The key informant interviews explored facilitating and hindering factors for interdepartmental collaboration. In terms of facilitating factors, those most commonly mentioned by respondents included: strong leadership within projects, including having credibility, ability to influence, great communication skills and the time to invest in the project and its leadership; and the project lead was effective at engaging all the players. Similarly, several different hindering factors were suggested, including these mentioned by at least two respondents: the fact that many scientists were accustomed to working independently rather than in collaboration; and bioinformatics capacity was limited, not consistent across DAs and is uncertain in the future.

Results of SPPs have been presented as part of Questions 4 and 5.

EQ 7: What lessons learned and best practices can be identified in terms of interdepartmental research collaboration based on the two projects funded as part of Phase V?

Summary: Many of the key lessons learned from the two Phase V SPPs were discussed at a meeting held in November 2014 and summarized in the IMS. The IMS was used during the Phase VI SPP development and planning. Lessons learned centered on the importance of, and best practices in, end-user engagement, mechanisms for sharing of data/materials and requirements regarding building bioinformatics capacity and the associated infrastructure, and the requirement for intellectual property (IP) tools and proactive management of IP. In addition, the SPP case studies conducted for the evaluation confirmed these key lessons/best practices and also identified a few other lessons and challenges, including: the need for better information sharing among collaborators; more time/longer timeframe for projects; the need for a group of subject matter experts to support SPPs; challenges with interdepartmental financial and administrative issues; and challenges with hiring of highly qualified personnel (HQP).

In November 2014, stakeholders discussed challenges pertaining to the first years of the interdepartmental projects at a half-day GRDI Strategic Planning Meeting. The purpose of this meeting was to "identify priority issues for resolution, solutions and best practices put in place by the existing projects and practical solutions for the next cycle of shared priority projects."Footnote 91 The challenges brought to the attention of GRDI WG by members of the teams of the first SPPs ranged from the management of information flowing into the projects to the management of research outcomes arising from the projects.Footnote 92 Many of these challenges were also highlighted during the consultations conducted for the SPP case studies for the evaluation.

A meeting was held in 2014 to discuss these challenges and GRDI subsequently developed an IMS for SPPs for application in Phase VI. The strategy focuses on three challenges/opportunities for improvement: end-user engagement, sharing of data and materials, and intellectual property tools and management. Ultimately, the IMS has been developed to enhance the value of GRDI research and ensure its results meet the needs of end-users. These three challenges/opportunities for improvement are explained below along with suggested change(s) for the next phases of GRDI.

  • End-user engagement. This challenge was noted for both SPPs in the documents as well as through the SPP case studies in terms of engaging early adopters to fully identify their needs. The documentary and case study evidence for the QIS project also noted challenges engaging end-users to validate the value of project research results. With respect to the first challenge (early engagement in order to identify needs), suggestions for improvement included: ensuring there is enough time when putting the research proposal together to identify, and discuss it with, potential end-usersFootnote 93; provide funds and opportunities to bring end-users and GRDI scientists together face-to-face very early on to establish and maintain strong relationships; and ensure project charters describe the benefits to Canadians and alignment with the GRDI logic model, etc.Footnote 94 In terms of the second challenge (validation of research results), QIS researchers noted that the main obstacle was the limited time to develop these relationships given that these projects were new to the DAs involved. Further, there was no public outlet (i.e., portal) for the information to be accessed at the beginning of the project. One of the potential solutions was to learn from Genome Alberta's requirement to organize a focus group session with end-users at the end of the project, to demonstrate that researchers have addressed the needs identified by end-users in the planning stage and ensure that knowledge and tools are disseminated.Footnote 95
  • Sharing of data and materials/bioinformatics capacity. Stakeholders identified difficulty in sharing biological samples among charter participants. More specifically, both SPPs experienced issues in using existing biological samples that came from multiple sources. This resulted in reduced productivity and delays on the deliverables, as well as extensive frustrations and time-wasting.Footnote 96 Difficulties related to the integration of prospective biological sample collection financed by other sources other than GRDI were also noted by both projects. Better alignment of other DAs and programs that conduct relevant field sampling would have been beneficial. Some potential solutions to these issues included: ensuring that there is enough time when putting the research proposal together to allow for discussions with potential partners and end-users; and providing a Material Transfer Agreement Template, and ensuring these are completed before the start of the project.Footnote 97

    Both projects also reported difficulties related to the protection of data integrity and provenance related to shared biological samples (i.e., accessing voucher samples and having different data processing solutions across different data sources). It was suggested that a mechanism to protect data integrity in case of change (e.g., new taxonomic determination) should be implemented when collections come from non-federal partners.Footnote 98

    Those involved in the FWS and QIS both experienced issues in sharing information and managing the use of information derived from a variety of sources. Data storage and exchange for bioinformatics constituted the major bottleneck for genomics projects and developing an integrated infrastructure common to all participating DAs was a challenge. The participating DAs were not on the same network and did not have the same connectivity features (e.g., bandwidth). Taking the QIS project for example, while CFIA and AAFC were on the same network, NRCan and DFO were on another network. As a result, participating DAs had difficulties accessing each other's computing facilities for data transfer and exchange.

    To overcome these issues, the evaluation found that Shared Services Canada (SSC) should be engaged early in the SPP lifecycle to provide solutions for shared access, and/or to set up a more dynamic system using SharePoint or Google Drive. Also, the Strategic Planning Meeting (from November 2014) report suggested that the bioinformatics challenge be presented to Chief Information Officers, informing them of capacity requirements. Finally, it was suggested that third party long-term solutions (e.g. Compute Canada, Joint Genome Institute) as well as collaborative solutions with government-wide initiatives (such as Treasury Board Open Science), etc., be explored.Footnote 99 Respondents for the QIS case study also suggested that more should be done to inform SSC (or the external service provider, more broadly) of the needs and priorities of GRDI projects so that projects can deliver according to the project plan.

    Based on the GRDI Secretariat, negotiations are currently underway with SSC for the provision of bioinformatics support for GRDI projects. While it is understood there will be a significant annual cost to GRDI, the amount is not yet known.

  • Intellectual Property (IP). The IMS outlines a number of principles regarding IP, including prompt disclosure and that the management and exploitation of IP "are to be undertaken in the best interest of Canada in the context of GRDI's goals."Footnote 100 While results of SPPs are expected to be disseminated as openly as possible and to as many end-users as possible, the IMS acknowledges that there may be times when the commercialization of outputs may be appropriate and thus wide dissemination of results would not be undertaken in favour of the private sector organization licensing the technology. The IMS further clarifies where control and administration of IP rests (e.g., with the DA or DAs whose scientists made a contribution to the IP).

Other opportunities for improvements identified through the SPP case studies included:

  • Better information sharing among collaborators. While partners within specific projects shared reports, some respondents for both case studies felt that more teleconferences would have been beneficial. A few indicated that there was more interaction upfront to determine the roles and responsibilities of each team than there was during the later stages of the project. It was suggested that perhaps a conference call every six months would have addressed the communication gap by offering opportunities to share ideas and methods, and help collaborators to answer questions such as "how can I fix this?" Also, data collection for case studies revealed missed opportunities to collaborate between SPPs and selected mandated projects (i.e., where the research topic was the same/similar).
  • More time/longer timeframe. A few respondents mentioned that projects of this complexity require more than five years (one SPP identified they would need up to eight years) to achieve its main objectives. While some work will continue after March 31st 2016 (which is the end of GRDI funding for the project), some researchers wonder how these aspects of the project will be completed in the absence of the GRDI funding support.
  • Need for support to SPPs by a group of subject matter experts (SMEs). According to the IMS, these SMEs could be called upon, as required, to support GRDI SPPs to facilitate effective project development and execution. This community of SMEs would be guided by the Innovation Management Guiding Principles. SMEs would have expertise, or knowledge of, one or more of a number of areas, such as end-user engagement, open source software, intellectual property, etc.
  • Better management of interdepartmental financial and administrative issues. In the QIS project, ECCC and DFO had initial challenges in establishing sequencing capacity and expertise as per their original agreement. This agreement approved the hiring of a shared post-doctoral researcher jointly paid by DFO and ECCC. However, it took nine months and a large amount of time/paperwork to approve the transfer of funds from ECCC to DFO. It was suggested that, in the future, financial and administrative processes involved in the delivery of the project should be verified (pre-arranged and pre-approved) and harmonized across participating departments and agencies.
  • Simplified/improved hiring of HQP. According to respondents, while specific to GRDI, the hiring of HQP was complicated and impacted the delivery of the QIS project. Genomic R&D highly depends on the contribution of post-docs. Hiring through the regular Government of Canada human resources (HR) staffing process can take up to a year. This constitutes a barrier to project success because federal scientists need to hire post-docs on short notice when they are available and ready to work. Going forward, a more flexible HR model of HQP would be beneficial. In fact, such a model for this category of HQP is currently being pilot-tested across federal science-based DAs. The Postdoctoral Research Pilot Program led by NRCan includes the necessary labour market exemptions to hire international and Canadian post-docs. Successful candidates are hired as term employees.Footnote 101

In terms of best practices, the SPP case studies emphasized the management and governance of the projects as being particularly effective and appropriate. Evidence on these aspects of the projects is presented above under Question 6 (management of SPPs).

Recommendation #3: The ADM CC should consider and implement additional opportunities to increase communication between DAs participating in SPPs to improve collaboration and joint problem solving, for example by supporting interactive mid-year sessions.

 

2.3 Performance – Efficiency and Economy

EQ 8: To what extent have the horizontal governance components of GRDI (i.e., ADM CC, Interdepartmental Working Group, GRDI Secretariat) been efficient?

Summary: All lines of evidence illustrated that the horizontal governance components of GRDI are efficient and effective. They are well communicated via program documents and appear to be well understood and praised among those consulted for the evaluation. Performance measurement is in place and improvements have been made since the last evaluation. However, a few challenges remain, including reporting inconsistencies, the lack of mechanisms to compile and/or track supporting evidence by project.

Governance mechanisms

Overall, the evaluation found that horizontal governance components of GRDI are efficient. The document review indicated that effective mechanisms have been developed that contribute to the efficiency of the horizontal governance structure. The interdepartmental Governance Framework established under the leadership of NRC for previous phases of GRDI continues to oversee the coordination of GRDI. According to the program's Annual Reports, participating DAs received timely Secretariat support and a specific governance framework. As well, management and operating processes were put in place for Phase V. In 2013-14, four meetings of the ADM CC and eight meetings of the GRDI WG were held to ensure collaborative decisions. Overall, the WG and ADM CC members found these meetings to be very effective, giving the GRDI Secretariat a satisfaction level of 97% on a client satisfaction survey.Footnote 102

As well, in 2012, GRDI was included in the Office of the Comptroller General's Horizontal Internal Audit of Compliance with the Policy on Management, Resources and Results Structures (MRRS). According to the 2012-13 APR, the audit concluded that: "GRDI's roles and responsibilities were well defined; accountability structures adequately designed; horizontal initiative activities monitored; and the results of the overall initiative were reported publicly."Footnote 103 Moreover, the audit did not include any recommendations for improvement to the administrative role of the NRC. Similarly, most management interview respondents felt that the horizontal governance of GRDI is efficient and appropriate. The factors noted by these respondents that reinforced this efficiency and appropriateness largely focused on the ADM CC and its willingness to listen to the needs of DAs and its ability to make decisions based on information and recommendations provided by all the players (such as the WG and the SPP teams). As well, while one respondent praised the ADM CC for its role in monitoring project results and commissioning of mid-term reviews, another respondent liked that the ADM CC stays out of the areas where they do not bring value (e.g., identifying possible mandated research projects). A few respondents also mentioned that the role of the WG was appropriate and that the Secretariat was efficient and supportive.

Performance measurement

In 2011, a horizontal PMS was developed for GRDI to formalize the commitment of the eight participating DAs in terms of the common measurement and accountability requirements associated with this program. It also serves to clarify the roles and responsibilities of the seven funded federal DAs in Phase V. The PMS considers relevant conclusions and recommendations resulting from the formative evaluation of GRDI completed in 2006, as well as the impact evaluation completed in 2010.Footnote 104

Overall, with respect to performance measurement, the 2016 evaluation found much work has been done since the last evaluation (e.g., development and sharing of templates to increase the standardization of the reporting of indicators). However, despites clear guidance for DAs to report actual impacts, there are instances in APR narrative sections that include anticipated impacts and the current system does not allow for compilation/tracking of supporting evidence at the project level (which could then allow for breaking down results by project size, sector, whether the project is a follow-up to an earlier project, etc.). Finally, although some data elements were reported inconsistently across DAs where guidance was not sufficiently detailed, some actions have taken place to mitigate this issue. For example, guidance documents have already been re-done for the planning of Phase VI SPPs that further clarify requirements.

Recommendation #4: The ADM CC should explore additional opportunities to improve performance measurement and reporting, including:

  • Ongoing monitoring on Phase V SPPs (and the next evaluation should also explore what has happened with these projects). SPP leads should be consulted regarding what would be the most meaningful set of indicators to track that are manageable in terms of reporting burden.
  • Implementing a database to capture information by project, which would allow for searching and analysis by key project characteristics (such as project type, impact area, DA, GRDI funding phase, GRDI funding, DA-leveraged funding, HQP, etc.) and would encourage projects with similar objectives to communicate with one another. This database should be accompanied by a detailed definition of variables and indicators to be reported.
  • Streamlining SPP reporting, for example by revisiting the number of indicators to be reported in SPP mid-year reports. Interactive mid-year sessions rather than detailed mid-year reports, and revisiting the number of indicators, are examples of opportunities for streamlining and improvements in communications.
 

EQ 9: To what extent does the current delivery model for GRDI allow for a cost-effective use of federal government resources? Are there alternatives that are more cost-efficient?

Summary: The evidence reveals that GRDI is cost-effective, as determined by the number of collaborations and the extent of leveraging, both of which exceeded the targets set out in the PMS. In terms of delivery, evidence from the interviews indicates that the current mix of mandated projects versus SPPs, the use of applying for funding via periodic funding cycles (using TB submissions), and the reliance on leveraging of DA funds were all deemed to be appropriate and an effective use of resources. No alternative cost-effective delivery approaches were identified by the evaluation. Some respondents suggested that the overall GRDI funding envelope should be increased (note that respondents were not asked to comment on the adequacy of funding but rather volunteered this information when asked about alternative delivery approaches).

 

Overall, the evaluation found that GRDI is cost-effective and the delivery of the program is appropriate and efficient. According to the evaluation matrix (Appendix A), cost-effectiveness was measured by exploring the number of collaborations and the degree of leveraging of funds (i.e., the value of non-GRDI funding in support of GRDI projects from A-base and other sources). The administrative data reveals that DAs reported a total of 1,234 formal collaborations over the period covered by the evaluation. This figure exceeds the target of 792 collaborations set out in the PMS. In terms of leveraging, as presented under Question 1, participating DAs leveraged $89 million, which exceeded the target of $60 million and accounted for 60% of total DA investments in Phase V projects.

In order to assess the delivery model, three components were specifically explored: funding projects on a 3 and 5-year cycle via TB submissions; the mix of SPPs and mandated research projects; and the use of leveraging A-base funds to complement GRDI project funding. The existence of alternative delivery models was also probed during interviews.

Mix of funding for SPPs and mandated projects

The current proportional mix of funding for SPPs and mandated projects was generally seen to be appropriate by most respondents. However, some of the respondents interviewed for the evaluation wanted more funding for SPPs because they believed the funding would have larger impacts through the conduct of additional and/or larger projects. On the other hand, some management respondents felt strongly about maintaining the same level of funding for mandated projects since mandated research projects help DAs address important needs (as highlighted above). A few scientists wanted to see more funding for mandated projects since A-based funding for research has been declining for years.

The evaluation confirmed that SPPs require a high level of funding due to the complexity of the problems and to ensure collaboration occurs. While originally scheduled to occur over three years, SPP funding was planned for five years to increase the likelihood that the projects would realize their expected impacts. Respondents for both SPP case studies indicated that five years of funding was the minimum required to realize impacts for SPPs.

A-base versus funding cycles

Most consulted for the evaluation supported the status quo of using periodic funding cycles via TB Submission rather than A-base. The most commonly mentioned reason for this is the concern that A-base funding can more readily be diverted to address evolving DA priorities. Funding cycles encourage competition, and review by committees and ensures high quality research is occurring. Review committees validate priorities and investments to drive high quality research. While there are downsides (in terms of the uncertainty of the availability of funds, that those who do not receive funding are shut-out for 3 or 5 years, and the level of effort required to access the funds), there are more advantages and fewer risks with the current approach.

Leveraging

Leveraging of DA funds was seen to be appropriate and a good mechanism for cost-efficiency. All participating DAs have leveraged GRDI funds with allocations from their A-base resources and from successful collaborations. In both 2012-13Footnote 105 and 2013-14 non-GRDI funds represented approximately twice GRDI investments, with additional in-kind investments consisting of sharing of technology platforms, materials, and expertise with a variety of collaborators in research areas that cut across traditional DA sectors.Footnote 106 The total amount of leveraged funds ($89M) exceeded the target of $60M and accounted for 60% of total investments in Phase V.

However, the degree of leveraging of funds outside of the federal government A-base appears to vary based on DA and project. According to the interview results, this type of leveraging does not appear to be widespread. When this type of leveraging does occur, it is usually with Genome Canada funded scientists where projects are co-funded with GRDI. A few management interview respondents felt that more should be done to encourage collaboration with other funders and improve alignment with other funders to encourage more leveraging.

Alternative delivery models

The evaluation did not identify any alternatives to GRDI's delivery approach. While not related to alternative delivery models, some management interview respondents wanted to see GRDI's funding envelope increase to have a greater impact. Of these, a few felt that any additional investments should be put into SPPs rather than mandated projects (since, as highlighted above, SPPs were seen to have large-scale impacts). Some of the management respondents pointed to the fact that the budget has not increased since the program started and that purchasing power of the budget is much lower (as noted, the real value of the $19.9 million annual investment has declined to $13.36 million in 2015). Note that respondents were not asked to comment on the adequacy of funding but rather volunteered this information when asked about alternative delivery approaches.

3.0 Conclusions and Recommendations

3.1 Conclusions

The evaluation found that, while the needs of DAs for genomics R&D funding have evolved, there is a continued need for GRDI. For many DAs, the need continues to relate to capacity building (in both basic and advanced techniques). Several DAs have also seen the need move towards the application of genomics-based technologies and approaches. In addition, there is clear alignment of GRDI with the mandates and priorities of participating DAs and the federal government more generally. The evaluation confirmed that the program is consistent with, and contributes to, participating DAs' legislated mandates related to the health and safety of Canadians as well as the sustainability of Canada's natural resources (via regulatory activities) and support for industry (via economic development and regulatory activities). The evaluation further found that there is little duplication with other similar organizations or federal programs.

In spite of large amounts of funds being leveraged and some DAs making investments in genomics R&D, it is clear that the need for continued support for these types of projects does not show signs of abating. In the absence of GRDI, the evaluation found that DAs' abilities to deliver on their mandates in the future might be negatively impacted as it is unclear what amount of DA funding for genomics R&D would be available given other DA priorities.

The continued need for GRDI is taking place within a context of a decreasing real value of the funding (due to inflation since inception in 1999 without commensurate increases in funding), funding being shared among more participating DAs, SPPs vying for part of the GRDI funding envelope and increasing costs associated with the research. Research costs are increasing because more effort is now being focused on applications and transferring knowledge to end-users, which are more resource intensive phases of R&D.

In terms of effectiveness, the evaluation found that Phase V projects have been successful in the development of innovative knowledge and technologies and influencing evidence-based public policy. The program has produced expected outputs and exceeded targets related to these. Moreover, the evaluation confirmed that many GRDI projects have successfully transferred knowledge and technologies to end-users, both internal and external to the federal government. Many more GRDI projects have the potential to similarly impact end-users. Nevertheless, there continue to be opportunities, especially for the SPPs, to build on the lessons and best practices from Phase V in subsequent phases of the program. Some of these have already been addressed. In particular, content outlined in the IMS, though intended for SPPs, offers examples and tools to ensure end-user engagement and take-up of GRDI-funded knowledge and technologies that could be considered beyond the SPPs.

In addition to the transfer of knowledge and technologies, the evaluation found evidence that GRDI-funded projects are likely to have real and lasting longer-term impacts, most of which remain anticipated or potential impacts at this time. These impacts are significant and are expected to lead to billions of dollars in benefits for end-users (through avoided costs, increased trade, etc.). More specifically, the evaluation found that GRDI-funded projects are likely to result in: improved public health and wellness; avoided health system costs; efficiencies and avoided losses for the public and private sectors; environmental sustainability; improved detection of invasive species; improved strains/traits of plants, trees and animals of commercial value; and improved fisheries tracking and management.

The introduction of interdepartmental SPPs in Phase V has proven to be a strong feature of the program and the evaluation found that there is a continued need for this type of interdepartmental collaboration. The two SPPs funded in Phase V were found to be well-managed and have achieved significant results, which likely would not have occurred in the absence of GRDI funding for the interdepartmental nature of the projects. As well, the degree of interdepartmental collaboration stemming from the SPPs has been significant. However, there is no consistent or planned approach related to what will happen to ongoing SPPs when the GRDI funding ends. Consequently, there is a risk that the results achieved to date and the potential for further impacts might be slowed or otherwise diminished. Some concerns were also raised about the selection of SPPs (including the role of scientists), and a review of related documents confirmed some lack of transparency in terms of how sub-projects are selected. As well, since sub-projects are ultimately directed by the Scientific Coordinator from one participating DA (with support from Sub-Project Leads), there is the potential for the perception of conflict of interest. Finally, the QIS case study suggests that mid-year reporting may be overly burdensome (on top of annual reporting). Both case studies brought the suggestion that there should be more interaction between the various scientists contributing to the SPP goals.

Other challenges associated with the SPPs included those pertaining to: end-user engagement; sharing of data/materials and building bioinformatics capacity; management of IP; short timeframe for projects of this magnitude; interdepartmental financial and administrative issues; and delays associated with the hiring of HQP. The IMS was developed with a focus on the first three challenges in particular (among other matters), and is being applied during Phase VI SPPs.

In terms of efficiency and effectiveness, the evaluation found that the program is both efficient (in terms of its horizontal governance) and effective (in terms of its use of collaborations, leveraging of funds and delivery approach). The evaluation did not reveal any significant opportunities for improvement in program governance or delivery. As well, the evaluation found that the program has made improvements in terms of its performance measurement approaches and systems. However, a few challenges remain, including reporting inconsistencies and the lack of mechanisms to compile and/or track supporting evidence by project.

3.2 Recommendations

GGI makes the following recommendations:

Recommendation #1: The ADM CC should consider exploring and formally defining how SPP sub-projects are selected (including how and when the input of scientists is considered). Part of this selection process should consider and evaluate the potential for conflict of interest during the approval process as well as the input of scientists.

 

Recommendation #2: The ADM CC should consider requiring SPPs funded in Phase VI and beyond to develop a plan exploring how the technology/knowledge developed during the project will be transferred to, and used by, end-users outlining follow-on work that should be conducted by the partners to ensure such transfer and identifying potential funding sources to maximize uptake.

 

Recommendation #3: The ADM CC should consider and implement additional opportunities to increase communication between DAs participating in SPPs to improve collaboration and joint problem solving, for example by supporting interactive mid-year sessions.

 

Recommendation #4: The ADM CC should explore additional opportunities to improve performance measurement and reporting, including:

  • Ongoing monitoring on Phase V SPPs (and the next evaluation should also explore what has happened with these projects). SPP leads should be consulted regarding what would be the most meaningful set of indicators to track that are manageable in terms of reporting burden.
  • Implementing a database to capture information by project, which would allow for searching and analysis by key project characteristics (such as project type, impact area, DA, GRDI funding phase, GRDI funding, DA-leveraged funding, HQP, etc.) and would encourage projects with similar objectives to communicate with one another. This database should be accompanied by a detailed definition of variables and indicators to be reported.
  • Streamlining SPP reporting, for example by revisiting the number of indicators to be reported in SPP mid-year reports. Interactive mid-year sessions rather than detailed mid-year reports, and revisiting the number of indicators, are examples of opportunities for streamlining and improvements in communications.
 

4.0 Management Response and Action Plan

Recommendation Response Planned Action(s) Responsibilities Expected Completion (M/D/Y)
Recommendation #1: The ADM CC should consider exploring and formally defining how Shared Priority Project (SPP) sub-projects are selected (including how and when the input of scientists is considered). Part of this selection process should consider and evaluate the potential for conflict of interest during the approval process as well as the input of scientists. Accepted The GRDI will update its Governance Framework and articulate the SPP planning process more clearly, particularly vis-à-vis sub-project development and selection. GRDI ADM CC with support from the WG and Secretariat 06/30/17
Recommendation #2: The ADM CC should consider requiring SPPs funded in Phase VI and beyond to develop a plan exploring how the technology/knowledge developed during the project will be transferred to, and used by, end-users outlining follow-on work that should be conducted by the partners to ensure such transfer and identifying potential funding sources to maximize uptake. Accepted SPPs will include a technology/knowledge transfer plan that identifies clients and end users who will be engaged throughout the project. GRDI ADM CC with support from the WG and Secretariat 03/31/2017
SPP teams will periodically review and update the initial technology/knowledge plan in order to maximize uptake. At the end of the project funding period, GRDI will identify follow-on work required to fully implement the plan, identifying potential resources and fostering external linkages as appropriate. GRDI ADM CC with support from the WG and Secretariat 03/31/2019
Recommendation #3: The ADM CC should consider and implement additional opportunities to increase communication between Departments and Agencies (DAs) participating in SPPs to improve collaboration and joint problem solving, for example by supporting interactive mid-year sessions. Accepted Internal communications plans to facilitate collaboration and joint problem solving will be required in SPP Management Plans. GRDI will support additional interactive team sessions. GRDI ADM CC with support from the WG and Secretariat 03/31/2017

Recommendation #4: The ADM CC should explore additional opportunities to improve performance measurement and reporting, including:

  • Ongoing monitoring on Phase V SPPs (and the next evaluation should also explore what has happened with these projects). SPP leads should be consulted regarding what would be the most meaningful set of indicators to track that are manageable in terms of reporting burden.
  • Implementing a database to capture information by project, which would allow for searching and analysis by key project characteristics (such as project type, impact area, DA, GRDI funding phase, GRDI funding, DA-leveraged funding, HQP, etc.) and would encourage projects with similar objectives to communicate with one another. This database should be accompanied by a detailed definition of variables and indicators to be reported.
  • Streamlining SPP reporting, for example by revisiting the number of indicators to be reported in SPP mid-year reports. Interactive mid-year sessions rather than detailed mid-year reports, and revisiting the number of indicators, are examples of opportunities for streamlining and improvements in communications.
Accepted In consultation with Phase V and Phase VI SPP leads, GRDI will develop meaningful methodologies for examining the ongoing legacy and impact of SPPs. SPPs will be required to identify end-users and stakeholders that have benefited, or could benefit, from project outputs for follow-up assessments. GRDI ADM CC with support from the WG and Secretariat 12/31/2016
An adoption and impact study of Phase V SPPs will be undertaken as part of the next GRDI evaluation. GRDI ADM CC with support from the WG, IEWG, and Secretariat 03/31/2021
GRDI will define key project characteristics to be reported, and develop a database which can be used to foster interdepartmental communication and support reporting activities. GRDI ADM CC with support from the WG and Secretariat 06/30/2017
GRDI will streamline SPP mid-year reporting to the ADM CC and will revisit the variables and indicators reported annually for mandated and SPPs. GRDI ADM CC with support from the WG and Secretariat 03/31/2017
 

Appendix A: Evaluation Matrix

Evaluation Questions Judgment Criteria Indicators Document / Literature Review Review of Administrative Data Key Informant Interviews Case Studies
Relevance
1.How have the needs of federal departments / agencies (D/A) for GRDI funding evolved since its inception and to what extent is the Initiative still aligned with current needs?
To what extent is there a continued need for interdepartmental research collaboration (i.e., shared priority projects)?
  • The Initiative is addressing the current needs of the GRDI partners in terms of genomic research and these needs would not be met in the absence of GRDI.
  • Interdepartmental research collaborations are addressing needs that could not be met as part of GRDI mandated research projects or by other federal government programs.

1.1 Documentation / federal government stakeholder views of the evolution of GRDI and the program's current ability to address their needs.

1.2 Evidence from the literature / stakeholder views regarding the use of genomics by government organizations in support of their mandate.

1.3 Federal government stakeholder views on their ability to achieve their mandate in the absence of the GRDI program.

1.4 Documentation / federal government stakeholder views on the specific needs addressed by the shared priority projects.

X - X X
2. To what extent do the activities and outcomes of GRDI align with the priorities of the federal government and the priorities of partner departments and agencies?
  • The activities and outcomes of the GRDI projects show a strong level of alignment with Government of Canada priorities and with the mandates of the GRDI partners.

2.1 Evidence that processes / mechanisms were established by the partners to ensure that the funded projects align with the priorities of the government and the mandate of GRDI partners.

2.2 Federal government stakeholder views on the level of alignment of GRDI projects with the mandate of their department / agency.

2.3 Linkage of GRDI activities and outcomes to Government of Canada priorities (i.e., as described in the 2014 Federal Science &Technology Strategy, recent federal Budgets, Speeches from the Throne, etc.).

X - X X
3. To what extent is GRDI consistent with federal roles and responsibilities?
  • Genomic R&D investment in support of federal government mandates is consistent with federal roles and responsibilities.
  • GRDI complements the genomic R&D activities performed at the federal government level and does not duplicate the activities of other organizations.

3.1 Documentation / stakeholder views regarding the appropriateness of federal involvement in genomic R&D.

3.2 Presence / absence of other organizations or federal programs that duplicate or that could perform the work conducted as part of GRDI.

X - X -
Performance - Effectiveness
4. To what extent have the research projects funded as part of Phase V contributed to the development of evidence based public policy and innovative knowledge / technologies?
To what extent and under what conditions were this knowledge and these technologies transferred to end-users inside and outside of the federal government?
  • GRDI has met its performance targets in terms of number of regulatory, policy and resource management decisions informed by GRDI research.
  • GRDI has met its performance targets in terms of knowledge and technologies produced as part of Phase V.
  • GRDI has met its performance targets in terms of dissemination / transfer of knowledge and technologies.
  • Knowledge / Technology transfer mechanisms have been effective in reaching targeted end users.

4.1 GRDI Performance indicators / targets re: knowledge and technology:

  • Scientific information and publications.
  • Research tools and processes.

4.2 Documented evidence of innovative tools and knowledge produced as part of Phase V

4.3 GRDI Performance indicators / targets regarding dissemination / transfer of knowledge (including factual information describing how project results have been used to inform decisions, changes and associated benefits):

  • Communication products;
  • Number of outreach activities for disseminating results to end-users;
  • Number and type of transfer activities;
  • Training of Highly Qualified Personnel (HQP); and,
  • Number and examples of innovative tools and processes that have been adopted in Canada based on GRDI research.
  • Number of regulatory, policy and resource management decisions informed by GRDI research.

4.4 Documentation / stakeholder views on GRDI's performance in terms of knowledge and technology transfer to end-users.

4.5 Success and hindrance factors that explain GRDI's performance in terms of knowledge / technology transfer to end-users.

X X X X
5. What impacts have been generated from the use of GRDI's research results by end-users and external stakeholders?
  • What success / hindrance factors have influenced the use of knowledge and technologies by end-users and external collaborators?
  • To what extent would those research results have occurred without the funding from GRDI?
  • The knowledge and technologies developed as part of GRDI projects have been adopted and/or used by end-users in support of federal government programs.
  • The use of the knowledge and technologies developed as part of GRDI projects have resulted in benefits to end users and Canadians.

5.1 Documentation / end-user views regarding the impacts of GRDI projects.

5.2 Evidence of likely long-term impacts stemming from GRDI projects in terms of:

  • Improved processes
  • Improved diagnostics/instrumentation
  • Improved products
  • Reduced harm (health), improved care, lives saved
  • Reduced health care costs (use of health case system);
  • Improved human health in Canada;
  • Other cost savings
  • Enhanced sustainability and management of Canada's environment, agriculture, forestry and fisheries sectors;
  • Reduced environmental harm
  • Improved food safety and security in Canada.
  • Economic benefits (sales, jobs, business creation)

5.3 Success and hindrance factors that influenced GRDI's ability to generate impacts on end-users.

5.4 Number and level of other contributing factors to the success of the projects, including other funding sources

5.5 Extent to which projects would have been completed in the absence of GRDI based on opinions and other contributing factors

X X X X
6. To what extent have the GRDI partners been successful at selecting, managing, collaborating, and achieving results as part of interdepartmental research projects?
  • The GRDI partners successfully selected planned and delivered interdepartmental projects that generated impacts on the partner D/As and other end-users.
  • The projects resulted in increased collaboration between the D/As and provided innovative solutions to problems that could not have been achieved by one department.

6.1 Documentation / stakeholder views on the effectiveness and efficiency of interdepartmental research projects.

6.2 Examples of impacts resulting from the projects in terms of increased interdepartmental collaboration and benefits to end-users.

6.3 Bibliometric Analysis – Network map of collaboration based on co-authorship.

X X X X
7. What lessons learned and best practices can be identified in terms of interdepartmental research collaboration based on the two projects funded as part of Phase V? -

7.1 Documentation / stakeholder views regarding the lessons learned and best practices stemming from the two interdepartmental projects.

X - X X
Performance – Efficiency and Economy
8. To what extent have the horizontal governance components of GRDI (i.e., ADM CC, Interdepartmental Working Group, GRDI Secretariat) been efficient?
  • The costs and level of effort to administer the horizontal management of the Initiative (i.e., the Secretariat) is deemed appropriate by the GRDI partners.
  • The governance structure is deemed by the partner D/As to be appropriate to support effective decision making while ensuring accountability.

8.1 Review of the administrative costs and level of effort associated with the horizontal management of GRDI.

8.2 Documentation / stakeholder views on the level of efficiency of the horizontal governance structure.

8.3 Review of the administrative processes and reporting mechanisms in support of horizontal Initiative.

X X X -
9. To what extent does the current delivery model for GRDI allow for a cost-effective use of federal government resources? Are there alternatives that are more cost-efficient?
  • The current delivery model is deemed by GRDI partners to be appropriate to ensure an efficient use of federal government's resources.

9.1 GRDI performance indicators / targets:

  • Number of formal research collaborations by organization type.
  • Value of non-GRDI funding in support of GRDI projects from A-base and other sources.

9.2 Administrative cost (estimated) incurred by GRDI partners for the delivery of the Initiative.

9.3 Administrative data / documentation / stakeholder views on the level of resources leveraged as part of GRDI.

9.4 Stakeholder views regarding the extent to which the current delivery model for GRDI is appropriate.

  • 3/5-year funding cycles vs. A-base funding;
  • Proportion of mandated vs. shared priority projects; and,
  • Reliance on leveraged funds.

9.5 Stakeholder views on alternative delivery models for GRDI.

X X X -
 

Appendix B: Bibliography

  • AAFC. (2009). AAFC Business Plan for Genomics Research in Support of the Canadian Agriculture and Agri-food Sector and the Bioeconomy.
  • AAFC DPR 2013-14.
  • CFIA. (2009). Canadian Food Inspection Agency Genomics R&D Initiative: Business Case – Draft VII
  • Department of Finance (2006). Advantage Canada: Building a Strong Economy for Canadians.
  • DFO. (2009). Genomics Research and Development Initiative Draft DFO Business Case
  • DFO DPR 2013-14.
  • DFO (2009). Genomics R&D Initiative – Environment Canada Business Cases. Follow-up GRDI MRAP Status Summarized October 2013.
  • Genome Canada. (2015). 2014-2015 Annual Report: Global Challenges – Genomic Solutions.
  • Genome British Columbia. (2015). Genome BC Strategic Plan 2015-2020.
  • Genome Québec. (2015). Annual Report – Special Edition. pg. 17:
  • Genome Canada. (2014). - Genome Canada Five Year Evaluation. pg. 11: (2015). Budget 2015. Retrieved by http://www.budget.gc.ca/2015/docs/plan/budget2015-eng.pdf
  • GoC. (2015). Genomics Research and Development Initiative (GRDI). Innovation management strategy for shared priority projects. About GRDI (2015).
  • GoC. (2014). Genomics R&D Initiative Strategic Planning Meeting: Innovation Management Strategy. Meeting Report.
  • Government of Canada. (2013). Genomics research and development (R&D) investments in departments and agencies participating in the genomics R&D initiative in fiscal year 2011-2012:
  • Government of Canada. (2012). Companion Document to the Performance Measurement Strategy for the Genomics R&D Initiative.
  • Government of Canada. (2012). Genomics R&D Initiative (GRDI) Phase V: Mandated Research Best Practices.
  • Government of Canada. (2011). Announcements - Government of Canada announces Phase V funding for the Genomics R&D Initiative.
  • Government of Canada. (2011). Genomics R&D Initiative (GRDI): Governance Framework - Phase V.
  • GC (2010). GRDI and Genome Canada.
  • GRDI Governance Framework Phase VI April 16, 2015.
  • GRDI MRAP Status Summarized. (2013).
  • GRDI Examples of results to date and economic impacts: 10 examples with economic impacts (2014).
  • Health Canada (2009). A Business Case for Health Canada.
  • Horizontal Performance Measurement Strategy (PMS) for the Genomics R&D Initiative (GRDI) (2011).
  • Collaboration Study of Genomics R&D Initiative (GRDI) Phase V Publications (2015)
  • NRC Knowledge Management, Mandate: Collaboration Study of Genomics R&D Initiative (GRDI) Phase V Publications, October 20th 2015.
  • NRC DPR 2013-14
  • NRC (2016). GRDI Evaluation Collaboration Networks Presentation.
  • NRC. (2009). NRC Business Case: GRDI Policy Framework: Component 2 – Mandated Research,
  • NRCan. (2009). Natural Resources Canada Business Case.
  • Ontario Genomics Institute. (2015). Genomics: Your Solution? Annual report 2014 -2015.
  • PHAC. (2009). PHAC Business Case: GRDI Policy Framework.
  • PHAC DPR 2013-14.
  • Science-Metrix. (2011). Evaluation of the Genomics R&D Initiative (GRDI) - Final Evaluation Report.
  • Story 1: Canadian innovation in food safety poised to hit commercial market:
  • Story 2:GRDI-funded research builds Canada's reputation as food safety leader:
  • Story 3: Chasing E. coli: genomics research tracks dangerous contamination to the source
  • Story 4:Genomics brings new precision to identifying, characterizing, and tracking threats to food and water safety:
  • Treasury Board Secretariat. (2015). Genomics R-D Initiative. Planning Information.
  • TBS. (2013). Horizontal Internal Audit of Compliance with the Policy on Management, Resources and Results Structures.
  • UNESCO. (2015). UNESCO SCIENCE REPORT. Towards 2030. pg. 144:
  • Working Group for the Genomics R&D Initiative. GRDI Annual Report 2013-2014
  • Working Group for the Genomics R&D Initiative. GRDI Annual Report 2012-2013
  • Working Group for the Genomics R&D Initiative. GRDI Annual Report 2011-2012
  • Workshop Report (2016). FWS- GRDI Knowledge Transfer Workshop –Ottawa, February 24-25, 2016

Appendix C: Literature Review Web Search

  1. UNESCO. (2015). UNESCO SCIENCE REPORT. Towards 2030.
    The UNESCO Science Report: towards 2030 provides more country-level information than ever before. The trends and developments in science, technology and innovation policy and governance between 2009 and mid-2015 described here provide essential baseline information on the concerns and priorities of countries that should orient the implementation and drive the assessment of the 2030 Agenda for Sustainable Development in the years to come.  See all of Chapter 4 - Canada (p. 113-122). Genomics: a growing priority for Canada. Canadian public has a positive attitude towards science.  Canada's federal priorities by sector for 2007-2014
  2. Phillips, P. (2010). Genomics and public policy: Wealth for Canadians, The Integrated Assessment Journal, 10 (1), 7-21.
    The Canadian government has issued a number of declarations in recent years that it seeks to make Canada a world leader in life science research, development and commercialization. To that end, the federal government has produced a series of strategy and policy statements in support of science and technology, created a number of new institutions, and refocused a variety of existing science programs to nurture public and private investment in a wide range of projects in the life sciences, including the plant, animal and microbial kingdoms. Ultimately, the goal of this effort and these investments is to create new technologies, new products and new services that will generate economic activity, higher skilled and paid employment in Canada and a higher quality of life for Canadians. Given the intensity of rhetoric and effort, it is worth asking a few basic questions about this strategy. This paper addresses four basic issues. First, is this focus on life sciences, often called the "knowledge" economy, appropriate for Canada? Second, what benefits (and costs) will the knowledge economy, and particularly the life science sector, generate? Third, how confident are we that Canadians will gain from such investments? Fourth, if the focus is appropriate, what is the best way of achieving the goals?
  3. Genome Canada. (2014). - Genome Canada Five Year Evaluation.
    This report presents the findings, conclusions and recommendations resulting from the second five-year evaluation of Genome Canada, which assessed the organization's relevance and retrospective performance in the context of the Canadian research and innovation system from 2009-10 to 2013-14. It also comprises a prospective dimension as it seeks to inform management and other stakeholders on how best to implement the organization's strategic direction (Strategic Plan 2012-2017). See Key Evaluation Findings section (p. 7-33): Relevance, socio-economic benefits, etc. See Figures section: growth in genomics in Canada, research expenditures, projects, etc.
  4. Joly, Yann; Caulfield, Timothy. (2010). The Commercialization of Genomic Research in Canada.
    The commercialization of academic research has been promoted by North American policy makers for over 30 years as a means of increasing university financing and to ensure that promising research would eventually find its way to the marketplace. The following issues paper constitutes a reflection on the impact of the Canadian commercialization framework on academic research in the field of genomics. It was written following two workshops and two independent studies organized by academic groups in Quebec (Centre of Genomics and Policy) and Alberta (Health Law Institute). The full sets of recommendations are available upon request to the authors.
  5. Treasury Board Secretariat (2015). Genomics R-D Initiative. Planning Information.
    Lists federal organizations working with Genome, allocated funds and etc. Lists strategic objectives of using genomics
  6. Genome Canada. (2015). 2014-2015 Annual Report: Global Challenges - Genomic Solutions
    The report is organized by looking at projects that happened throughout the year, bold breakthroughs, reducing genome related risks, funding, and financial highlights and outlooks.  *Reports also available dating 2009-2010
  7. Genome Canada. (2016). GENOME CANADA STRATEGIC PLAN 2012-2017Genomics: DISCOVERY. IMPACT. SUCCESS

Lead Genome Centres reports

  1. Genome Prairie. (2015). Decoding the value of genomics. 2014-15: Annual Report.
    Includes Project Highlights, Impacts and financial statements for the year.
  2. Genome B.C. (2016). British Columbia.2014 - 2015 Annual Report Connectivity It's in our DNA. Financials, statistics, corporate information and success stories.
  3. Genome Québec. (2015). Annual Report - Special Edition.
    Contains activity reports on: scientific affairs, technology centres, public affairs and communications, human resources and legal service and finance. Also contains large scale project outcomes and assessment of completed projects.
  4. Genome British Columbia. (2015). Genome BC Strategic Plan 2015-2020.Highlights achievements to date, Genome BC's Logic Chart, deliverables, etc.
  5. Genome Atlantic. (2015). Genomics at Work. Annual Report 2014-2015.Year in review: agriculture, fisheries and aquaculture, energy, environment, mining, health, forestry.Also lists current projects
  6. Ontario Genomics Institute. (2015). Genomics: Your Solution? Annual report 2014 -2015.

Knowledge transfer

  1. Hall. (2013). Moving Promising Technologies off the Shelf.
    This Policy Brief explores how potential economic and social benefits of applied research can be more fully realized. Structural hurdles that prevent innovations from making the leap 'from lab to living room' are considered. Two projects funded by Genome Canada are used to illustrate the challenges of innovation, emphasizing the need to build economic and social considerations into technical developments. We discuss key technological, commercial, organizational and social hurdles that must be overcome, and how corresponding levers can be exploited for more efficient diffusion. Specific recommendations are offered with implications to both science policy and universities: 1) Improve scientists' awareness of organizational, commercial and social aspects at an early stage in the innovation process; 2) Improve technology transfer offices' abilities to partner with more passive knowledge-seeking industries; 3) Provide longer-term projects to allow the exploration and initial development of the benefits of a science or a technology. We conclude with policy issues for further research.
  2. Knoppers, B., & Leroux, T. (2010). Framing Genomics, Public Health Research and Policy: Points to Consider. KARGER, 224-234.
    Genetic information can be used to target interventions that improve health and prevent disease. Indeed, the results of population genomics research could be useful for public health and national pandemic plans. Yet, firm scientific evidence originating from such research and the indicators of the role of health determinants, gene-gene and gene-environment interaction remain to be assessed and validated before being integrated into pandemic plans or public health programmes. It is not clear what is the role of the State in research on the elucidation of the determinants of gene-gene and gene-environment interactions and how, when, and if such data can be accessed and used for such planning. Over a period of 3 years, we sought to address these questions by gathering data and literature relevant to research in public health genomics, preparing issues papers and, finally, consulting with stakeholders on a provisional 'points to consider' document at various times. Examining in turn the issues of privacy, State powers, stakeholder perceptions, and public participation, we propose in this article, for each of these themes, a series of recommendations aiming to provide guidance on the role of the State in the use of genomic information for public health research, prevention and planning.
  3. Global Alliance for Genomics and Health. (2015). Summary of Third Plenary Meeting.
    This document serves as an interactive record of the third plenary meeting of the Global Alliance for Genomics and Health. An executive summary provides a short, high-level view of the three-day event. More detailed, modular session summaries follow, and include links (where available) to presentation videos. An appendix provides summaries of Working Group and Demonstration Projects. SEE PAGE: 2, 3, 19
  4. Ontario Genomics Institute. (2011). Why Genomics.
    Website highlights importance of genomic studies. Report looks at future solutions, genomics in Ontario, highlights of the year, how to spark solutions, applying these solutions, policy as a type of solution and financials. *Reports are available from 2003-2004

Use of Genomics by Government organizations in support of their mandate

  1. Industry Canada. (2015). 2015-16 Report on Plans and Priorities, 73 p.
    PAGE 16: Strategic Outcome: Advancements in science and technology, knowledge and innovation strengthen the Canadian economy (Genome is mentioned)
    Genome also mentioned on 43rd page

Genome Canada Articles

  1. Genome Canada. (2010). Perspectives on Progress.
    See section The Look Ahead (p.22): Canada's contribution to genomics research
  2. Genome Canada. (2014). Genome Canada Five Year Evaluation Report: Management Response/
    Lists 6 recommendations from previous reports (annual) and actions taken (envisioned or underway) by management.
  3. Genome Canada. (2014). Issue 8. Receptor Capacity for Biotechnology Innovation in Canada.
    Biotechnology innovation and its impact on private sector growth has been an interest of the Canadian federal government for more than 40 years. Despite this perennial interest, the evidence from life science and biotechnology related funding policies and programs, as well as the extent of private sector biotechnology activity, indicates that Canada mostly remains in a 'science and technology push' rather than 'market pull' mode. Public sector investments continue to generate a strong base in the life sciences, stimulating discoveries and generating inventions, training successive generations of students and technical staff, and contributing to respectable achievements in international scientific scholarship. Yet the gap between the research base and expectations of development of commercialisable technologies remains. Many explanations of this gap have been offered, including a culture of risk aversion in Canadian businesses, a lack of direct public investment in early stage innovation, a small domestic venture capital base to bridge the innovation 'valley of death', lack of public procurement strategies for new technology, and weakness in Canadian intellectual property protection. All of these putative causes might contribute to the unchanging nature of the Canadian life science industry in which the positive feedback loops that would build and sustain growth are absent. Among the most important of these is the development of private sector capacity to absorb and exploit the new knowledge arising from inventions and discoveries. This receptor capacity is linked, theoretically and empirically, to a firm's dynamic capabilities to anticipate, monitor, and respond to new knowledge and remain competitive. As described in this policy brief, there are cases in Canadian life science innovation where dynamic capabilities have been cultivated and create desired positive feedback loops for innovation. Policy options are suggested for moving towards regionally focused smart specialisation and greater direct support for early stage innovation. Both options will foster receptor capacity and will improve the dynamic capabilities of the life science sector.
  4. Genome Canada. (2014). - Issue 9. Personalized Medicine and Health Care Policy.
    This Policy Brief explores needed steps in research, development and regulation to facilitate the translation and adoption of high value personalized medicine in the Canadian health care system. It examines the basis of effective development and commercialization including coordination across science, industry and payer communities. All stakeholders will need to fully understand and identify the societal value associated with research and commercialization to make the most of investment in research and to realize the full potential of these interventions for patients, while ensuring they do not drive out existing high value health care activities. As researchers may not have clearly specified value targets from payers or have the capacity to link to those who understand payer need, there may be inherent inefficiencies in approaches to the financing of and conduct of translational research. Our specific recommendations offered with implications to both science policy and universities include 1) insisting that health system payers clearly define what constitutes value; 2) exploring options to more clearly align evidentiary requirements and processes between regulators and health technology assessment bodies that support payers; and 3) increasing strategic focus in applied and technology-oriented basic research that includes emphasizing the need and alignment of experts in HTA, decision-making and economic evaluation in all applied health research activities.

Relevance

  1. Science Metrix Inc. (2011). Evaluation of the Genomics R&D Initiative (GRDI): Final Evaluation Report
    This document presents the key findings of the horizontal evaluation of the Genomics Research and Development Initiative (GRDI). Conclusions and recommendations stem from these key findings, which are based on the integrated analysis of multiple lines of evidence. In 2010, the GRDI Assistant Deputy Minister (ADM) Coordinating Committee mandated the Planning and Performance Management Directorate of the National Research Council Canada (NRC)'s Strategy and Development Branch to lead an evaluation of the GRDI. An Interdepartmental Evaluation Working Group (IEWG) was established, to support the evaluation process. An independent firm, Science-Metrix, was contracted to design and implement this evaluation and a mixed-team approach was used to conduct the fieldwork (evaluators from both Science-Metrix and NRC-SDB-PPM).

Genome in the news

  1. Semeniuk, I. (2015). Genome Canada to create 'innovation network' of 10 research centres. The Globe and Mail.
    "Citing the need to keep up with a rapidly changing field of science, Canada's principal entity for  supporting research in genomics is changing its approach."
  2. Abraham, C. (2014). Unlocking DNA secrets with a Canadian genome search engine. The Globe and Mail.
    "Like a powerful search engine that mines the web for answers, the new computational system combs the human genome to seek and sort meaningful mutations. Google Inc., along with other companies, has already expressed an interest in it - raising questions about what could, or should, happen with publicly funded technology that is likely to be in demand in a growing world of Big Data."
  3. Semeniuk, I. (2014). Canadian companies to look to genomics to drive innovation. The Globe and Mail.
    "For Michel Pouliot, a fine cheese is not just something to be savoured, it's a microcosmic universe to be explored."
  4. Canadian Food Inspection Agency. (2016). Government of Canada Contributes to World Class Research on Species Identification. Yahoo Finance.
    "The Canadian Food Inspection Agency (CFIA) and the University of Guelph's Biodiversity Institute of Ontario (U of G - BIO) will partner to create tools to use genomics and DNA barcoding to improve species identification for early detection of plant pests and mislabelled fish and seafood. Additionally, the funding will support the development of a program to help enhance cooperation between CFIA and (U of G - BIO) scientists, leverage opportunities in genomics to modernize regulatory programs, and prevent entry of invasive plant pests."

Appendix D: Key Informant Interview Guides

Horizontal Evaluation of the Genomics R&D Initiative
Interview Guide – Program Managers and Directors

Context

On behalf of the National Research Council of Canada (NRC), Goss Gilroy is conducting a horizontal evaluation of the Genomics Research & Development Initiative (GRDI). The main goal of the evaluation is to assess the relevance and performance of the Initiative (in accordance with the Treasury Board Policy on Evaluation) with regard to the GRDI's expected outcomes.

The GRDI coordinates federal science departments and agencies in the field of genomics research. Its strategic goal is to contribute solutions to issues that are important to Canadians, focusing on the role of federal government research. Specific applications of GRDI seek to protect and improve human health, develop new treatments for chronic and infectious diseases, protect the environment, manage agricultural and natural resources in a way that is sustainable, and thus support the health and wealth of Canadian communities. Currently eight departments and agencies receive funding under the initiative, these include: The National Research Council of Canada; Agriculture and Agri-Food Canada; Health Canada; the Public Health Agency of Canada; Natural Resources Canada; Environment and Climate Change Canada; Fisheries and Oceans Canada; and the Canadian Food Inspection Agency.

The evaluation will examine the overall initiative with a particular focus on the newly implemented shared priority projects and horizontal governance. In order to fully capture the impacts of the projects funded as part of Phase V of the GRDI (2011-2014), the scope will go beyond 2014. Because of the horizontal nature of the evaluation, the report will present aggregated information on how departments and agencies have contributed to the GRDI expected outcomes as opposed to presenting individual descriptions of the impacts of GRDI on each partner. The expectation of the evaluation is to provide evidence-based information to GRDI senior management and program managers based on the relevance and performance of the initiative.

The evaluation methods include interviews with federal department representatives and interviews of external stakeholders. They also include case study interviews with federal department representatives and external stakeholders that will be complemented by a review of documents to create two extensive case studies for shared priority projects and fourteen case studies for mandated research projects. Your responses will be reported in aggregate only, and you will not be personally identified in any reports.

The interview will take approximately 60 minutes.

Thank you for your collaboration.

Background

  1. Please briefly describe your:
    1. current roles and responsibilities in your department / agency with regards to GRDI?
    2. involvement with the GRDI over time?

Relevance

  1. To what extent the needs of your department / agency in terms of genomic R&D evolved over the last five years? [1.1]
    1. To what extent the GRDI is still addressing the needs?

Probe: If any, what are some of the internal and/or external factors that had influenced your needs (e.g. international or national research environment)

  1. Please describe the mechanisms in place to ensure that the funded projects (GRDI) align with the: [2.1]
    1. priorities of the federal government
    2. mandate of your department / agency [2.2]
  2. To what extent is the federal government involvement in genomics R&D appropriate and consistent with federal roles and responsibilities? [3.1]
  3. In the absence of the GRDI program:
    1. What would happen to genomics research in your department / agency? [5.5]
    2. What would be the effects on the ability of your department / agency to achieve its mandate? [1.3]

Effectiveness

  1. For the projects funded as part of Phase V, would you say that GRDI resulted in:
    1. knowledge and technology transfer to end-users? [4.4]
    2. the use of GRDI's research results by end-users and external stakeholders? [5.1]

Probe: Details on the impact of the specific mandated research projects

Probe: Details on the use of the research results in support of a) decision making and b) the mandate of their department/ agency

  1. Please provide examples / highlights of mechanisms that have been effective in transferring knowledge and technology to targeted end-users. [4.4]
  2. Please provide examples / highlights of benefits and impacts that have been generated from the use of GRDI's research results by: [5.2]
    1. your department / agency?
    2. collaborators and end-users (inside or outside federal government)?

Probe: When applicable, please refer to the following categories of benefits and impacts:

  1. Improved processes;
  2. Improved diagnostics/instrumentation;
  3. Improved products;
  4. Reduced harm (health), improved care, lives saved;
  5. Reduced health care costs (use of health care system);
  6. Improved human health in Canada;
  7. Other cost savings;
  8. Enhanced sustainability and management of Canada's environment, agriculture, forestry and fisheries sectors;
  9. Reduced environmental harm;
  10. Improved food safety and security in Canada;
  11. Economic benefits;
  12. Other benefits.
  1. Can you identify factors that influenced GRDI's:
    1. Level of performance in terms of knowledge / technology transfer to end-users? [4.5]
    2. Ability to generate benefits and impacts on end-users? [5.3]

Shared Priority Projects

  1. Are you aware of the two GRDI Shared Priority Projects?
    1. If yes, please briefly describe your involvement with these two projects?
  2. From your knowledge of the GRDI shared priority projects initiated in Phase V: [1.4]
    1. What were the specific needs addressed by the shared priority projects?
    2. To what extent is there a continued need for interdepartmental research collaboration in genomics?
  3. From your knowledge of the GRDI shared priority projects (i.e., Food and Water Safety and Quarantine Invasive Species) initiated in Phase V, to what extent: [6.1 & 6.2]
    1. Were they appropriately selected and managed?
    2. Did they enhance interdepartmental collaborations as expected?
    3. Did they provide innovative solutions to problems that could not have been achieved by one department /agency?
  4. How important was the level of interdepartmental collaboration in the achievement of shared project results and impacts? [6.2]
    1. What are the key factors that have facilitated or hindered the collaboration between the partners as part of the SPP?
  5. From your perspective, what are the main lessons learned and best practices stemming from the two interdepartmental projects? [7.1]

Governance structure

  1. To what extent the current horizontal governance structure is appropriate to support effective decision making while ensuring accountability (i.e., ADM Coordinating Committee, Interdepartmental Working Group, GRDI Secretariat)? [8.2 & 8.3]
    1. Could you please provide examples of decisions that actually influenced GRDI strategic direction, progress and capacity to deliver on outcomes?
    2. If any, what could be adjusted or changed to improve this aspect of the program.
  2. What are the costs associated with the horizontal management of GRDI that are incurred by your department / agency? [8.1]
    1. Can you please specify these costs? (administrative, human resources/salary, meetings, travel and accommodation, others)
    2. Can you provide an estimate of these costs?
  3. To what extent the costs and level of effort to administer and coordinate the horizontal structure (i.e., the Secretariat) is appropriate? Please explain [8.2]

Probe: Cost and level of effort of the Secretariat support to GRDI departments and agencies and the implementation of the GRDI governance framework, management and operating processes.

  1. Are the current administrative processes and reporting mechanisms in support of this horizontal initiative appropriate? [8.3]

Delivery model

  1. To what extent are the GRDI, Genome Canada and other national granting agencies collaborating to ensure an efficient use of federal government resources dedicated to genomics? [9.4]
    1. Is the level and nature of resources leveraged appropriate?
  2. To what extent is the current delivery model for GRDI appropriate}to ensure an efficient use of federal government's resources? [9.4]
    • 3 to5-year funding cycles vs. A-base funding;
    • Proportion of mandated vs. shared priority projects; and,
    • Reliance on leveraged funds.
  3. Can you think of alternative delivery models for GRDI? Please explain. [9.5]
    1. How the GRDI could be improved to better address the needs of your department / agency?

Additional comments

  1. Finally, do you have any additional comment for this evaluation?
  2. Can you think of any documentation that would be useful for the evaluation of the GRDI?

Thank you.

Horizontal Evaluation of the Genomics R&D Initiative
Interview Guide – Senior Management (VP, ADM, Director General, Executive Director)

Context

On behalf of the National Research Council of Canada (NRC), Goss Gilroy is conducting a horizontal evaluation of the Genomics Research & Development Initiative (GRDI). The main goal of the evaluation is to assess the relevance and performance of the Initiative (in accordance with the Treasury Board Policy on Evaluation) with regard to the GRDI's expected outcomes.

The GRDI coordinates federal science departments and agencies in the field of genomics research. Its strategic goal is to contribute solutions to issues that are important to Canadians, focusing on the role of federal government research. Specific applications of GRDI seek to protect and improve human health, develop new treatments for chronic and infectious diseases, protect the environment, manage agricultural and natural resources in a way that is sustainable, and thus support the health and wealth of Canadian communities. Currently eight departments and agencies receive funding under the initiative, these include: The National Research Council of Canada; Agriculture and Agri-Food Canada; Health Canada; the Public Health Agency of Canada; Natural Resources Canada; Environment Canada; Fisheries and Oceans Canada; and the Canadian Food Inspection Agency.

The evaluation will examine the overall initiative with a particular focus on the newly implemented shared priority projects and horizontal governance. In order to fully capture the impacts of the projects funded as part of Phase V of the GRDI (2011-2014), the scope will go beyond 2014. Because of the horizontal nature of the evaluation, the report will present aggregated information on how departments and agencies have contributed to the GRDI expected outcomes as opposed to presenting individual descriptions of the impacts of GRDI on each partner. The expectation of the evaluation is to provide evidence-based information to GRDI senior management and program managers based on the relevance and performance of the initiative.

The evaluation methods include interviews with federal department representatives and interviews of external stakeholders. They also include case study interviews with federal department representatives and external stakeholders that will be complemented by a review of documents to create two extensive case studies for shared priority projects and fourteen case studies for mandated research projects. Your responses will be reported in aggregate only, and you will not be personally identified in any reports.

The interview will take approximately 60 minutes.

Thank you for your collaboration.

Background

  1. Please briefly describe your:
    1. Current roles and responsibilities in your department / agency?
    2. Involvement with the GRDI over time?

Relevance

  1. To what extent the needs of your department / agency in terms of genomic R&D evolved over the last five years? [1.1]
    1. To what extent the GRDI is still addressing the needs?

Probe: If any, what are some of the internal and/or external factors that had influenced your needs (e.g. international or national research environment)

  1. To what extent is the federal government involvement in genomics R&D appropriate and consistent with federal roles and responsibilities? [3.1]
  2. In the absence of the GRDI program:
    1. What would happen to genomics research in your department / agency? [5.5]
    2. What would be the effects on the ability of your department / agency to achieve its mandate? [1.3]

Effectiveness

  1. In your views, to what extent are the results of GRDI projects being used:
    1. within your department / agency? [4.4]
    2. by other users? [5.1]

Probe: Could you please provide more details on the use of the research results in support of decision making?

  1. What are some of the factors that positively or negatively influence knowledge / technology transfer to end-users? [4.5]
  2. Please provide examples / highlights of benefits and impacts that have been generated from the use of GRDI's research results by: [5.2]
    1. Your department / agency?
    2. Collaborators and end-users (inside or outside federal government)?

Shared Priority Projects

  1. Are you aware of the two GRDI Shared Priority Projects?
    1. If yes, please briefly describe your involvement with these two projects?
  2. From your knowledge of the GRDI shared priority projects (i.e., Food and Water Safety and Quarantine Invasive Species) initiated in Phase V, to what extent: [6.1 & 6.2]
    1. Were they appropriately selected and managed?
    2. Did they enhance interdepartmental collaborations as expected? Please explain.
    3. Did they provide innovative solutions to problems that could not have been achieved by one department / agency?
  3. To the best of your knowledge, what are the key factors that have facilitated or hindered the partner's ability to collaborate as part of an interdepartmental project?
  4. From your perspective, what are the main lessons learned and best practices stemming from the two interdepartmental projects? [7.1]
  5. To what extent is there a continued need for interdepartmental research collaboration in genomics? [1.4]

Governance structure

  1. To what extent the current horizontal governance structure is appropriate to support effective decision making while ensuring accountability (i.e., ADM Coordinating Committee, Interdepartmental Working Group, GRDI Secretariat)? [8.2 & 8.3]
    1. Could you please provide examples of decisions that actually influenced GRDI strategic direction, progress and capacity to deliver on outcomes?
    2. If any, what could be adjusted or changed to improve this aspect of the program.
  2. What are the costs associated with the horizontal management of GRDI that are incurred by your department / agency? [8.1]
    1. Can you please specify these costs? (administrative, human resources/salary, meetings, travel and accommodation, others)
    2. Can you provide an estimate of these costs?
  3. To what extent the costs and level of effort to administer and coordinate the horizontal structure (i.e., the Secretariat) is appropriate? Please explain [8.2]

Probe: Cost and level of effort of the Secretariat support to GRDI departments and agencies and the implementation of the GRDI governance framework, management and operating processes.

Delivery model

  1. To what extent is the current delivery model for GRDI appropriate to ensure an efficient use of federal government's resources? [9.4]
    1. 3 to 5-year funding cycles vs. A-base funding;
    2. Proportion of mandated vs. shared priority projects; and,
    3. Reliance on leveraged funds.
  2. Can you think of alternative delivery models for GRDI? Please explain. [9.5]

Additional comments

  1. Do you have any additional comment for this evaluation?

Thank you.

Horizontal Evaluation of the Genomics R&D Initiative
Interview Guide – External respondents (Genome Canada)

Context

On behalf of the National Research Council of Canada (NRC), Goss Gilroy is conducting a horizontal evaluation of the Genomics Research & Development Initiative (GRDI). The main goal of the evaluation is to assess the relevance and performance of the Initiative (in accordance with the Treasury Board Policy on Evaluation) with regard to the GRDI's expected outcomes.

The GRDI coordinates federal science departments and agencies in the field of genomics research. Its strategic goal is to contribute solutions to issues that are important to Canadians, focusing on the role of federal government research. Specific applications of GRDI seek to protect and improve human health, develop new treatments for chronic and infectious diseases, protect the environment, manage agricultural and natural resources in a way that is sustainable, and thus support the health and wealth of Canadian communities. Currently eight departments and agencies receive funding under the initiative, these include: The National Research Council of Canada; Agriculture and Agri-Food Canada; Health Canada; the Public Health Agency of Canada; Natural Resources Canada; Environment Canada; Fisheries and Oceans Canada; and the Canadian Food Inspection Agency.

The evaluation will examine the overall initiative with a particular focus on the newly implemented shared priority projects and horizontal governance. In order to fully capture the impacts of the projects funded as part of Phase V of the GRDI (2011-2014), the scope will go beyond 2014. Because of the horizontal nature of the evaluation, the report will present aggregated information on how departments and agencies have contributed to the GRDI expected outcomes as opposed to presenting individual descriptions of the impacts of GRDI on each partner. The expectation of the evaluation is to provide evidence-based information to GRDI senior management and program managers based on the relevance and performance of the initiative.

The evaluation methods include interviews with federal department representatives and interviews of external stakeholders. They also include case study interviews with federal department representatives and external stakeholders that will be complemented by a review of documents to create two extensive case studies for shared priority projects and fourteen case studies for mandated research projects. Your responses will be reported in aggregate only, and you will not be personally identified in any reports.

The interview will take approximately 60 minutes.

Thank you for your collaboration.

Background

  1. Please briefly describe your:
    1. Current roles and responsibilities in your organization?
  2. For context, how would you describe the evolution of the: [1.1]
    1. Genomics research environment over the last five years?
    2. Genome Canada's programs over the last five years?
  3. From your perspective, to what extent is the federal government involvement in genomics R&D appropriate and consistent with federal roles and responsibilities? [3.1]
  4. To what extent are the large-scale programs financed by Genome Canada and the work conducted by federal scientists as part of GRDI complementary? [3.2]
  5. If any, are you aware of other organizations or programs that: [3.2]
    1. Complement the work conducted as part of GRDI?
    2. Duplicate or that could perform the work conducted as part of GRDI?
  6. To what extent were synergies developed over time between the GRDI program and programs financed by Genome Canada? [3.2]
    1. Any change over the last five years?
    2. Did you experienced specific challenges in this area?
    3. To what extent are federal scientists currently working with academic scientists active under large-scale programs financed by Genome Canada?
    4. Can you provide an example of synergies and of leveraged expertise / resources?
  7. Please provide examples / highlights of benefits and impacts that have been generated from the use of GRDI's research results by: [5.2]
    1. Your organization?
    2. Collaborators and end-users (inside or outside federal government)?
  8. To what extent are Genome Canada, the GRDI and other national granting agencies collaborating to ensure an efficient use of federal government's resources dedicated to genomics? [9.4]

Additional comments

  1. Can you identify any documentation that would be useful for the evaluation of the GRDI?
  2. Finally, do you have any additional comment for this evaluation of the performance and relevance of the GRDI?

Thank you.

Horizontal Evaluation of the Genomics R&D Initiative
Interview Guide – Scientists

Context

On behalf of the National Research Council of Canada (NRC), Goss Gilroy is conducting a horizontal evaluation of the Genomics Research & Development Initiative (GRDI). The main goal of the evaluation is to assess the relevance and performance of the Initiative (in accordance with the Treasury Board Policy on Evaluation) with regard to the GRDI's expected outcomes.

The GRDI coordinates federal science departments and agencies in the field of genomics research. Its strategic goal is to contribute solutions to issues that are important to Canadians, focusing on the role of federal government research. Specific applications of GRDI seek to protect and improve human health, develop new treatments for chronic and infectious diseases, protect the environment, manage agricultural and natural resources in a way that is sustainable, and thus support the health and wealth of Canadian communities. Currently eight departments and agencies receive funding under the initiative, these include: The National Research Council of Canada; Agriculture and Agri-Food Canada; Health Canada; the Public Health Agency of Canada; Natural Resources Canada; Environment Canada; Fisheries and Oceans Canada; and the Canadian Food Inspection Agency.

The evaluation will examine the overall initiative with a particular focus on the newly implemented shared priority projects and horizontal governance. In order to fully capture the impacts of the projects funded as part of Phase V of the GRDI (2011-2014), the scope will go beyond 2014. Because of the horizontal nature of the evaluation, the report will present aggregated information on how departments and agencies have contributed to the GRDI expected outcomes as opposed to presenting individual descriptions of the impacts of GRDI on each partner. The expectation of the evaluation is to provide evidence-based information to GRDI senior management and program managers based on the relevance and performance of the initiative.

The evaluation methods include interviews with federal department representatives and interviews of external stakeholders. They also include case study interviews with federal department representatives and external stakeholders that will be complemented by a review of documents to create two extensive case studies for shared priority projects and fourteen case studies for mandated research projects. Your responses will be reported in aggregate only, and you will not be personally identified in any reports.

The interview will take approximately 60 minutes.

Thank you for your collaboration.

Background

  1. Please briefly describe your:
    1. Current roles and responsibilities in your department / agency?
    2. Involvement with GRDI projects over time?

Relevance

  1. To what extent the needs of your department / agency in terms of genomic R&D evolved over the last five years? [1.1]
    1. To what extent the GRDI is still addressing the needs?

Probe: If any, what are some of the internal and/or external factors that had influenced your needs (e.g. international or national research environment)

  1. To what extent is there alignment between your GRDI projects and the mandate of your department / agency? [2.2]
  2. To what extent is the federal government involvement in genomics R&D appropriate and consistent with:
    1. mandate of your department / agency[2.2]
    2. federal roles and responsibilities? [3.1]
  3. In the absence of the GRDI program:
    1. What would happen to genomics research in your department / agency? [5.5]
    2. What would be the effects on the ability of your department / agency to achieve its mandate? [1.3]
    3. What would not have gone ahead?

Research Results to date

  1. For the projects funded as part of Phase V, would you say that GRDI resulted in:
    1. knowledge and technology transfer to end-users? [4.4]
    2. the use of GRDI's research results by end-users and external stakeholders? [5.1]

Probe: Details on the impact of the specific mandated research projects

Probe: Details on the use of the research results in support of a) decision making and b) the mandate of their department/ agency

  1. Please provide examples / highlights of mechanisms that have been effective in transferring knowledge and technology to targeted end-users. [4.4]
  2. Please explain to what extent the project contributed to any of the following potential outcomes. Please quantify estimates to the extent possible. [5.2]
    • Improved processes, reduced production costs
    • Improved diagnostics/instrumentation
    • Improved products
    • Reduced harm (health), improved care/health, lives saved
    • Reduced health care costs;
    • Other cost savings (please provide range of estimates);
    • Enhanced sustainability and management of Canada's agriculture, forestry and fisheries sectors;
    • Reduced environmental harm (e.g., reduced use of chemicals, reduced pollution, GHG emissions, etc.);
    • Improved food safety and security in Canada;
    • Improved/informed policies, regulations, legislation, programs;
    • Other

Please provide any documents or data that you would be willing to share with us about the potential outcomes.

  1. Please provide examples / highlights of benefits and impacts that have been generated from the use of GRDI's research results by: [5.2]
    1. your department / agency?
    2. collaborators and end-users (inside or outside federal government)?
  2. Have there been major facilitators (positive factors) or challenges (negative factors) to your GRDI project? Please explain.
  3. In particular, can you identify factors that influenced GRDI's:
    1. Level of performance in terms of knowledge / technology transfer to end-users? [4.5]
    2. Ability to generate benefits and impacts on end-users? [5.3]

Shared Priority Projects

  1. Are you aware of the two GRDI Shared Priority Projects?
    1. If yes, please briefly describe your involvement with these two projects?
  2. From your knowledge of the GRDI shared priority projects (i.e., Food and Water Safety and Quarantine Invasive Species) initiated in Phase V, to what extent: [6.1 & 6.2]
    1. Were they appropriately selected and managed?
    2. Did they enhance interdepartmental collaborations as expected?
    3. Did they provide innovative solutions to problems that could not have been achieved by one department /agency?
  3. How important was the level of interdepartmental collaboration in the achievement of shared project results and impacts? [6.2] What factors influenced your ability to work collaboratively in these research projects?
  4. From your perspective, what are the main lessons learned and best practices stemming from the two interdepartmental projects? [7.1]
  5. To what extent is there a continued need for interdepartmental research collaboration in genomics?[1.4]

Delivery model

  1. To what extent is the current delivery model for GRDI appropriate to ensure an efficient use of federal government's resources? [9.4]
    1. 3 to 5-year funding cycles vs. A-base funding; and
    2. Reliance on leveraged funds.

Additional comments

  1. Finally, do you have any additional comment for this evaluation?
  2. Can you think of any documentation that would be useful for the evaluation of the GRDI?

Thank you.

Appendix E: Case Study Interview Guides

Horizontal Evaluation of the Genomics R&D Initiative
Case Study Interview Guide: Research Project Leaders

On behalf of the National Research Council of Canada (NRC), Goss Gilroy is conducting a horizontal evaluation of the Genomics Research & Development Initiative (GRDI). The main goal of the evaluation is to assess the relevance and performance of the Initiative (in accordance with the Treasury Board Policy on Evaluation) with regard to the GRDI's expected outcomes. The evaluation will examine the overall initiative with a particular focus on the newly implemented shared priority projects and horizontal governance. In order to fully capture the impacts of the projects funded as part of Phase V of the GRDI (2011-2014), the scope will go beyond 2014. Because of the horizontal nature of the evaluation, the report will present aggregated information on how departments and agencies have contributed to the GRDI expected outcomes as opposed to presenting individual descriptions of the impacts of GRDI on each partner. The expectation of the evaluation is to provide evidence-based information to GRDI senior management and program managers based on the relevance and performance of the initiative.

The evaluation methods include interviews with federal department representatives and interviews of external stakeholders. They also include case study interviews with federal department representatives and external stakeholders that will be complemented by a review of documents to create two extensive case studies for shared priority projects and fourteen case studies for mandated research projects. Your responses will be reported in aggregate only, and you will not be personally identified in any reports.

The interview will take approximately 60 minutes.

Thank you for your collaboration

For this interview, we would like to focus on ….(identify project).

Background

  1. For context, could you please describe your involvement in this project?
  2. Could you please explain what need is driving this research? In other words, why is this research being conducted? [1.1]
  3. Could you please describe the activities and partners involved in this project, including:
    1. Research activities
    2. Partners and collaborators (internal and external)
    3. Funding partners
  4. To date, is the research project delivered on time and on budget? [9.4]

Research results to date

  1. Based on the following generic R&D phases, where would you situate the state of progress of your project at this point?
    1. Basic/fundamental research, pre-proof of concept
    2. Proof of concept achieved, prototype development, lab scale production
    3. Pre-clinical trials, clinical trials, field trials, scale-up production development
    4. Application/commercialization
      1. Given the length of time and nature of the research, are you satisfied with the progress of the project? Why/why not? How could progress be improved? [4.4]
  2. Please describe how the research results have been disseminated, transferred to end-users inside and out of the federal government, and protected (i.e., IP) to date. [4.4]
  3. What have been the key factors that have facilitated or hindered the use of the knowledge and technologies by the end-users? [4.5, 5.3]
  4. How likely will this project lead to applications/commercialization? What needs to be done to achieve this? [5.2]
  5. Please explain to what extent the project contributed to any of the following potential outcomes. Please quantify estimates to the extent possible. [4.2, 5.2] Please provide examples of results to end-users achieved through these interdepartmental collaborations?
    • Improved processes, reduced production costs
    • Improved diagnostics/instrumentation
    • Improved products
    • Reduced harm (health), improved care/health, lives saved
    • Reduced health care costs;
    • Other cost savings (please provide range of estimates);
    • Enhanced sustainability and management of Canada's agriculture, forestry and fisheries sectors;
    • Reduced environmental harm (e.g., reduced use of chemicals, reduced pollution, GHG emissions, etc.) ;
    • Improved food safety and security in Canada;
    • Improved/informed policies, regulations, legislation, programs;
    • Other

Please provide any documents that you would be willing to share with us about the potential outcomes.

  1. Has this project led to business creation, sales and/or jobs created? Please provide range of estimates. [5.2]
  2. Did this project involve the development/training of highly qualified personnel (HQP)? Please explain. [4.3]
  3. What would have happened in the absence of GRDI funding? [1.1, 5.4, 5.5]
    1. What would not have gone ahead?
    2. What would happen to genomics research in your unit in the absence of GRDI?
  4. Did the project reports provided to GRDI reflect achievements to date? [8.3]
  5. To date, what lessons have been learned? [7.1]
    1. What could have been done differently?
    2. What worked particularly well?

Thank you.

Horizontal Evaluation of the Genomics R&D Initiative
Case Study Interview Guide: Research Partners and End-Users

On behalf of the National Research Council of Canada (NRC), Goss Gilroy is conducting a horizontal evaluation of the Genomics Research & Development Initiative (GRDI). The main goal of the evaluation is to assess the relevance and performance of the Initiative (in accordance with the Treasury Board Policy on Evaluation) with regard to the GRDI's expected outcomes. The evaluation will examine the overall initiative with a particular focus on the newly implemented shared priority projects and horizontal governance. In order to fully capture the impacts of the projects funded as part of Phase V of the GRDI (2011-2014), the scope will go beyond 2014. Because of the horizontal nature of the evaluation, the report will present aggregated information on how departments and agencies have contributed to the GRDI expected outcomes as opposed to presenting individual descriptions of the impacts of GRDI on each partner. The expectation of the evaluation is to provide evidence-based information to GRDI senior management and program managers based on the relevance and performance of the initiative.

The evaluation methods include interviews with federal department representatives and interviews of external stakeholders. They also include case study interviews with federal department representatives and external stakeholders that will be complemented by a review of documents to create two extensive case studies for shared priority projects and fourteen case studies for mandated research projects. Your responses will be reported in aggregate only, and you will not be personally identified in any reports.

The interview will take 60 minutes at most.

Thank you for your collaboration

For this interview, we would like to focus on …. (identify project).

Background

  1. For context, could you please describe your involvement in this project?
  2. Could you please explain what need is driving this research? In other words, why is this research being conducted? [1.1]

Research results to date

  1. Based on the following generic R&D phases, where would you situate the state of progress of the project at this point? [4.4]
    • Basic/fundamental research, pre-proof of concept
    • Proof of concept achieved, prototype development, lab scale production
    • Pre-clinical trials, clinical trials, field trials, scale-up production development
    • Application/commercialization
      1. Given the length of time and nature of the research, are you satisfied with the progress of the project? Why/why not? How could progress be improved? [4.4]
  2. Please describe how the research results have been disseminated, transferred to end-users inside and out of the federal government, and protected (i.e., IP) to date. [4.4]
  3. What have been the key factors that have facilitated or hindered the use of the knowledge and technologies by the end-users? [4.5, 5.3]
  4. How likely will this project lead to applications/commercialization? What needs to be done to achieve this? [5.2]
  5. Please explain to what extent the project contributed to any of the following potential outcomes. Please quantify estimates to the extent possible. [4.2, 5.2] Please provide examples of results to end-users achieved through these interdepartmental collaborations?
    • Improved processes, reduced production costs
    • Improved diagnostics/instrumentation
    • Improved products
    • Reduced harm (health), improved care/health, lives saved
    • Reduced health care costs;
    • Other cost savings (please provide range of estimates);
    • Enhanced sustainability and management of Canada's agriculture, forestry and fisheries sectors;
    • Reduced environmental harm (e.g., reduced use of chemicals, reduced pollution, GHG emissions, etc.) ;
    • Improved food safety and security in Canada;
    • Improved/informed policies, regulations, legislation, programs;
    • Other

Please provide any documents that you would be willing to share with us about the potential outcomes.

  1. Has this project led to business creation, sales and/or jobs created? Please provide range of estimates. [5.2]
  2. Did this project involve the development/training of highly qualified personnel (HQP)? Please explain. [4.3]
  3. What would have happened in the absence of GRDI funding? [1.1, 5.4, 5.5]
    1. What would not have gone ahead?
  4. To date, what lessons have been learned? [7.1]
    1. What could have been done differently?
    2. What worked particularly well?

Thank you.

Horizontal Evaluation of the Genomics R&D Initiative
Shared Priority Project Case Study Interview Guide: Research Collaborators and End-Users

On the behalf of the National Research Council of Canada (NRC), Goss Gilroy is conducting a horizontal evaluation of the Genomics Research & Development Initiative (GRDI). The main goal of the evaluation is to assess the relevance and performance of the Initiative (in accordance with the Treasury Board Policy on Evaluation) with regard to the GRDI's expected outcomes. The evaluation will examine the overall initiative with a particular focus on the newly implemented shared priority projects and horizontal governance. In order to fully capture the impacts of the projects funded as part of Phase V of the GRDI (2011-2014), the scope will go beyond 2014. Because of the horizontal nature of the evaluation, the report will present aggregated information on how departments and agencies have contributed to the GRDI expected outcomes as opposed to presenting individual descriptions of the impacts of GRDI on each partner. The expectation of the evaluation is to provide evidence-based information to GRDI senior management and program managers based on the relevance and performance of the initiative.

The evaluation methods include interviews with federal department representatives and interviews of external stakeholders. They also include case study interviews with federal department representatives and external stakeholders that will be complemented by a review of documents to create two extensive case studies for shared priority projects and fourteen case studies for mandated research projects. Your responses will be reported in aggregate only, and you will not be personally identified in any reports.

The interview will take approximately 60 minutes.

Thank you for your collaboration.

Background

  1. For context, could you please describe your involvement in this project?
  2. Could you please explain what need is driving this research? In other words, why is this research being conducted? [1.1]

Shared Priority Projects – Interdepartmental Collaboration

  1. From your knowledge of the GRDI shared priority projects initiated in Phase IV: [1.4]
    1. What were the specific needs to be addressed by the shared priority projects?
    2. To what extent is there a continued need for interdepartmental research collaboration in genomics?
  2. To what extent did the project enhance interdepartmental collaborations (for example, joint research, joint publications, joint conference presentations)? [6.1 & 6.2]
    Note to interviewer: Review the collaboration study beforehand and be prepared to probe regarding co-authored publications, number of collaborations, type of collaboration (i.e., within or outside of government). Try to get their opinion on whether they feel the degree of collaboration is appropriate and/or could be improved and, if so, how.
    1. In your view, could interdepartmental collaborations be improved? How? [6.1]
    2. What are the key factors that have facilitated or hindered the collaboration between departments as part of the SPP?
  3. How important was the level of interdepartmental collaboration in the achievement of shared project results and impacts? [6.2]
  4. From your perspective, what are the main lessons learned and best practices stemming from your participation in the interdepartmental project? [7.1]

Research results to date

  1. Based on the following generic R&D phases, where would you situate the state of progress of the project at this point? [4.4]
    • Basic/fundamental research, pre-proof of concept
    • Proof of concept achieved, prototype development, lab scale production
    • Pre-clinical trials, clinical trials, field trials, scale-up production development
    • Application/commercialization
      1. Given the length of time and nature of the research, are you satisfied with the progress of the project? Why/why not? How could progress be improved? [4.4]
  2. Please describe whether/how the research results have been disseminated, transferred to end-users inside and out of the federal government, and protected (i.e., IP) to date. [4.4]
  3. What have been the key factors that have facilitated or hindered the use of the knowledge and technologies by the end-users? [4.5, 5.3]
  4. How likely will this project lead to applications? What needs to be done to achieve this? [5.2]
  5. Please explain to what extent the project contributed to any of the following potential outcomes. Please quantify estimates to the extent possible. [4.2, 5.2] Please provide examples of results to end-users achieved through these interdepartmental collaborations?
    • Improved processes, reduced production costs
    • Improved diagnostics/instrumentation
    • Improved products
    • Reduced harm (health), improved care/health, lives saved
    • Reduced health care costs;
    • Other cost savings (please provide range of estimates);
    • Enhanced sustainability and management of Canada's agriculture, forestry and fisheries sectors;
    • Reduced environmental harm (e.g., reduced use of chemicals, reduced pollution, GHG emissions, etc.) ;
    • Improved food safety and security in Canada;
    • Improved/informed policies, regulations, legislation, programs;
    • Other

Please provide any documents that you would be willing to share with us about the potential outcomes.

  1. What would have happened in the absence of GRDI funding? [1.1, 5.4, 5.5]
    1. What would not have gone ahead?
  2. To date, what lessons have been learned in terms of interdepartmental collaboration? [7.1]
    1. What could have been done differently?
    2. What worked particularly well?

Thank you.

Horizontal Evaluation of the Genomics R&D Initiative
Shared Priority Project Case Study Interview Guide: Federal Government representatives (senior management, program managers/directors, or researchers)

On the behalf of the National Research Council of Canada (NRC), Goss Gilroy is conducting a horizontal evaluation of the Genomics Research & Development Initiative (GRDI). The main goal of the evaluation is to assess the relevance and performance of the Initiative (in accordance with the Treasury Board Policy on Evaluation) with regard to the GRDI's expected outcomes. The evaluation will examine the overall initiative with a particular focus on the newly implemented shared priority projects and horizontal governance. In order to fully capture the impacts of the projects funded as part of Phase V of the GRDI (2011-2014), the scope will go beyond 2014. Because of the horizontal nature of the evaluation, the report will present aggregated information on how departments and agencies have contributed to the GRDI expected outcomes as opposed to presenting individual descriptions of the impacts of GRDI on each partner. The expectation of the evaluation is to provide evidence-based information to GRDI senior management and program managers based on the relevance and performance of the initiative.

The evaluation methods include interviews with federal department representatives and interviews of external stakeholders. They also include case study interviews with federal department representatives and external stakeholders that will be complemented by a review of documents to create two extensive case studies for shared priority projects and fourteen case studies for mandated research projects. Your responses will be reported in aggregate only, and you will not be personally identified in any reports.

The interview will take approximately 60 minutes.

Thank you for your collaboration.

Background

  1. For context, could you please describe your involvement in this project?
  2. Could you please explain what need is driving this research? In other words, why is this research being conducted? [1.1]
  3. Could you please describe the activities and partners involved in this project, including:
    1. Research activities
    2. Partners and collaborators (internal and external)
    3. Funding partners
  4. To date, is the research project delivered on time and on budget? [9.4]

Shared Priority Projects – Interdepartmental Collaboration

  1. From your knowledge of the GRDI shared priority projects initiated in Phase IV: [1.4]
    1. To what extent is there a continued need for interdepartmental research collaboration in genomics?
  2. To what extent did the project enhance interdepartmental collaborations (for example, joint research, joint publications, joint conference presentations)? [6.1 & 6.2]
    Note to interviewer: Review the collaboration study beforehand and be prepared to probe regarding co-authored publications, number of collaborations, type of collaboration (i.e., within or outside of government). Try to get their opinion on whether they feel the degree of collaboration is appropriate and/or could be improved and, if so, how.
    1. In your view, could interdepartmental collaborations be improved? How? [6.1]
    2. What are the key factors that have facilitated or hindered the collaboration between departments as part of the SPP?
  3. How important was the level of interdepartmental collaboration in the achievement of shared project results and impacts? [6.2]
    1. To what extent could these results/impacts been achieved by one department or agency alone?
  4. From your perspective, what are the main lessons learned and best practices stemming from the interdepartmental project? [7.1]

Research results to date

  1. Based on the following generic R&D phases, where would you situate the state of progress of your project at this point? [4.4]
    • Basic/fundamental research, pre-proof of concept
    • Proof of concept achieved, prototype development, lab scale production
    • Pre-clinical trials, clinical trials, field trials, scale-up production development
    • Application/commercialization
      1. Given the length of time and nature of the research, are you satisfied with the progress of the project? Why/why not? How could progress be improved? [4.4]
  2. Please describe whether/how the research results have been disseminated, transferred to end-users inside and out of the federal government, and protected (i.e., IP) to date. [4.4]
  3. What have been the key factors that have facilitated or hindered the use of the knowledge and technologies by the end-users? [4.5, 5.3]
  4. How likely will this project lead to applications? What needs to be done to achieve this? [5.2]
  5. Please explain to what extent the project contributed to any of the following potential outcomes. Please quantify estimates to the extent possible. [4.2, 5.2] Please provide examples of results to end-users achieved through these interdepartmental collaborations?
    • Improved processes, reduced production costs
    • Improved diagnostics/instrumentation
    • Improved products
    • Reduced harm (health), improved care/health, lives saved
    • Reduced health care costs;
    • Other cost savings (please provide range of estimates);
    • Enhanced sustainability and management of Canada's agriculture, forestry and fisheries sectors;
    • Reduced environmental harm (e.g., reduced use of chemicals, reduced pollution, GHG emissions, etc.) ;
    • Improved food safety and security in Canada;
    • Improved/informed policies, regulations, legislation, programs;
    • Other

Please provide any documents that you would be willing to share with us about the potential outcomes.

  1. What would have happened in the absence of GRDI funding? [1.1, 5.4, 5.5]
    1. What would not have gone ahead?
  2. Do the project reports provided to GRDI reflect the project's achievements to date? How could reporting be improved? [8.3]
  3. To date, what lessons have been learned in terms of interdepartmental collaboration? [7.1]
    1. What could have been done differently?
    2. What worked particularly well?

Thank you.

Appendix F: Performance Indicators

Scientific Contributions

Scientific contributions: The last year of Phase V was particularly prolific with 1,506 scientific contributions, almost twice as many as the preceding years (741 in 2011-12, and 789 in 2012-13). The total figure for scientific contributions is well above the range recorded for Phase IV of the initiative (1,871).

  • Publications in refereed journals accounted for 584 of the total scientific contributions recorded between 2011-12 and 2013-14 (Table 2).
  • Publications in refereed conference proceedings accounted for 111 of the total scientific contributions between 2011-12 and 2013-14 (Table 3).
  • In total, 47 technical reports were produced between 2011-12 and 2013-14 (Table 4). The first year of Phase V of the GRDI corresponded with the highest number of technical reports produced (22).
  • Seven books in total were written or edited as a result of GRDI Phase V projects (Table 5).
  • Other publications (book chapters, monographs, abstracts etc.) however accounted for 137 of total scientific contributions under Phase V of the initiative (Table 6). The number of other publications remained stable over the 3-year period, ranging from 43 to 49.
  • A total of 309 instances of poster presentations at conferences were recorded by the program (Table 7). The number of poster presentations at conferences ranged from 93 to 121 over the 3-year period, peaking in 2012-2013.
  • Invited presentations accounted for 371 of the total scientific contributions recorded between 2011-12 and 2013-14 (Table 8). The number of invited presentations ranged from 102 to 156 over the 3-year period, peaking in 2013-14.
  • A total of 140 national conference presentations were recorded between 2011-12 and 2013-14 (Table 9). The number of national conference presentations ranged from 33 to 49 per year over the 3-year period.
  • International conference presentations accounted for 235 of the total scientific contributions recorded under Phase V of the initiative (Table 10).
  • A total of 57 instances of active participation (organizer, chair, panel etc.) in national conferences were recorded between 2011-12 and 2013-14 (Table 11).
  • In total, 62 instances of active participation (organizer, chair, panel etc.) in international conferences were recorded between 2011-12 and 2013-14 (Table 12).
  • Editorial posts for national and international journals accounted for 86 instances of scientific contributions between 2011-12 and 2013-14 (Table 13).
  • A total of 812 deposits in genomics related database or libraries were recorded between 2011-12 and 2013-14 (Table 14). As can be seen in
  • Table 14, 700 instances of deposits in genomics-related databases or libraries were recorded in 2013-2014, a significant increase compared to the preceding years.
  • New genomics related databases or libraries accounted for 34 instances of scientific contribution between 2011-12 and 2013-14 (Table 15).
  • A total of 44 awards and prizes were recorded between 2011-12 and 2013-14 (Table 16).
Table 1: Total Scientific Contributions
  2011-12 2012-13 2013-14 Total
AAFC 213 224 138 575
DFO 49 37 55 141
ECCC 146 84 161 391
HC 43 85 70 198
NRC 92 17 61 170
NRCan 140 179 174 493
PHAC 58 76 727 861
QIS N/A 43 95 138
FWS N/A 44 25 69
Total 741 789 1,506 3036

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 2: Publications in refereed journals
  2011-12 2012-13 2013-14 Total
AAFC 61 61 28 150
DFO 8 13 7 28
ECCC 45 20 53 118
HC 6 17 20 43
NRC 33 9 17 59
NRCan 28 42 30 100
PHAC 9 12 22 43
QIS N/A 16 17 33
FWS N/A 4 6 10
Total 190 194 200 584

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 3: Publications in refereed conference proceedings
  2011-12 2012-13 2013-14 Total
AAFC 4 7 14 25
DFO 0 0 1 1
ECCC 1 1 9 11
HC 3 6 0 9
NRC 17 0 3 20
NRCan 2 5 12 19
PHAC 2 8 11 21
QIS N/A 0 1 1
FWS N/A 2 2 4
Total 29 29 53 111

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 4: Technical Reports
  2011-12 2012-13 2013-14 Total
AAFC 7 0 0 7
DFO 9 1 0 10
ECCC 2 5 9 16
HC 0 0 0 0
NRC 2 0 1 3
NRCan 0 0 3 3
PHAC 2 2 1 5
QIS N/A 0 2 2
FWS N/A 1 0 1
Total 22 9 16 47

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 5: Books (edited, written)
  2011-12 2012-13 2013-14 Total
AAFC 1 1 0 2
DFO 0 0 0 0
ECCC 1 0 0 1
HC 0 0 0 0
NRC 2 0 0 2
NRCan 0 0 0 0
PHAC 0 0 0 0
QIS N/A 0 0 0
FWS N/A 0 2 2
Total 4 1 2 7

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 6: Other publications (ex. book chapters, monographs, abstracts, notes, etc. industry magazines)
  2011-12 2012-13 2013-14 Total
AAFC 26 12 6 44
DFO 1 1 4 6
ECCC 5 4 7 16
HC 0 0 1 1
NRC 1 0 1 2
NRCan 15 21 8 44
PHAC 1 3 7 11
QIS N/A 1 9 10
FWS N/A 1 2 3
Total 49 43 45 137
Table 7: Poster presentations at conferences
  2011-12 2012-13 2013-14 Total
AAFC 29 32 12 73
DFO 5   4 9
ECCC 15 6 15 36
HC 15 29 15 59
NRC 0 2 8 10
NRCan 23 27 19 69
PHAC 8 18 14 40
QIS N/A 2 4 6
FWS N/A 5 2 7
Total 95 121 93 309
Table 8: Invited presentations
  2011-12 2012-13 2013-14 Total
AAFC 16 19 28 63
DFO 10 12 18 40
ECCC 19 9 21 49
HC 8 8 9 25
NRC 34 0 13 47
NRCan 21 20 26 67
PHAC 5 6 8 19
QIS N/A 12 27 39
FWS N/A 16 6 22
Total 113 102 156 371

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 9: National conference presentations
  2011-12 2012-13 2013-14 Total
AAFC 16 5 8 29
DFO 1 0 4 5
ECCC 11 5 6 22
HC 0 5 4 9
NRC 0 0 2 2
NRCan 14 20 14 48
PHAC 2 10 7 19
QIS N/A 1 2 3
FWS N/A 1 2 3
Total 44 47 49 140

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 10: International conference presentations
  2011-12 2012-13 2013-14 Total
AAFC 17 31 19 67
DFO 8 2 10 20
ECCC 16 7 16 39
HC 9 11 6 26
NRC 0 3 9 12
NRCan 8 9 12 29
PHAC 6 8 11 25
QIS N/A 3 9 12
FWS N/A 4 1 5
Total 64 78 93 235

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 11: Active participations in national conferences (organizer, chair, panel discussion etc.)
  2011-12 2012-13 2013-14 Total
AAFC 11 5 5 21
DFO 0 0 0 0
ECCC 2 4 5 11
HC 0 0 1 1
NRC 2 0 2 4
NRCan 4 4 4 12
PHAC 1 5 2 8
QIS N/A 0 0 0
FWS N/A 0 0 0
Total 20 18 19 57

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 12: Active participation in international conferences (organizer, chair, panel etc.)
  2011-12 2012-13 2013-14 Total
AAFC 2 10 4 16
DFO 0 0 0 0
ECCC 6 3 3 12
HC 1 3 5 9
NRC 0 0 0 0
NRCan 1 3 3 7
PHAC 2 1 4 7
QIS N/A 2 3 5
FWS N/A 4 2 6
Total 12 26 24 62

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 13: Editorial posts for national and international journals (excludes peer reviewers)
  2011-12 2012-13 2013-14 Total
AAFC 11 17 6 34
DFO 0 3 2 5
ECCC 3 4 4 11
HC 0 0 0 0
NRC 0 0 3 3
NRCan 6 4 6 16
PHAC 2 2 2 6
QIS N/A 6 4 10
FWS N/A 1 0 1
Total 22 37 27 86

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 14: Deposits in genomics related databases or libraries
  2011-12 2012-13 2013-14 Total
AAFC 11 16 3 30
DFO 7 4 0 11
ECCC 6 13 8 27
HC 0 5 3 8
NRC 0 0 1 1
NRCan 16 17 35 68
PHAC 17 0 635 652
QIS N/A 0 15 15
FWS N/A 0 0 0
Total 57 55 700 812

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 15: New genomics related databases or libraries
  2011-12 2012-13 2013-14 Total
AAFC 0 6 2 8
DFO 0 0 2 2
ECCC 0 0 2 2
HC 0 0 0 0
NRC 0 2 1 3
NRCan 1 6 1 8
PHAC 1 1 3 5
QIS N/A 0 2 2
FWS N/A 1 0 1
Total 2 17 15 34

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 16: Awards, prizes
  2011-12 2012-13 2013-14 Total
AAFC 1 2 3 6
DFO 0 1 3 4
ECCC 14 2 1 17
HC 1 1 6 8
NRC 1 1 0 2
NRCan 1 1 1 3
PHAC 0 0 0 0
QIS N/A 0 0 0
FWS N/A 4 0 4
Total 18 12 14 44

Source: GRDI – 3 Years Performance Tables 2011-2014

Research Tools and Processes

Research Tools and Processes: Between 2011-12 and 2013-14, the GRDI projects led to the production of 283 research tools and processes (Table 17). The number of research tools and processes produced increased from 78 in 2011-12 to 111 in 2013-14. The production of research tools and processes under Phase V of the initiative is well above the range recorded for Phase IV of the initiative (30). Across departments and agencies, AAFC was the most prolific with 85 research tools and processes produced, followed by ECCC (52), and PHAC (32).

  • A total of 184 research tools were produced under Phase V of the initiative between 2011-12 and 2013-14 (Table 18). The number of research tools produced increased from 50 to 79 over the 3-year period.
  • Research processes accounted for 99 of the research tools and processes produced under Phase V of the initiative (Table 19). The number of research processes developed ranged from 28 to 32 over the 3-year period.
Table 17: Total Research Tools and Processes
  2011-12 2012-13 2013-14 Total
AAFC 34 24 27 85
DFO 4 0 5 9
ECCC 12 23 17 52
HC 1 12 11 24
NRC 8 7 15 30
NRCan 13 7 6 26
PHAC 6 12 14 32
QIS N/A 4 10 14
FWS N/A 5 6 11
Total 78 94 111 283

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 18: Research Tools
  2011-12 2012-13 2013-14 Total
AAFC 19 16 20 55
DFO 3 0 5 8
ECCC 10 13 15 38
HC 4 9 11 24
NRC 4 2 6 12
NRCan 9 4 4 17
PHAC 4 5 6 15
QIS N/A 2 7 9
FWS N/A 4 5 9
Total 50 55 79 184

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 19: Research processes
  2011-12 2012-13 2013-14 Total
AAFC 15 8 7 30
DFO 1 0 0 1
ECCC 2 10 2 14
HC 0 3 0 3
NRC 4 5 9 18
NRCan 4 3 2 9
PHAC 2 7 8 17
QIS N/A 2 3 5
FWS N/A 1 1 2
Total 28 39 32 99

Source: GRDI – 3 Years Performance Tables 2011-2014

Knowledge and Technology Transfer

Knowledge and Technology Transfer: Under Phase V of the GRDI, a total of 513 Knowledge and Technology Transfer activities/outputs were reported (Table 20). The number of knowledge and technology transfer activities and outputs increased from 90 in 2011-12 to 252 in 2013-14. The production of knowledge and technology transfer activities and outputs under Phase V of the initiative is well above the range recorded for Phase IV of the initiative (366).

  • Outreach activities, which include lectures, forums, and open doors, with targeted audiences such as the scientific community, secondary school, industry, and policy makers to cite a few, accounted for 115 of the total number of knowledge and technology transfer activities recorded between 2011-12 and 2013-14 (Table 21). The number of outreach activities gradually increased over the 3-year period from 21 in 2011-12 to 55 in 2013-14.
  • In total, 84 material transfer agreements were reported under Phase V of the initiative (Table 22). Only 6 material transfer agreements were reported in 2011-12, but the following years saw that number increase markedly.
  • A total of 37 transfers of standard operating procedures were recorded between 2011-12 and 2013-14 (Table 23). This category of activities ranged from 4 to 25 per year over the 3-year period, peaking in 2013-14. PHAC (9) and the two shared priority projects contributed the most to this category.
  • Disclosures, which involve sharing of research results with specific collaborators accounted for 14 of the total number of knowledge and technology transfer activities recorded between 2011-12 and 2013-14 (Table 24). The vast majority of disclosures were ported in 2013-14 (12), most of which were undertaken by NRC (11).
  • GRDI projects between 2011-12 and 2013-14 led to 62 active patents, patent applications, and patents issued (Table 25). This category of activities ranged from 13 to 32 per year over the 3-year period, peaking in 2013-14. AAFC (35) and NRC (15) led this category.
  • A total of 4 licenses were issued as a result of GRDI Phase V projects (Table 26).
  • In total, Phase V projects led to 29 new formal collaborative agreements/standard operating protocols (Table 27). The first year corresponded with the highest number of new formal collaborative agreements/standard operating protocols (16).
  • Knowledge transfer workshops with stakeholders/end-users accounted for 46 of the total number of knowledge and technology transfer activities recorded between 2011-12 and 2013-14 (Table 28). This category of activities ranged from 8 to 29 per year over the 3-year period, peaking in 2012-13. The Quarantine and Invasive Species (QIS) project (22), and ECCC mandated projects (10) were particularly successful in leveraging this type of activity.
  • GRDI projects between 2011-12 and 2013-14 generated 122 requests for research results, papers, collaborations (Table 29), the bulk of which were received in 2013-14 (67).
Table 20: Total Knowledge and Technology Transfer
  2011-12 2012-13 2013-14 Total
AAFC 21 85 117 223
DFO 14 9 3 26
ECCC 6 19 21 46
HC 1 14 6 21
NRC 21 9 44 74
NRCan 12 0 3 15
PHAC 15 10 24 49
QIS N/A 19 16 35
FWS N/A 6 18 24
Total 90 171 252 513

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 21: Outreach activities
  2011-12 2012-13 2013-14 Total
AAFC 1 9 18 28
DFO 5 1 0 6
ECCC 6 7 7 20
HC 0 12 6 18
NRC 0 0 8 8
NRCan 0 0 2 2
PHAC 9 8 11 28
QIS N/A 1 2 3
FWS N/A 1 1 2
Total 21 39 55 115

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 22: Material transfer agreements
  2011-12 2012-13 2013-14 Total
AAFC 2 27 21 50
DFO 0 0 0 0
ECCC 0 0 4 4
HC 0 0 0 0
NRC 3 9 12 24
NRCan 1 0 0 1
PHAC 0 0 1 1
QIS N/A 0 0 0
FWS N/A 1 3 4
Total 6 37 41 84

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 23: Transfer of standard operating procedures
  2011-12 2012-13 2013-14 Total
AAFC 0 4 1 5
DFO 1 0 0 1
ECCC 0 1 1 2
HC 0 0 0 0
NRC 0 0 5 5
NRCan 0 0 1 1
PHAC 3 1 5 9
QIS N/A 0 8 8
FWS N/A 2 4 6
Total 4 8 25 37

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 24: Disclosures
  2011-12 2012-13 2013-14 Total
AAFC 0 0 0 0
DFO 0 0 0 0
ECCC 0 0 0 0
HC 0 0 0 0
NRC 2 0 11 13
NRCan 0 0 0 0
PHAC 0 0 1 1
QIS N/A 0 0 0
FWS N/A 0 0 0
Total 2 0 12 14

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 25: Active patents, patent applications, patents issued
  2011-12 2012-13 2013-14 Total
AAFC 1 16 18 35
DFO 0 0 0 0
ECCC 0 0 1 1
HC 1 1 0 2
NRC 11 0 4 15
NRCan 0 0 0 0
PHAC 0 0 1 1
QIS N/A 0 2 2
FWS N/A 0 6 6
Total 13 17 32 62

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 26: Licenses issued
  2011-12 2012-13 2013-14 Total
AAFC 0 0 0 0
DFO 0 0 0 0
ECCC 0 1 0 1
HC 0 0 0 0
NRC 1 0 0 1
NRCan 0 0 0 0
PHAC 0 0 1 1
QIS N/A 0 0 0
FWS N/A 0 1 1
Total 1 1 2 4

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 27: New formal collaborative agreements / standard operating protocols
  2011-12 2012-13 2013-14 Total
AAFC 11 0 0 11
DFO 0 1 1 2
ECCC 0 3 3 6
HC 0 0 0 0
NRC 4 0 3 7
NRCan 1 0 0 1
PHAC 0 0 1 1
QIS N/A 0 0 0
FWS N/A 0 1 1
Total 16 4 9 29

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 28: Knowledge transfer workshops with stakeholders/end-users
  2011-12 2012-13 2013-14 Total
AAFC 6 1 0 7
DFO 1 1 0 2
ECCC 0 7 3 10
HC 0 0 0 0
NRC 0 0 1 1
NRCan 0 0 0 0
PHAC 1 0 1 2
QIS N/A 18 4 22
FWS N/A 2 0 2
Total 8 29 9 46

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 29: Requests for research results, papers, collaborations
  2011-12 2012-13 2013-14 Total
AAFC 0 28 59 87
DFO 7 6 2 15
ECCC 0 0 2 2
HC 0 1 0 1
NRC 0 0 0 0
NRCan 10 0 0 10
PHAC 2 1 2 5
QIS N/A 0 0 0
FWS N/A 0 2 2
Total 19 36 67 122

Source: GRDI – 3 Years Performance Tables 2011-2014

Communication Products

Communication Products: Between 2011-12 and 2013-14, the GRDI projects led to the production of 241 communication products (Table 30: Total Communications Products). The number of communication products increased from 33 in 2011-12 to 144 in 2013-14. Overall, the number of communication products under Phase V well exceeds the number reported under Phase IV (151). Across departments and agencies, NRC was by far the most prolific with more than half (138) of all communication products.

  • Media interviews accounted for 49 of the total number of communication products recorded between 2011-12 and 2013-14 (Table 31). After producing only 4 media interviews in 2011-12, 20 and 25 respectively were reported in 2012-13 and 2013-14.
  • A total of 15 press releases and announcements were reported between 2011-12 and 2013-14 (Table 32). The number of press releases and announcements ranged from 4 to 6 per year over the 3-year period.
  • In total, 103 newspaper and magazine articles were produced as a result of Phase V projects (Table 33). The number of newspaper and magazine articles produced increased dramatically between 2011-12 (6) and 2013-14 (72).
  • GRDI projects under Phase V led to 27 community presentations between 2011-12 and 2013-14 (Table 34). Among DAs, DFO accounted for the largest number of community presentations (10).
  • A total of 47 brochures, factsheets and web pages were reported between 2011-12 and 2013-14 (Table 35). This type of communication products increased from 8 in 2012-13 to 29 in 2013-14.
Table 30: Total Communications Products
  2011-12 2012-13 2013-14 Total
AAFC 4 12 10 26
DFO 7 11 6 24
ECCC 10 10 10 21
HC 3 0 0 3
NRC 6 22 110 138
NRCan 10 1 1 12
PHAC 2 2 4 8
QIS N/A 4 1 5
FWS N/A 2 2 4
Total 33 64 144 241

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 31: Media interviews
  2011-12 2012-13 2013-14 Total
AAFC 0 3 8 11
DFO 0 7 1 8
ECCC 5 0 0 5
HC 0 0 0 0
NRC 0 3 15 18
NRCan 4 0 0 4
PHAC 0 0 0 0
QIS N/A 0 1 1
FWS N/A 2 0 2
Total 4 20 25 49

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 32: Press releases and announcements
  2011-12 2012-13 2013-14 Total
AAFC 1 1 0 2
DFO 0 0 0 0
ECCC 1 1 1 3
HC 0 0 0 0
NRC 3 1 4 8
NRCan 1 0 0 1
PHAC 0 1 1 2
QIS N/A 0 0 0
FWS N/A 0 0 0
Total 5 4 6 15

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 33: Newspaper and magazine articles
  2011-12 2012-13 2013-14 Total
AAFC 2 6 1 9
DFO 0 2 0 2
ECCC 0 3 3 3
HC 0 0 0 0
NRC 1 17 68 86
NRCan 3 0 0 3
PHAC 0 0 0 0
QIS N/A 0 0 0
FWS N/A 0 0 0
Total 6 25 72 103

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 34: Community presentations
  2011-12 2012-13 2013-14 Total
AAFC 0 0 0 0
DFO 6 0 4 10
ECCC 1 4 4 5
HC 3 0 0 3
NRC 0 0 2 2
NRCan 1 0 0 1
PHAC 0 0 0 0
QIS N/A 4 0 4
FWS N/A 0 2 2
Total 10 5 12 27

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 35: Brochures, fact sheets, web pages
  2011-12 2012-13 2013-14 Total
AAFC 1 2 1 4
DFO 1 2 1 4
ECCC 3 2 2 7
HC 0 0 0 0
NRC 2 1 21 24
NRCan 1 1 1 3
PHAC 2 1 3 6
QIS N/A 0 0 0
FWS N/A 0 0 0
Total 10 9 29 48

Source: GRDI – 3 Years Performance Tables 2011-2014

Research and Technical Personnel

Research & technical personnel involved: Under Phase V of the GRDI, a total of 2,410 personnel were involved in projects (Table 36), accounting for almost 1,100 FTEs (Table 37). The number of personnel involved in GRDI projects grew from 661 in 2011-12 to 893 in 2012-13 before slightly decreasing the following year (856). The total number of research and technical personnel involved under Phase V well exceeds the numbers reported under Phase IV (1,690). Across departments and agencies, AAFC led the way with 545 research and technical personnel, followed by NRC (432).

  • A total of 737 research scientists were involved in projects under Phase V (Table 38). The number of scientists involved in projects ranged from 193 to 274 per year over the period.
  • In total, 348 research professionals were involved in GRDI projects between 2011-12 and 2013-14 (Table 39). The number of research professionals involved in projects ranged from 81 to 138 per year over the period. The Food and Water Safety shared priority project (69) and projects under HC (58) reported the highest number of research professionals over the 3-year period.
  • Research technicians accounted for 779 of the total number of research and technical personnel involved under Phase V (Table 40). The number of research technicians involved in projects ranged from 207 to 303 per year over the period.
  • A total of 189 post-doctoral fellows were involved in projects under Phase V (Table 41). The number of research technicians involved in projects ranged from 59 to 67 per year over the period. The highest numbers of post-doctoral fellows were reported by NRC (52), NRCan (41) and AAFC (40).
  • In total, 165 graduate students were involved in GRDI projects between 2011-12 and 2013-14 (Table 42: Graduate students). Graduate student involvement gradually decreased from 69 students in 2011-12 to 44 in 2013-14. AAFC reported by far the largest number of graduate students (68) involved in the initiative.
  • Undergraduate students accounted for 175 of the total number of research and technical personnel involved under Phase V. Undergraduate student involvement unlike that of graduate students increased from 43 students in 2011-12 to 78 in 2013-14. The QIS project reported the largest number of undergraduate students (57).
Table 36: Total Personnel
  2011-12 2012-13 2013-14 Total
AAFC 203 159 183 545
DFO 32 38 42 112
ECCC 81 77 85 243
HC 60 58 55 173
NRC 162 178 92 432
NRCan 74 69 64 207
PHAC 61 65 68 194
QIS N/A 103 119 222
FWS N/A 146 148 294
Total 661 893 856 2,410

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 37: Total FTEs
  2011-12 2012-13 2013-14 Total
AAFC 76.8 82.9 92.2 251.9
DFO 24 10.3 9.3 43.6
ECCC 56 29.5 39.9 125.4
HC 46.6 24 20.3 90.9
NRC 84.3 92.6 68.5 245.4
NRCan 38.6 38.6 35 112.2
PHAC 21 22.2 22.9 66.1
QIS N/A 28.8 28.8 57.6
FWS N/A 51.5 53.5 105
Total 347.3 380.4 370.4 1098.1

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 38: Research scientists
  2011-12 2012-13 2013-14 Total
AAFC 59 31 55 145
DFO 11 11 12 34
ECCC 20 19 17 56
HC 17 16 15 48
NRC 40 53 27 120
NRCan 14 15 13 42
PHAC 32 30 32 94
QIS N/A 30 30 60
FWS N/A 69 69 138
Total 193 274 270 737

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 39: Research professionals
  2011-12 2012-13 2013-14 Total
AAFC 0 5 5 10
DFO 8 7 8 23
ECCC 12 11 6 29
HC 19 22 17 58
NRC 14 15 12 41
NRCan 15 13 13 41
PHAC 13 14 16 43
QIS N/A 17 17 34
FWS N/A 34 35 69
Total 81 138 129 348

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 40: Research technicians
  2011-12 2012-13 2013-14 Total
AAFC 79 87 83 249
DFO 12 15 16 43
ECCC 17 21 28 66
HC 11 10 15 36
NRC 64 87 37 188
NRCan 17 17 18 52
PHAC 7 6 4 17
QIS N/A 23 31 54
FWS N/A 37 37 74
Total 207 303 269 779

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 41: Post-doctoral fellows
  2011-12 2012-13 2013-14 Total
AAFC 19 10 11 40
DFO 0 1 1 2
ECCC 6 7 9 22
HC 3 6 6 15
NRC 22 20 10 52
NRCan 13 15 13 41
PHAC 2 3 3 8
QIS N/A 5 6 11
FWS N/A 0 0 0
Total 63 67 59 189

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 42: Graduate students
  2011-12 2012-13 2013-14 Total
AAFC 30 26 12 68
DFO 1 2 5 8
ECCC 18 5 10 33
HC 5 3 2 10
NRC 8 0 0 8
NRCan 5 5 4 14
PHAC 2 3 3 8
QIS N/A 2 2 4
FWS N/A 6 6 12
Total 69 52 44 165

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 1: Actual program expenditures
Table 43: Undergraduate students
  2011-12 2012-13 2013-14 Total
AAFC 16 0 14 30
DFO 0 2 0 2
ECCC 7 10 15 32
HC 5 1 0 6
NRC 0 3 5 8
NRCan 10 4 2 16
PHAC 5 9 9 23
QIS N/A 25 32 57
FWS N/A 0 1 1
Total 43 54 78 175

Source: GRDI – 3 Years Performance Tables 2011-2014

Table 44: Administrative officers
  2011-12 2012-13 2013-14 Total
AAFC 0 0 3 3
DFO 0 0 0 0
ECCC 1 4 0 5
HC 0 0 0 0
NRC 4 0 1 5
NRCan 0 0 1 1
PHAC 0 0 1 1
QIS N/A 1 1 2
FWS N/A 0 0 0
Total 5 5 7 17

Source: GRDI – 3 Years Performance Tables 2011-2014

Footnotes

Footnote 1

Science-Metrix (2011). Evaluation of the Genomics R&D Initiative (GRDI) - Final Evaluation Report.

Return to footnote 1 referrer

Footnote 2

Horizontal Evaluation of GRDI – Meeting in preparation for the data collection and reporting phases WebEx Presentation to the IEWG December 3, 2015.

Return to footnote 2 referrer

Footnote 3

Horizontal Performance Measurement Strategy (PMS) for the Genomics R&D Initiative (GRDI) (2011).

Return to footnote 3 referrer

Footnote 4

GRDI Annual Performance Report 2013-2014.

Return to footnote 4 referrer

Footnote 5

Genome Canada (2015). 2014-2015 Annual Report: Global Challenges – Genomic Solutions.

Return to footnote 5 referrer

Footnote 6

Genome Canada (2010). Perspectives on Progress | Annual Report 2009–2010.

Return to footnote 6 referrer

Footnote 7

GRDI Annual Performance Report 2013-2014.

Return to footnote 7 referrer

Footnote 8

Ibid.

Return to footnote 8 referrer

Footnote 9

Ibid.

Return to footnote 9 referrer

Footnote 10

GRDI Best Practices 2012.

Return to footnote 10 referrer

Footnote 11

Government of Canada (GoC) (2013). Genomics research and development (R&D) investments in departments and agencies participating in the genomics R&D initiative in fiscal year 2011-2012.

Return to footnote 11 referrer

Footnote 12

Ibid.

Return to footnote 12 referrer

Footnote 13

Ibid.

Return to footnote 13 referrer

Footnote 14

Ibid.

Return to footnote 14 referrer

Footnote 15

Bank of Canada. Consumer Price Index, 2000 to Present

Return to footnote 15 referrer

Footnote 16

NRC (2009). NRC Business Case: GRDI Policy Framework: Component 2 – Mandated Research.

Return to footnote 16 referrer

Footnote 17

ECCC (2009). Genomics R&D Initiative – Environment Canada Business Case.

Return to footnote 17 referrer

Footnote 18

GoC (2013). Genomics research and development (R&D) investments in departments and agencies participating in the genomics R&D initiative in fiscal year 2011-2012.

Return to footnote 18 referrer

Footnote 19

GRDI Best Practices 2012.

Return to footnote 19 referrer

Footnote 20

Ibid.

Return to footnote 20 referrer

Footnote 21

Story 3 – source tracking.

Return to footnote 21 referrer

Footnote 22

Ibid.

Return to footnote 22 referrer

Footnote 23

GRDI (2011). Performance Measurement Strategy.

Return to footnote 23 referrer

Footnote 24

GoC (2014). Genomics R&D Initiative: Annual Performance Report 2013-2014

Return to footnote 24 referrer

Footnote 25

Ibid.

Return to footnote 25 referrer

Footnote 26

Ibid.

Return to footnote 26 referrer

Footnote 27

Ibid.

Return to footnote 27 referrer

Footnote 28

CFIA (2009) Canadian Food Inspection Agency GRDI: Business Case – Draft VII.

Return to footnote 28 referrer

Footnote 29

GoC (2014). Genomics R&D Initiative: Annual Performance Report 2013-2014.

Return to footnote 29 referrer

Footnote 30

Ibid.

Return to footnote 30 referrer

Footnote 31

Ibid.

Return to footnote 31 referrer

Footnote 32

Ibid.

Return to footnote 32 referrer

Footnote 33

Ibid.

Return to footnote 33 referrer

Footnote 34

Minister of Science Mandate Letter

Return to footnote 34 referrer

Footnote 35

GoC (2013). Genomics research and development (R&D) investments in departments and agencies participating in the genomics R&D initiative in fiscal year 2011-2012.

Return to footnote 35 referrer

Footnote 36

Science-Metrix (2011). Evaluation of the Genomics R&D Initiative (GRDI) - Final Evaluation Report.

Return to footnote 36 referrer

Footnote 37

GoC (2011). Genomics R&D Initiative (GRDI): Governance Framework - Phase V.

Return to footnote 37 referrer

Footnote 38

Ibid.

Return to footnote 38 referrer

Footnote 39

GRDI (2014. Examples of Results to Date and Economic Impacts.

Return to footnote 39 referrer

Footnote 40

Ibid.

Return to footnote 40 referrer

Footnote 41

Applying toxicogenomics to new and emerging environmental issues

Return to footnote 41 referrer

Footnote 42

Scientific contributions include scientific information and publications from any of the project team member as long as they are related to the GRDI project. This category includes publications in refereed journals, publications in refereed conference, proceedings, technical reports, book chapters, other publications, poster presentations at conferences, invited presentations, national conference presentations, international conference presentations, participations in national conferences, participations in international, conferences, editorial posts for national and international Journals, and deposits in genomics related databases or libraries, new genomics related databases or libraries genomics related databases or libraries, awards, prizes.

Return to footnote 42 referrer

Footnote 43

Research tools and processes deriving from previous phases of the GRDI if produced in 2012-2013, as well as produced in previous years if they have been improved since last reported. This category includes: inter alia, DNA extraction protocols, standard operating procedures, assays, diagnostic markers, data visualization tools, etc.

Return to footnote 43 referrer

Footnote 44

This category includes: material transfer agreements, transfer of standard operating, procedures, disclosures, active patents, patent applications, patents issued, licenses issued, formal collaborative agreements standard operating protocols, knowledge transfer workshops with stakeholders/end-users, requests for research results, papers, collaborations, and outreach activities for disseminating results to end-users.

Return to footnote 44 referrer

Footnote 45

This category includes: media interviews, press releases, community presentations (science fairs and events, schools), brochures, fact sheets, web pages.

Return to footnote 45 referrer

Footnote 46

Scientific and technical personnel involved includes everyone who worked on the project, including but not exclusive of personnel financed through GRDI funds.

Return to footnote 46 referrer

Footnote 47

Working Group for the Genomics R&D Initiative. GRDI Annual Performance Report 2013-2014.

Return to footnote 47 referrer

Footnote 48

Krukczkiewicz P., Mutschall S., Barker D., Thomas J., Van Domselaar G., P. J. Gannon V., D. Carrillo C. and N. Taboada E.. MIST: A Tool for Rapid in silico Generation of Molecular Data from Bacterial Genome Sequences. In Proceedings of the International Conference on Bioinformatics Models, Methods and Algorithms (BIOINFORMATICS-2013), pages 316-323.

Return to footnote 48 referrer

Footnote 49

Gene Expression Omnibus (GEO) is a database repository of high throughput gene expression data and hybridization arrays, chips, microarrays. www.ncbi.nlm.nih.gov/geo

Return to footnote 49 referrer

Footnote 50

GRDI Annual Performance Report 2012-2013.

Return to footnote 50 referrer

Footnote 51

Ibid.

Return to footnote 51 referrer

Footnote 52

GRDI Annual Performance Report 2013-2014.

Return to footnote 52 referrer

Footnote 53

Ibid.

Return to footnote 53 referrer

Footnote 54

Ibid.

Return to footnote 54 referrer

Footnote 55

Science-Metrix (2011). Evaluation of the Genomics R&D Initiative (GRDI) - Final Evaluation Report

Return to footnote 55 referrer

Footnote 56

Science-Metrix (2011). Evaluation of the Genomics R&D Initiative (GRDI) - Final Evaluation Report, pg.4.

Return to footnote 56 referrer

Footnote 57

GRDI Annual Performance Report 2013-2014.

Return to footnote 57 referrer

Footnote 58

GoC (2015). GRDI Innovation Management Strategy for Shared Priority Projects.

Return to footnote 58 referrer

Footnote 59

GoC (2014). Genomics R&D Initiative Strategic Planning Meeting: Innovation Management Strategy.

Return to footnote 59 referrer

Footnote 60

Ibid.

Return to footnote 60 referrer

Footnote 61

Ibid.

Return to footnote 61 referrer

Footnote 62

Ibid.

Return to footnote 62 referrer

Footnote 63

Workshop Report (2016). FWS – GRDI Knowledge Transfer Workshop –Ottawa, February 24-25, 2016

Return to footnote 63 referrer

Footnote 64

Ibid.

Return to footnote 64 referrer

Footnote 65

Ibid.

Return to footnote 65 referrer

Footnote 66

Ibid.

Return to footnote 66 referrer

Footnote 67

GoC (2015). Genomics Research and Development Initiative (GRDI) innovation management strategy for shared priority projects.

Return to footnote 67 referrer

Footnote 68

Ibid.

Return to footnote 68 referrer

Footnote 69

Ibid.

Return to footnote 69 referrer

Footnote 70

GoC (2015). GRDI Innovation Management Strategy.

Return to footnote 70 referrer

Footnote 71

Ibid.

Return to footnote 71 referrer

Footnote 72

Ibid.

Return to footnote 72 referrer

Footnote 73

Osterwalder, Alexander and Yves Pigneur. 2010. Business Model Generation: A Handbook for Visionaries, Game Changers, and Challengers

Return to footnote 73 referrer

Footnote 74

The Business Model Canvas

Return to footnote 74 referrer

Footnote 75

Ibid.

Return to footnote 75 referrer

Footnote 76

Ibid.

Return to footnote 76 referrer

Footnote 77

Ibid.

Return to footnote 77 referrer

Footnote 78

GRDI Interdepartmental ADM Coordination Committee Terms of Reference (Annex 4 of GRDI Governance Framework Phase V. Final, November 2011).

Return to footnote 78 referrer

Footnote 79

GRDI Governance Framework Phase V. Final, November 2011.

Return to footnote 79 referrer

Footnote 80

Ibid.

Return to footnote 80 referrer

Footnote 81

Ibid.

Return to footnote 81 referrer

Footnote 82

Ibid. Note that planning of projects in the first years of the SPPs pertained to Phase V only. In Phase VI, projects are planned in advance of funding availability.

Return to footnote 82 referrer

Footnote 83

GRDI Annual Performance Report 2013-2014.

Return to footnote 83 referrer

Footnote 84

Ibid.

Return to footnote 84 referrer

Footnote 85

NRC (2016). GRDI Evaluation Collaboration Networks Presentation.

Return to footnote 85 referrer

Footnote 86

Note that it is possible for a publication to feature collaboration (i.e., more than one DA or scientist contributing to the research) but not co-publication (i.e., authors from more than one DA).

Return to footnote 86 referrer

Footnote 87

NRC (2016). GRDI Evaluation Collaboration Networks Presentation.

Return to footnote 87 referrer

Footnote 88

Ibid.

Return to footnote 88 referrer

Footnote 89

Ibid.

Return to footnote 89 referrer

Footnote 90

GRDI. Success Story 2 – Collaboration.

Return to footnote 90 referrer

Footnote 91

GoC (2014). Genomics R&D Initiative Strategic Planning Meeting: Innovation Management Strategy. Meeting Report.

Return to footnote 91 referrer

Footnote 92

Ibid.

Return to footnote 93 referrer

Footnote 93

Note that this challenge has been addressed to some extent since the SPP project management plans for Phase VI (formerly referred to as charters) have been reviewed and validated by end-users and peer reviewers.

Return to footnote 93 referrer

Footnote 94

GoC (2014). Genomics R&D Initiative Strategic Planning Meeting: Innovation Management Strategy. Meeting Report.

Return to footnote 94 referrer

Footnote 95

Ibid.

Return to footnote 95 referrer

Footnote 96

Ibid.

Return to footnote 96 referrer

Footnote 97

Ibid.

Return to footnote 97 referrer

Footnote 98

Ibid.

Return to footnote 98 referrer

Footnote 99

Ibid.

Return to footnote 99 referrer

Footnote 100

Ibid.

Return to footnote 100 referrer

Footnote 101

Postdoctoral Research Pilot Program

Return to footnote 101 referrer

Footnote 102

GRDI Annual Performance Report 2013-2014.

Return to footnote 102 referrer

Footnote 103

GRDI Annual Performance Report 2012-2013.

Return to footnote 103 referrer

Footnote 104

Ibid.

Return to footnote 104 referrer

Footnote 105

GRDI Annual Performance Report 2013-2014.

Return to footnote 105 referrer

Footnote 106

Ibid.

Return to footnote 106 referrer