Summary Report of the Horizontal Evaluation of the Genomics R&D Initiative

Program description

The Genomics Research and Development Initiative (GRDI) was created in 1999 and is currently implementing its Phase VI round of funding. The program's objective is to establish and maintain core genomics R&D capacity in the following federal departments and agencies (DAs):

• Agriculture and Agri-food Canada (AAFC);
• Environment and Climate Change Canada (ECCC);
• Fisheries and Oceans Canada (DFO);
• Health Canada (HC);
• Public Health Agency of Canada (PHAC);
• National Research Council (NRC);
• Natural Resources Canada (NRCan); and,
• Canadian Food Inspection Agency (CFIA) – since Phase VI.

A total of $59.7 million was invested in support of genomic R&D as part of Phase V. Research supported by GRDI seeks to uphold regulatory, public policy, and operational mandates in important areas such as health, food safety, sound management of natural resources, a sustainable and competitive agriculture sector, and environmental protection, with strong collaborations with university and private sectors. In Phase V (2011-14), GRDI supported 73 of these individual DA-led mandated research projects. As well, the program funded two shared priority projects (SPPs) between January 2012 and March 2016: Food and water safety in Canada through an integrated federal genomics initiative (also referred to as the Food and Water Safety or FWS project); and Protection of Canadian biodiversity and trade from the impacts of global change through improved ability to monitor invasive alien and quarantine species (also referred to as the Quarantine and Invasive Species or QIS project).

There are two types of end-users for GRDI-funded projects: internal and external end-users. Internal end-users are the most common type for GRDI-funded projects and are those inside federal government (e.g., scientists in the labs, inspectors on the ground, border agents, trade negotiators). External end-users are those outside federal government (e.g., industry using a patented technology, industry changing their processes due to policy change, international regulatory agencies using/adopting the technology).

Scope and methodology

GRDI foundational documents for Phase VI funding require a horizontal evaluation of the program in 2015-16. The evaluation focused on Phase V of GRDI, which spanned fiscal years 2011-12 to 2013-14. However, fiscal years 2014-15 and 2015-16 were included insofar as work associated with SPPs continued to be conducted during these years. The evaluation addressed nine questions in three issue areas: relevance; performance- effectiveness; and performance – efficiency and economy. The methodology employed multiple lines of evidence, including both qualitative and quantitative and relying on both primary and secondary sources of data. The main limitations of the methodology included: validity and reliability of administrative data; and a likely underestimation of the extent of collaborations in SPPs assessed through the bibliometric study.

Overall evaluation findings

The evaluation found that, while the needs of DAs for genomics R&D funding have evolved, there is a continued need for GRDI. For many DAs, the need continues to relate to capacity building (in both basic and advanced techniques). Several DAs have also seen the need move towards the application of genomics-based technologies and approaches. In addition, there is clear alignment of GRDI with the mandates and priorities of participating DAs and the federal government more generally. The evaluation confirmed that the program is consistent with, and contributes to, participating DAs' legislated mandates related to the health and safety of Canadians as well as the sustainability of Canada's natural resources (via regulatory activities) and support for industry (via economic development and regulatory activities). The evaluation further found that there is little duplication with other similar organizations or federal programs.

In spite of large amounts of funds being leveraged and some DAs making investments in genomics R&D, it is clear that the need for continued support for these types of projects does not show signs of abating. In the absence of GRDI, the evaluation found that DAs' abilities to deliver on their mandates in the future might be negatively impacted as it is unclear what amount of DA funding for genomics R&D would be available given other DA priorities.

The continued need for GRDI is taking place within a context of a decreasing real value of the funding (due to inflation since inception in 1999 without commensurate increases in funding), funding being shared among more participating DAs, SPPs vying for part of the GRDI funding envelope and increasing costs associated with the research. Research costs are increasing because more effort is now being focused on applications and transferring knowledge to end-users, which are more resource intensive phases of R&D.

In terms of effectiveness, the evaluation found that Phase V projects have been successful in the development of innovative knowledge and technologies and influencing evidence-based public policy. The program has produced expected outputs and exceeded targets related to these. Moreover, the evaluation confirmed that many GRDI projects have successfully transferred knowledge and technologies to end-users, both internal and external to the federal government. Many more GRDI projects have the potential to similarly impact end-users. Nevertheless, there continue to be opportunities, especially for the SPPs, to build on the lessons and best practices from Phase V in subsequent phases of the program. Some of these have already been addressed. In particular, content outlined in the Innovation Management Strategy (IMS), though intended for SPPs, offers examples and tools to ensure end-user engagement and take-up of GRDI-funded knowledge and technologies that could be considered beyond the SPPs.

In addition to the transfer of knowledge and technologies, the evaluation found evidence that GRDI-funded projects are likely to have real and lasting longer-term impacts, most of which remain anticipated or potential impacts at this time. These impacts are significant and are expected to lead to billions of dollars in benefits for end-users (through avoided costs, increased trade, etc.). More specifically, the evaluation found that GRDI-funded projects are likely to result in: improved public health and wellness; avoided health system costs; efficiencies and avoided losses for the public and private sectors; environmental sustainability; improved detection of invasive species; improved strains/traits of plants, trees and animals of commercial value; and improved fisheries tracking and management.

The introduction of interdepartmental SPPs in Phase V has proven to be a strong feature of the program and the evaluation found that there is a continued need for this type of interdepartmental collaboration. The two SPPs funded in Phase V were found to be well-managed and have achieved significant results, which likely would not have occurred in the absence of GRDI funding for the interdepartmental nature of the projects. As well, the degree of interdepartmental collaboration stemming from the SPPs has been significant. However, there is no consistent or planned approach related to what will happen to ongoing SPPs when the GRDI funding ends. Consequently, there is a risk that the results achieved to date and the potential for further impacts might be slowed or otherwise diminished. Some concerns were also raised about the selection of SPPs (including the role of scientists), and a review of related documents confirmed some lack of transparency in terms of how sub-projects are selected. As well, since sub-projects are ultimately directed by the Scientific Coordinator from one participating DA (with support from Sub-Project Leads), there is the potential for the perception of conflict of interest. Finally, the QIS case study suggests that mid-year reporting may be overly burdensome (on top of annual reporting). Both case studies brought the suggestion that there should be more interaction between the various scientists contributing to the SPP goals.

Other challenges associated with the SPPs included those pertaining to: end-user engagement; sharing of data/materials and building bioinformatics capacity; management of IP ; a short timeframe for projects of this magnitude; interdepartmental financial and administrative issues; and delays associated with the hiring of highly qualified personnel. The IMS was developed with a focus on the first three challenges in particular (among other matters), and is being applied during Phase VI SPPs.

In terms of efficiency and effectiveness, the evaluation found that the program is both efficient (in terms of its horizontal governance) and effective (in terms of its use of collaborations, leveraging of funds and delivery approach). The evaluation did not reveal any significant opportunities for improvement in program governance or delivery. As well, the evaluation found that the program has made improvements in terms of its performance measurement approaches and systems. However, a few challenges remain, including reporting inconsistencies and the lack of mechanisms to compile and/or track supporting evidence by project.

Recommendations and Management Response