Status: | Active |
---|---|
Effective date: | December 5, 2023 |
Prepared by: | Office of Audit and Evaluation |
Approval: | NRC's President |
Cat. No.: | NR16-434/2-2024E-PDF |
ISBN: | 978-0-660-69473-3 |
Related document: | Evaluation of the Collaborative Science, Technology and Innovation Program 2023-24 |
Alternate format: | Evaluation of the Collaborative Science, Technology and Innovation Program 2023-24 - summary report (PDF, 229 KB) |
About the program
The National Research Council’s Collaborative Science, Technology and Innovation Program (CSTIP) is a transfer payment program established in fiscal year (FY) 2018-19 to position the NRC at the centre of research excellence and collaboration. With $150 million in federal funding over 5 years and $30M per year ongoing, CSTIP aims to catalyze collaborative, high-risk, high-reward research and deliver tangible scientific and technological breakthroughs that address some of Canada’s most pressing public policy challenges. CSTIP uses grant and contribution (G&C) funding to convene research teams from across academia, industry and government.
NRC’s National Program Office (NPO) is responsible for day-to-day administration and oversight of CSTIP, including program development support, due diligence, funding decisions and agreements, and performance monitoring. A key role within NPO is the innovation investment advisor (IIA). IIAs act as the liaison between NPO and Challenge programs, playing an impartial advisory role or ‘challenge function’, in the strategic oversight and delivery of the programs and individual research and development (R&D) projects.
The focus of the evaluation was placed on the Challenge program stream within CSTIP. Challenge programs present a new platform for the NRC to collaborate with external stakeholders, both as a funding agent and an active R&D collaborator.
Challenge programs
- $75 million in G&Cs were disbursed to external collaborators between April 2018 and February 2023
- 219 grants and 44 contributions were provided to external collaborators
- 85% of G&Cs were provided to academia, versus 5% to industry
- All 14 of the NRC’s research centres participate with an estimated $47 million in in-kind contributions
Challenge programs active during the evaluation period (2018-19 to 2022-23)
- Aging in Place
- Applied Quantum Computing
- Arctic and Northern
- Artificial Intelligence for Design
- Disruptive Technology Solutions for Cell and Gene Therapy
- High-throughput and Secure Networks
- Internet of Things: Quantum Sensors
- Materials for Clean Fuels
- Pandemic Response
Key findings
Program model
The CSTIP collaborative R&D initiative provides a flexible program model for funding collaborative research towards addressing societal challenges. Its flexibility was by design, which enabled some programs to leverage both in-house resources and external funding opportunities where they align with their specific challenge mission. The model is effective at enhancing collaboration with external researchers, mainly in academia, through the provision of G&Cs.
To date, Challenge programs have attracted collaborators to work with the NRC, co-author publications and generate intellectual property (IP). Feedback from G&C recipients and NRC researchers on collaborations has been positive.
As familiarity with CSTIP increased, the model has also been effective at enhancing program level collaboration across the NRC, with CSTIP acting as a ‘change program’ for the NRC. Amendments to CSTIP’s terms and conditions supported more inclusive collaboration. Programs incorporated external guidance through committees at different stages in their lifecycles.
It is too early to conclude whether the Challenge program model provides an effective structure to consistently advance funded research towards breakthrough outcomes, given that round 1 programs only passed the midpoint of the 7-year lifecycles in FY 2023. The program model does not provide a common and well-defined structure for managing the advancement of project portfolios (i.e., project maturation and scale-up or termination).
Other challenges to the model included resource allocation pressures at both the CSTIP and program levels, unclear roles and responsibilities, and mixed expectations for research centre participation and program outcomes.
Processes and controls
CSTIP processes and controls have been adequate to support program implementation, and NPO has demonstrated a commitment to continuous improvement. Processes are generally effective, and viewed positively by collaborators and programs, but may be strengthened overall if integrated. Intra-NRC collaboration, tech transfer, and risk-assessment could be strengthened to enhance CSTIP impact.
Overall, CSTIP was launched before processes and controls could be fully developed by NPO. Therefore, NPO has been playing ‘catch up’ over the past 5 years, developing the structure necessary to support CSTIP collaborative R&D’s goals, while also responding to changes in CSTIP’s terms and conditions and to the growth in the total number of Challenge programs.
Information management
Program management has sufficient information to administer CSTIP. NPO and programs are tracking program implementation and project activity, generally, but there are gaps for tracking efficiency and progress towards outcomes. Some key performance indicators tracked at the program or project level are not integrated, which limits overall performance monitoring.
Mid-term reviews provide necessary checks on program performance, but those conducted thus far were limited in their ability to inform adjustments and assess success to date. Additionally, there is no plan for the assessment of individual programs’ long-term outcomes (i.e., beyond the midpoint). As such, program management and NRC executives lack information on the extent to which programs are addressing their respective challenges.
Recommendations
There were 6 recommendations for improvement made to the NRC’s Business and Professional Services (BPS) in the areas of:
- priorities, expectations and organizational culture
- accountability, ownership and oversight
- program assessment
- roles and responsibilities for the Challenge function
- sustainability and scalability
- system integration and outcome-based managemen
About the evaluation approach
This evaluation of CSTIP covered all fiscal years since the program’s inception, from 2018-19 to 2022-23. This first evaluation of CSTIP is a process evaluation focused on collaborative R&D initiatives, particularly the Challenge programs. It assessed the extent to which Challenge programs were set up to achieve their intended goals and whether progress to date has been supported by CSTIP’s collaborative R&D framework. A mixed-methods approach included document and data reviews, internal and external interviews, case studies and a comparative analysis of similar programs.