Evaluation of the Collaborative Science, Technology and Innovation Program 2023-24

Abbreviations

AIP

Aging in Place

ARPA-E

Advanced Research Projects Agency-Energy

BPS

Business and Professional Services

CGT

Disruptive Technology Solutions for Cell and Gene Therapy

COI

Conflict of interest

CSTIP

Collaborative Science, Technology and Innovation Program

DARPA

Defense Advanced Research Projects Agency

DG

Director general

EDI

Equity, diversity and inclusion

EPRC

Expert Peer Review Committee

EIC

European Innovation Council

FRPC

Financial Resources and Program Committee

G&C

Grants and contributions

GBA Plus

Gender-based Analysis Plus

HTSN

High-throughput and Secure Networks

ICI

Impact Canada Initiative

IIA

Innovation investment advisor

KPI

Key performance indicator

MCF

Materials for Clean Fuels

MTR

Mid-term review

NPBS

National Programs and Business Services

NPO

National Program Office

NRC IRAP

Industrial Research Assistance Program

O&M

Operations and maintenance

PAC

Program Advisory Committee

PCR

Project close-out report

PRCP

Pandemic Response Challenge Program

PSC

Program Steering Committee

PTP

Policy on Transfer Payments

R&D

Research and development

RC

Research centre

SEC

Senior Executive Committee

T&Cs

Terms and conditions

TBS

Treasury Board Secretariat

TRL

Technology readiness level

Introduction

The evaluation of the National Research Council’s (NRC) Collaborative Science, Technology and Innovation Program (CSTIP) covers the 5-year period from 2018-19 to 2022-23. It was carried out in accordance with the NRC Departmental Evaluation Plan, the Treasury Board Policy on Results (2016) and in compliance with the Financial Administration Act. This is the first evaluation of CSTIP and covers all fiscal years since the program’s inception.

This report begins by providing a profile of CSTIP. It then presents findings on the extent to which CSTIP’s program model, processes and controls, and information management are adequate to support the achievement of program objectives. Following the findings are recommendations for improvement.

Evaluation approach

Scope

This is a process evaluation of CSTIP’s collaborative research and development (R&D) initiatives, focussed particularly on the Challenge programs. As a process evaluation, it assessed the extent to which programs were set up to achieve their intended goals and whether progress to date has been supported by CSTIP’s collaborative R&D framework. It did not assess program outcomes. Focus was placed on the Challenge programs as they have been identified as an area of potential future growth for the NRC.

Approach

This evaluation was led by the NRC’s Office of Audit and Evaluation (OAE). It applied a mixed-methods approach, incorporating both qualitative and quantitative data from several lines of evidence. This allowed for the triangulation of data. A Gender-based Analysis Plus (GBA Plus) lens was applied throughout the evaluation.

Methods

Lines of evidence and analytical methods that support the evaluation include:

• document review (internal and external sources)
• data review (administrative, financial, program, project, and performance data)
• interviews (n=73 interviewees)
  • NRC staff (n=56 interviewees)
  • external stakeholders (n=17 interviewees)
• case studies (n=3 Challenge programs), including document and data review, case-study specific interviews (n=29 interviewees, included above) and the review of a non-random sample of projects (n=11)
• comparative analysis (n=4 comparator programs), including the review of program documents and interviews (n=6 interviewees)

See appendix A for detailed information on the methods and appendix B for limitations.

Evaluation questions

1. Is the CSTIP collaborative R&D program supportive of the intended goals to foster breakthrough research that will benefit Canada and enhance collaboration with and within the NRC?
   1. Does the program model support intended goals?
   2. Is progress towards the achievement of goals supported by effective and efficient processes and controls?
2. Does program management have access to information needed to make informed decisions?
3. How does the Challenge program model compare to similar international and Canadian programs?

Profile

The Collaborative Science, Technology and Innovation Program (CSTIP) uses grant and contribution (G&C) funding to bring together the best minds from across the research ecosystem to address Canada’s most pressing public policy challenges and deliver scientific and technological breakthroughs.

Program overview

The National Research Council’s (NRC) Collaborative Science, Technology and Innovation Program (CSTIP) is a transfer payment program established in fiscal year (FY) 2018-19 in an effort to position the NRC at the centre of research excellence and collaboration. With $150 million (M) in federal funding over 5 years and $30M per year ongoing, CSTIP aims to catalyze collaborative, high-risk, high-reward research and deliver tangible scientific and technological breakthroughs that address some of Canada’s most pressing public policy challenges. CSTIP uses G&C funding to convene strategic, highly-dynamic research teams from across academia, industry and government.

CSTIP includes 3 separate components that leverage G&C funding, provided and administered by the NRC’s National Program Office (NPO), to engage with external collaborators:

1. Collaborative Research and Development (R&D) initiatives ($107M in G&Cs to date): Comprised of the Cluster Support and Challenge programs, these initiatives bring together the NRC’s national network of researchers and facilities with collaborators from industry, academia, and government in order to tackle specific research objectives across a variety of scientific disciplines.
   Cluster Support programs ($32M in G&Cs to date): Support the objectives of Canada's 5 Global Innovation Clusters.
   Challenge programs^{Footnote 1} ($75M in G&Cs to date): Support the development of disruptive technologies to address social, economic and environmental challenges affecting Canadians.
2. Ideation Fund ($9.6M in G&Cs to date): The Ideation Fund, which includes the New Beginnings and Small Teams initiatives, supports exploratory, transformative research by NRC scientists collaborating with external partners.
3. Outreach initiative ($1.2M in G&Cs to date): The Outreach initiative aims to increase the NRC’s outreach, engagement, and support of research excellence across science, technology, engineering and mathematics, with a focus on under-represented diversity groups.

Focus on the Challenge programs

The Challenge programs present a new platform for the NRC to collaborate with external stakeholders—one where it is both a funding agent and an active collaborator. CSTIP has launched 9 Challenge programs since 2018-19, with 3 more currently in development.

The Challenge programs seek to:

• advance high-risk, high-reward research that is mission-oriented towards transformative outcomes
• catalyze multi-disciplinary collaborations across NRC research centres (RCs) and with external partners across research sectors and industries, and outside Canada (6% of funded projects were in Europe, Japan or the United States)
• provide scientific solutions to Canada’s most pressing challenges (current and emerging)

To accomplish this, the Challenge programs are:

• collaborative by design: developed in consultation with internal and external stakeholders, and strengthened through peer review, during program development and at the mid-term
• time-limited and staggered: typically 7 years in length, with mid-term reviews (MTRs) in year 4; new programs are generally launched every 3 years
• funded annually: baseline budget for G&Cs ($24M per year) and NRC-side operations ($4M per year)
• innovative: advancing projects that span the innovation continuum, from early-stage, foundational research to knowledge mobilization and pre-commercialization (TRL 1-9)

Over 5 years, the Challenge programs provided 219 grants and 44 contributions to collaborators:

• 85% academia
• 5% industry
• 2% provincial and territorial government
• 9% not-for-profit and others

Note: Total percentage of collaborators adds up to more than 100% due to rounding. Includes all G&C funded projects from April 1, 2018 through February 28, 2023.

Transfer payments at a glance

The Government of Canada’s Policy on Transfer Payments (PTP) defines a “transfer payment” as a monetary payment made by the government to an eligible, third-party recipient. Transfer payments, which include grants and contributions (G&Cs), are primarily used to enable and engage capabilities and resources that exist outside the federal government.

What is a transfer payment program?

Transfer payment programs, like CSTIP, are the mechanism by which the government leverages third-party assets and directs recipients’ activities towards the achievement of specific objectives and outcomes. Per the PTP, programs must:

• be designed, delivered, and managed with integrity, transparency and accountability
• clearly demonstrate value for money through tangible contributions to program objectives and departmental results
• be supported by appropriate administrative, oversight and control systems that are proportionate to the level of risk specific to the program
• remain recipient-focused, with an emphasis on accessibility, inclusivity and usability

What is the difference between grants and contributions?

• Grants are subject to pre-established eligibility and other entitlement criteria, but are typically not subject to audit. Under CSTIP, grants are primarily used to engage with academia and not-for-profit organizations.
• Contributions are subject to stricter performance conditions specified in a funding agreement, and are also subject to recipient audit. Under CSTIP, contributions are typically used for more complex projects with multiple collaborators, to engage with industry and to make foundational investments (i.e., equipment purchases).

As of FY 2023, the funding limit for both grants and contributions, per recipient per project, is $2 million per year. Oversight and reporting requirements for grants remain relatively lower than for contributions.

The National Program Office (NPO)

As of January 2023, NPO had a total of 36 staff:

• Annual expenditures: $4.0M per year (FY 2023)
• NPO Management: 3
• Operations and governance: 11
• Ideation program: 5
• R&D programs: 11 (including innovation investment advisor)
• Digital and data management: 4
• Program policy: 3

Roles and responsibilities

NPO is responsible for the day-to-day administration and oversight of CSTIP, including:

• program development support
• due diligence and advice on funding requests
• funding decisions related to G&Cs
• funding agreements administration and management
• program policy and operational guidance
• coordination and oversight of internal and external committees and peer review, including the MTRs
• monitoring and reporting on progress, performance and achievement of program outcomes

The innovation investment advisor

The innovation investment advisor (IIA) acts as the liaison between NPO and the collaborative R&D programs (including both Challenge and Cluster Support programs). Formerly known as Challenge officers, the IIA plays an impartial advisory role or “Challenge function,” in the strategic oversight and delivery of the programs and individual R&D projects. The Challenge function is also provided by other oversight bodies within the NRC. The IIA’s role includes:

• advising on NRC resource and funding investments
• establishing strategic co-funding partnerships
• supporting the design of programs through stakeholder engagement and the establishment of expert advisory committees
• performing due diligence and impartial assessment on program and project proposals
• funding projects in support of collaborative R&D programs, including management of programs’ G&C budgets, competitive funding calls and agreements

Challenge program governance

Figure 1 – Text version

This map represents the governance structure for typical Challenge programs.

Program management bodies:

• NRC Senior Executive Committee (SEC)
• NRC Financial Resources Program Committee (FRPC)
• VP Business and Professional Services (VP-BPS)
• National Program Office (NPO)
• Program host (VP or DG Research)
• Program director and management team

Advisory bodies:

• Program Steering Committee (PSC)
• External Peer Review Committee (EPRC)
• Program Advisory Committee (PAC)
• Mid-term review (MTR) including Peer Review Committee (PRC)

R&D collaborators:

• Participating NRC research centres (RCs)
• External collaborators

SEC and FRPC provide authority and approval to CSTIP Challenge programs.

VP-BPS and NPO provide program oversight, administration and G&C management to CSTIP Challenge programs. NPO issues G&Cs to external collaborators.

Program host, program director and management team provide program delivery and scientific direction to CSTIP Challenge programs.

PSC, PAC and MTR-PRC provide expert advice and guidance to program director and management team.

CSTIP Challenge programs produce R&D projects which involve participating RCs and external collaborators.

Note: The above represents the governance structure for a typical Challenge program. Some variation exists between programs in terms of the number and membership of advisory committees.

Challenge program lifecycle

Internal and external bodies fulfill program management, advisory, and collaborative research functions through a program’s lifecycle. 

Program management

Advisory bodies

Round 1: Challenge programs

CSTIP launched 4 Challenge programs between FY 2018-19 and FY 2019-20. All 4 were hosted by an NRC research centre (RC), and underwent MTRs in FY 2022-23.

Round 2: Challenge programs

CSTIP launched 4 Challenge programs between FY 2021-22 and FY 2022-23. Two round 2 programs were hosted at the division level, rather than by an RC.

Pandemic Response Challenge Program

Hosted by the Life Sciences division, the Pandemic Response Challenge Program (PRCP), was a special 2-year program that aimed to fast-track research and development (R&D) for pandemic-related challenges and gaps identified by Canada's health experts. The program was developed under tight timeframes with its own unique governance structure and external advisors for each project. Program leaders took a portfolio approach to risk management by selecting a mix of low to high-risk projects to mitigate overall program risk. By planning and acting quickly, the program was an example of how to structure a short-term Challenge program to respond quickly to a crisis like COVID-19.

Challenge program timeline and financials

CSTIP costs have increased over time with the launch of new Challenge programs and changes to its terms and conditions (T&Cs). Note the spikes in operations and maintenance (O&M) in FY 2020-21 due to extra dedicated funding to support the PRCP, then in FY 2021-22 in response to additional funding from Budget 2021 as part of Canada’s National Quantum Strategy (benefitting AQC and QSP).

FY 2018-19

• NPO formed
• Round 1 programs developed and 3 launched (CGT, HTSN, MCF)

FY 2019-20

• Fourth round 1 program launched (AI4D)
• T&Cs amended to expand grant eligibility, and enable PRCP

FY 2020-21

• PRCP launched in response to COVID-19 pandemic
• T&Cs amended for inclusivity

FY 2021-22

• Round 2 programs launched (AIP, ANCP, QSP)

FY 2022-23

• AQC program launched (via Budget 2021)

Comparator programs

The evaluation methodology included a comparative study of CSTIP’s Challenge program operational framework with that of similar, mission-oriented research programs. The intent of the study was to position the NRC’s overall approach in comparison to other approaches, and to identify best practices. To this end, 2 Canadian and 2 international programs were selected in collaboration with NPO for comparison:

Program model

The CSTIP collaborative R&D initiative provides a flexible program model for funding collaborative research towards addressing societal challenges. Its flexibility was by design, which enabled some programs to leverage both in-house resources and external funding opportunities where they align with their specific challenge mission. The model is effective at enhancing collaboration with external researchers, mainly in academia, through the provision of grants and contributions.

As familiarity with CSTIP increased, the model has also been effective at enhancing program-level collaboration across the NRC, with CSTIP acting as a ‘change program’ for the NRC. Amendments to CSTIP’s terms and conditions supported broader and more inclusive collaboration. Programs incorporated external guidance through a number of committees at different stages in their lifecycles. To date, round 1 Challenge programs have attracted collaborators to work with the NRC, co-author publications and generate IP, and feedback from G&C recipients and NRC researchers on collaborations has been positive.

It is not possible to conclude whether the Challenge program model provides an effective structure to advance funded research towards breakthrough outcomes, given that round 1 programs are only passed the midpoint of the 7-year lifecycles in FY 2023. However, the model provides insufficient structure for managing the advancement of project portfolios (i.e., project maturation and scale-up or termination), developing technology with industry or other partners, and defining successful, end-user adoption. More broadly, the model has suffered from a combination of resource allocation challenges at both the CSTIP and program level, unclear roles and responsibilities, mixed expectations, heavy oversight yet inconsistent structure, and possible change fatigue.

Governance structure: flexible program model

Stakeholders satisfied with centralized program office

NRC stakeholders,including senior management and program directors, were generally satisfied with the NPO-centralized governance model of Challenge programs. Challenge and Cluster Support program directors agreed that an NPO-like body is essential to support program establishment and implementation, while permitting flexibility in specific program delivery. Program directors and NPO leadership consulted regularly via a community of practice (CoP) to identify issues, share best practices, and recommend changes on program governance.

The NRC Challenge program model is unique

The comparative study found the Challenge program model—in which the NRC is both a funding agent and R&D collaborator—to be internationally unique. While some comparator programs do allow for the participation of national laboratories or other federal agencies as 3rd party R&D performers, the comparator organizations themselves do not directly collaborate with G&C recipients on funded projects.

Flexible models for program hosting within the NRC

Challenge programs may be hosted at the division level or hosted at the RC level. There were advantages associated with each approach:

• Division-hosted, VP sponsored: more aligned with the mission-oriented intent of CSTIP, tending to support greater intra-NRC collaboration and NRC-wide strategic planning
• RC-hosted, DG sponsored: more directly connected to both research staff and support capacities (e.g., business development, program management)

Ultimately, model suitability depends on multiple factors such as program scope, the nature of the challenge, and the resources and range of expertise needed. Overall, program directors and senior management agreed that this flexibility is warranted going forward, rather than selecting a single model.

Flexible program direction a best practice

Several comparator programs highlighted the importance of establishing flexible delivery models and authorities that support the program’s unique needs and objectives.

Governance structure: advice and oversight

Programs supported by advisory committees

The program advisory committees (PACs) provided expert advice and guidance to the Collaborative R&D programs. CSTIP’s flexible model allowed external expert advisory bodies to meet the differing needs of the Challenge programs. For example, the RC-hosted CGT employed its Research Centre Advisory Board (RCAB) to play this role, while the division-hosted PRCP deferred to interdepartmental committees on an ad-hoc basis for scientific advice and direction. HTSN however established its PAC relatively late in the program’s lifecycles, limiting its influence in program direction and project selection.

Role for new steering committees unclear

A new layer of internal governance, the program steering committee (PSC), was added to the program model in FY 2023. It is intended to liaise between programs and the NRC’s executive committees, and to guide mid-term changes. All future programs must have a PSC, but only 2 have one as of March 2023 (AQC, which launched in FY 2023, and HTSN, which piloted a PSC earlier). Existing programs may convene PSCs following their MTRs to support implementation of their subsequent management response and action plans (MRAPs), but this remains optional.

It is not yet possible to comment on PSCs’ effectiveness in supporting program development and implementation. Moreover, it was unclear how PSCs will complement existing committees.

Program oversight excessive

Challenge programs are answerable to various internal and external committees. Committee implementation was inconsistent across programs. Together, this resulted in reporting burden and inconsistent expectations for what programs report, and how they respond, to oversight bodies.

Program directors considered oversight to be excessive. At different points in their lifecycle, programs may be reviewed by or required to report to a combination of internal steering committees and advisory committees, unique external peer review committees for program design (EPRC), project selection committees, MTRs, and RCABs. Implementation has varied between programs, notably with some programs now adding PSCs and different approaches to PACs.

It was difficult to assess how oversight may be streamlined or consolidated as it is not consistently implemented across programs. It may be possible to standardize reporting so that programs present the same progress reports to committees, rather than unique ones for each.

Roles and responsibilities: National Program Office

Central office necessary but expectations mixed

The general role of NPO as a central administrative and support function for CSTIP programs was clear, and well-appreciated by program stakeholders. However, the extent to which NPO contributed to the scientific direction of Challenge programs, or advised on funding decisions and partnerships, varied across programs.

There was interest from some internal stakeholders for NPO to act as a facilitator between research centres, branches, and NRC Industrial Research Assistance Program (NRC IRAP) to increase awareness of Challenge programs and enhance intra-NRC collaboration. NPO did this to some extent, such as for the PRCP, but had limited capacity to fulfill this role.

“The roles and responsibilities are getting clearer. The lack of clarity at the beginning was necessary since there is a need for some flexibility within different Challenge programs.”

“The role of NPO needs to be better defined, whether purely administrative or if there is a Challenge function. I feel strongly that this is the weakness of the system right now.”

Innovation investment advisor (IIA) roles differ across Challenge programs

As of FY 2023, with regard to the role of the IIA, there remained considerable confusion and difference in experience between programs. There was a lack of consensus among program directors and IIAs on the nature of the IIA role. The core issue is to what extent IIAs were meant to provide a “Challenge function”, which is confounded by a mix of interpretations as to what this function entails. In practice, this function was performed by program directors and oversight committees, in addition to IIAs (see more on the Challenge Function under Processes and controls).

In addition to the necessary administrative due diligence, clarity is needed on the involvement of IIAs in providing advice on program and project direction at both the development and delivery level.

IIA are without comparator

The comparative study found the IIA and its role as the liaison between NPO and R&D programs to be unique to CSTIP, as it is necessitated by CSTIP’s own unique collaborative model.

Roles and responsibilities: committees and corporate services

Influence of advisory committees varies

Program Advisory Committees (PACs) were established as a source of impartial expert advice for each Challenge program. This general role was well communicated and understood by CSTIP stakeholders. PACs include external and internal members, including staff from the NRC IRAP. However, committees were inconsistently implemented across the Challenge programs and at different points within program lifecycles.

For most programs, PACs played an important role in challenging and strengthening their overall strategic direction. In some cases, PACs were involved in the selection of individual projects while others fulfilled a less direct, advisory role. Per case studies, PACs tended to focus either on larger grants or ‘master projects.’ NRC IRAP members brought industry perspectives to programs and connected them with businesses (e.g., they facilitated technology demonstration and testing for AIP stakeholders).

Impartial expert advice as a standard

Like CSTIP, the comparator programs rely on independent external experts and committees to evaluate project proposals for technical merit, provide scientific guidance and strategic direction to program management, as well as assess project performance and program impacts.

Unclear role for the NRC’s business services

Internal stakeholders who provide business development, contracting and other services to NPO and Challenge programs were at times unsure of their roles and responsibilities for CSTIP-funded projects. When developing G&C agreements under CSTIP, business development staff were not always involved, or involved late in the process, creating ambiguity about when and by whom NRC business decisions need to be made. These issues were further complicated by differences between the division and RC-hosted models, the latter of which have their own in-house business development resources.

Bilateral G&C agreements, which constituted the vast majority under CSTIP, were processed internally by IIAs by way of a standard template, as opposed to being routed through NRC contracting services as a typical collaborative research agreement would be. While NPO did seek input from the NRC’s contracting team for more complex agreements (i.e., multi-party agreements, international collaborations, or those requiring amendments), the existence of separate processes with varying degrees of involvement from corporate staff compounded inefficiencies.

There is an opportunity to develop a specific process, with clear roles and responsibilities for the NRC’s business services, that would help ensure appropriate negotiations and checks occur.

Program compliance

Evolving, flexible program-maintained compliance

When CSTIP first launched in FY 2019, there was no similar program in Canada (i.e., where a federal agency both works on collaborative projects and provides G&Cs to its external collaborators). Treasury Board Secretariat (TBS) allowed the NRC flexibility in its interpretation of the Policy on Transfer Payments (PTP) because of the unique collaborative nature of CSTIP.

In the spirit of continuous improvement, NPO and TBS worked together on T&Cs changes, ensuring oversight and policy compliance throughout the program’s evolution. When CSTIP began, the T&Cs were relatively restrictive, particularly for grant eligibility, and demanded a high level of scrutiny for all payments. This was warranted as CSTIP was a new funding program for the NRC. After 2 years, NPO demonstrated its competence in this area and was able to change the T&Cs to be more flexible, expand grant eligibility, and moderate scrutiny for payments.

Amendments to the T&Cs were also made specifically in response to COVID-19 in order to implement the PRCP, and to increase grant opportunities for Indigenous recipients.

Changes to CSTIP terms and conditions

Program changes and growth pose risk

As CSTIP evolves, NPO needs to continue working closely with TBS. To date, CSTIP has complied with federal legislation thanks to sufficient controls and processes for managing G&Cs and financial administration.

There is a risk that NPO will be insufficiently resourced to maintain compliance as the program changes and grows. NPO has been playing “catch up” with Challenge programs over the past 5 years. Their capacity to do so will be stretched thinner as the total number of Challenge programs and CSTIP eligibility criteria continue to grow without additional efficiencies or operations funding.

Model viability and scalability: NPO resources

NPO responsibilities may outweigh resources

Scale-up of CSTIP would likely encounter significant problems with NPO’s existing resource levels and responsibilities. The relative workload for NPO increased over the evaluation period, above and beyond initial expectations for the office. Notably, the NRC initially planned to launch 3 new Challenge programs every 3 years or “round”, but increased that to 4 per round, not counting the PRCP.

Currently, NPO has the human resources necessary to administer day-to-day G&C operations. As of FY 2022, NPO was responsible for roughly 6 times the number of projects as it was in FY 2019 with only twice the staff. NPO struggled to formalize its systems, processes and tools to support the implementation and monitoring of CSTIP programs. Should processes be improved, NPO may achieve the efficiencies needed to manage a larger workload.

New funding streams were approved in FY 2024, adding a strategic response stream with shorter program lifecycles and a new Challenge prize stream. The assignment of new responsibilities to NPO (e.g., serving a potential liaison role between research centres) would exacerbate resource issues should program growth not be paired with additional operations funding or efficiencies.

Figure 2 – Text version

Stacked column chart showing the number of CSTIP’s active projects in Challenge, Ideation and Cluster Support programs compared to the total number of NPO staff from fiscal year 2019 to fiscal year 2022. The number of CSTIP active projects has increased from 50 to 307. The number of total NPO staff has increased from 17 to 38.

Fiscal year	HR (total number of staff)	Active Challenge program projects	Active Ideation program projects	Active Cluster Support program projects	Total active CSTIP projects
FY 19	17	5	44	1	50
FY 20	28	53	44	15	112
FY 21	36	102	80	64	246
FY 22	38	158	30	119	307

Source: NRC Human Resources and CSTIP project database.

CSTIP manages more projects per resource

The comparative study estimated that the administrative cost ratio for NPO (10%) was within the range of comparator programs (8 to 10%). This ratio calculates total program administration costs relative to total G&Cs per year. However, NPO manages a greater number of total projects while the comparators tend to fund larger but fewer total projects. In addition, CSTIP project agreements involve both internal and external collaborators, which makes the administration complex.

Model viability and scalability: research centre participation

Limited funding to host programs, collaborate

Challenge program hosts are provided with limited operations and maintenance (O&M) funding. After deducting program direction and administration costs (i.e., program director and support role salaries), host RCs may have relatively little funding to resource NRC-side collaborators. This issue has been echoed by MTRs, that found round 1 programs lacked the internal resources necessary to achieve their ultimate outcomes.

The extent of this issue differs by program. Some hosts have accessed additional O&M monies through alternative sources, such as through revenue generation, other government department funding, or TB submissions for related national strategies. Overall, programs fell into 3 camps:

• one third of programs have dedicated funding for administration (e.g., via alternative sources)
• one third are covered by their host RC
• one third are using CSTIP O&M funding (up to 60%)

Hosts aligning RC priorities with Challenge programs

Program hosts acknowledged it can be difficult to balance Challenge program participation with other internal priorities (e.g., revenue generation). Where possible, hosts have aligned their Challenge programs with other priorities and projects, including funding sources, and thus buttressed their CSTIP commitments with additional resources.

Just over half of programs (5 of 9) sought out or accessed third-party funding or launched revenue projects (versus G&C projects). For examples, AIP, CGT and ANCP implemented projects where the NRC and external collaborators collectively seek third-party funding (e.g., AIP and CGT are leveraging funding from the Canadian Institutes for Health Research, and ANCP from the Canada-Inuit Nunagat-United Kingdom research programme). Other programs, such as MCF, included research service projects within their Challenge program portfolio.

Non-host RCs need to balance Challenge programs with other priorities

Similarly, non-host RCs must balance Challenge program participation with other priorities, and align them where possible. RCs felt they were required to cut from their regular business and operational budgets in order to support the Challenge programs, which they viewed as butting up against revenue targets and other strategic deliverables. At the same time, it has not been clearly communicated to RCs that Challenge programs are an organizational priority, and thus RCs must budget and plan accordingly.

Model effectiveness: advancing ‘breakthrough’ research

Incremental change, not yet ‘breakthrough’

It is unclear how CSTIP prioritizes the most promising research and defines success for each Challenge program. The model did not provide Challenge programs a structure to focus on achieving tangible solutions for Canadians. To date, programs funded a range of projects, each moving incrementally up the technology readiness level (TRL) scale but did not appear to be systematically managing their project portfolio towards planned outcomes. Internal interviewees were divided on whether the model supports breakthrough research, but tended to agree that the low TRL focus would preclude breakthroughs for Canadians within 7-year program lifecycles.

To date, the program model did not appear to be scaling up promising research with near-term commercialization potential and terminating activity without (i.e., the DARPA model). G&C size was relatively consistent over the first 5 years of the round 1 programs, suggesting programs are not focussing resources on a smaller selection of projects post-midpoint. This was affirmed by MTRs which found round 1 program goals were too ambitious given their timeframe, and suggested programs focus on scaling up technology and facilitating industry pull.

Figure 3 – Text version

Barbell chart showing the average TRL of projects at launch and the average TRL of completed projects for PRCP, AI4D, CGT, MCF and HTSN programs.

Challenge program	Average TRL launch	Average TRL completed
PRCP	3.7	5
AI4D	1.75	4.75
CGT	2.5	3.75
MCF	2.14	3.57
HTSN	2.27	3.5

Source: Self-reported TRL progression from round 1 project close-out reports (PCRs) (n=44).

Models for transformative research

International comparators provide an alternate model to achieve breakthrough research. ARPA-E and EIC Pathfinder focus only on early-stage (TRL 1-5), high-risk, high-reward projects, rather than the full TRL continuum, as CSTIP does. Upon completion, the most promising projects are then continued by in-house programs that focus on scale-up (TRL 6-10), in order to bring new technologies to market. The 3-year Pathfinder program, for instance, is linked to a 1-year transition program and a 3-year commercialization program—the same 7-year length as most Challenge programs.

Model effectiveness: enhancing external collaboration

Engaging both past and new collaborators

Challenge programs enhanced collaboration with external researchers and feedback from G&C recipients and NRC researchers on collaborations has been positive. Challenge programs partnered with players in their respective fields, leveraging unique and complimentary expertise and facilities towards their goals.

The availability of high-value grants was particularly effective at attracting academic collaborators; most funded projects (85%) were with academic institutions. Within 3 years, the NRC and G&C recipients co-authored over 240 publications, and submitted over 40 patent applications.

Programs have attracted new, external collaborators to work with the NRC. According to project close-out reports (PCRs) for round 1 programs, slightly more than half (55%) of principal investigators (PIs) had not worked previously with the NRC. However, PCRs were only available for two fifths of completed projects, and thus it is not possible to conclude definitively that most PIs were new to working with the NRC.

Figure 4 – Text version

Column chart showing round 1 program outputs in co-authored publications, invention disclosures and patent applications.

Fiscal year	Co-authored publications	Invention disclosures	Patent applications
FY 2020	53	10	6
FY 2021	108	23	23
FY 2022	81	8	13

Source: CSTIP program results database (NPO).

Programs tasked to engage industry more

As of FY 2023, only 5% of external collaborators were businesses. Though industry engagement may be expected to be low in the first half of programs’ lifecycles (i.e., low TRL), EIC Pathfinder had 22% industry participation for programs at a similar stage.

Generally, NRC Challenge programs planned to transfer technology to industry, rather than co-develop it. According to PCRs, 47% of G&C recipients felt commercialization within 5-10 years was very likely.

Program oversight bodies have recommended that programs use the latter half of lifecycles to implement models to engage industry. PACs and mid-term reviewers stressed the essential role industry should play in bringing CSTIP-supported technologies to market in order to address programs’ respective challenges. Obstacles to this shift to industry included the COVID-19 pandemic, and the limited funding available to attract larger companies.

Model effectiveness: enhancing internal collaboration

Research centres collaborating on challenges

The model has demonstrated effectiveness at enhancing program-level collaboration across the NRC, with CSTIP acting as a ‘change program’ for the NRC. It has directed RCs to lead projects that contribute towards common challenges. This has necessitated a significant culture shift for RCs and remains an ongoing effort.

• Round 1 programs demonstrated limited intra-NRC collaboration. On average, each program involved 3.5 RCs, but 85% of projects were led by their host RC. This may be attributed to limited familiarity with the new model (G&Cs for externals without funding for non-hosts) and a lack of early engagement with RC leadership during program design, which in turn hindered the necessary culture shift.
• Round 2 programs demonstrated significant intra-NRC collaboration and are less dominated by a single RC. On average, each program involved 5.3 RCs, and only half (49%) of projects were led by their host or single RC. This may be attributed to multiple factors, including increased familiarity with the model, the adoption of the division-hosted model by some programs, and the ability of some hosts to secure additional O&M funding (thus reducing costs to participate).

Figure 5 – Text version

100% stacked column chart showing percent of projects led by host or primary RC compared to percent of projects led by other RCs per Challenge program. More RCs leading program projects in round 2 compared to round 1.

Challenge program	Percent of projects led by host or primary RC	Percent of projects led by other RCs
AI4D	81	19
CGT	90	10
HTSN	73	27
MCF	97	3
PRCP	58	42
AIP	40	60
ANCP	50	50
QSP	50	50

Source: NPO project database (pre-dates launch of AQC).

Continuous improvement

Recognized room to improve

NPO and senior leadership recognized that CSTIP remains, in part, a work-in-progress. There is a need for more streamlined and standardized processes as well as for integrated systems at the NPO level. CSTIP was launched as NPO and its processes and tools were still in development. Some issues identified by NPO as part of a policy and process review in FY 2021 remain unaddressed or mitigations remain in development. In FY 2024, NPO established a CSTIP toolkit roadmap to implement outstanding actions within the year.

Ultimately, there is a need to allow flexibility and autonomy at the program level, balanced with common structures and processes to maintain appropriate oversight, resourcing and reporting. Programs have been uniquely governed and funded, resulting in mixed experiences among program leaders.

Leaders sharing what works

Continuous improvement is supported by the program directors’ community of practice (CoP), a platform to identify common problems, share best practices, and respond to NPO and senior executive with a single voice.

As of fall 2022, the CoP, at the President’s direction, has been meeting to discuss challenges in program execution, including program administration and funding, internal NRC collaboration, and program oversight. The discussions have carried over into division and research centres’ strategic planning in spring 2023. No formal plans or recommendations have been made by the CoP and on some matters, namely oversight, they are waiting on the results of this evaluation.

PRCP a rapid response ‘test case’

PRCP went through the complete program life-cycle from inception to delivery and completion in 2 years. According to a lessons-learned report prepared by PRCP leaders and teams, programs developed in tight timeframes and under extenuating circumstances benefit from:

• a multidisciplinary program launch tiger team, and the integration of NPO and NRC governance (i.e., including IIA in program management committee)
• taking a portfolio approach to program and risk management (have a portfolio of low to high-risk projects to mitigate program-level risk)

In FY 2023, NPO applied some of these lessons as part of changes to CSTIP’s T&Cs, formally adding a rapid response model as an alternative to the standard 7-year program.

Processes and controls

CSTIP processes and controls have been adequate to support program implementation, but are not necessarily as efficient or as effective as they could be. Overall, CSTIP was launched before processes and controls could be fully developed by NPO. Therefore, NPO has been playing ‘catch up’ over the past 5 years, developing the structure necessary to support CSTIP collaborative R&D’s goals, while also responding to the roughly biannual changes in CSTIP’s terms and conditions and to the growth in the total number of Challenge programs.

Process consistency

Program-level inconsistency may inhibit model

Programs have implemented their own processes, with differing amounts of documentation and data collected. As flexibility is a feature of the model, some inconsistency may be expected. However, processes and tools could be standardized to support program and information management. As of FY 2023, a field guide for program directors remained in development. This would provide much-needed standardized guidance for both current and future program staff, while still permitting some flexibility.

The design and development of individual Challenge programs follows a series of common steps. Differences, however, were observed in terms of sub-theme or “pillar” development among case study programs. The earlier programs studied, HTSN and PRCP, built their research pillars around the NRC’s in-house capabilities. This ran the risk of perpetuating existing approaches to research activities, though for PRCP this was justified by the need to act quickly. AIP, a round 2 program, built its pillars thematically around the “challenge” itself, which better aligned with CSTIP’s intent of encouraging collaboration within and outside of the NRC.

NPO-level processes need framework, field guide

NPO-level processes are implemented in a generally consistent manner (e.g., G&C process, payments) and were found to be aligned with the Policy on Transfer Payments (PTP). NPO mapped some of its processes in recent years, notably its financial controls and claims process, to ensure program compliance with the PTP and improve efficiency. However, NPO is generally using forms and templates to drive processes, rather than an overarching framework. Moreover, as already stated a field guide specifically for IIAs would better define that positions’ roles and responsibilities.

An overarching CSTIP framework may balance structure with flexibility. For example, a framework could provide guiding principles, and map and define the intent of core processes, while permitting flexible implementation so long as programs adhered to the principles. This would reduce inconsistencies in program implementation, management and oversight, but still allow programs to implement processes in a risk-appropriate manner suited to their unique needs. A program framework would also provide an anchor for a robust IIA field guide.

Internal controls

Opportunity to calibrate controls

Internal controls for project due diligence, approval, and monitoring are in place. NPO interviewees reported that due diligence and internal controls are generally appropriate, but sometimes excessive as the same level of controls are required regardless of the amount of funding. NPO leadership contended that heavy controls were warranted at CSTIP’s outset as it was a new program administered by a new office.

Having proven its competence, both internally and with TBS, there are opportunities for NPO to calibrate controls based on project attributes (e.g., size, grant vs. contribution, risk rating). Internal interviewees noted the relationship between due diligence and risk, where processes could improve if risk-based tools were standardized. At first, CSTIP did not leverage NRC IRAP’s existing processes, long used for contribution agreements, to the extent possible. To address this, NPO recently hired an NRC IRAP employee with PTP expertise to guide revisions of their funding instruments, processes, and risk-model framework.

“It’s the same level of rigour for any amount. In some cases, it’s the right amount but sometimes it is overkill.”

CSTIP choosing grants over contributions

To date, NPO has favoured the use of grants over contributions. Interviewees indicated that programs are using grants for academic partners, mainly for highly qualified personnel. As for contributions, NPO staff reported that contribution agreements are used for more complex projects. Contribution agreements take more effort in terms of documentation and milestone-based claim requirements, for both agreement development and implementation. T&Cs changes made in FY 2021 enabled CSTIP to disburse funding almost entirely by grants (96% of all agreements in FY 2023, up from 78% in FY 2020).

Building on the NRC’s past G&C experience

CSTIP avoided some of the issues experienced by the NRC’s former Canada Accelerator and Incubator Program (CAIP), as identified by that program’s evaluation in 2019. CAIP supported a broad range of activities and costs, beyond simply overhead and salaries, which in practice demanded regular line-by-line reviews of invoices. For CSTIP, grant eligibility is determined up front thus requiring relatively little follow-up, and contributions are generally directed towards salaries, thus avoiding demanding reviews.

Conflict of interest

NPO has identified and managed conflicts of interest (COI)

NPO has made sustained efforts to improve the identification and management of COI by maturing their processes, educating CSTIP stakeholders (primarily NRC staff), and including COI forms at specific program and project milestones. This will be especially important if CSTIP continues to grow, both adding new programs and expanding eligibility criteria, which will increase risk exposure.

NPO worked cooperatively with the NRC’s Value and Ethics team to strengthen COI management. In FY 2021, NPO implemented the “Conflict of interest attestation for NRC principal investigators (PIs)” in an effort to identify and manage issues of COI at the project level. There is an opportunity for CSTIP to further strengthen its process and ensure regular review of the NRC PI attestation, similar to what is required for external participants. As there are multiple potential sources for real or perceived COIs, it may be prudent for NRC staff to re-examine their situation whenever they sign onto a new collaboration agreement.

Potential sources of real or perceived COI:

• existing relationships between the NRC and a potential recipient
• NRC PIs who hold a position at recipient organizations (academia especially)
• committee members who are actual or potential recipients
• evolving relationships with recipients (e.g., student hiring, spin-offs companies)

Committee COI risks unique, but well-managed

COI for external EPRC and PAC members is well-managed. Upon selection, members are required to sign confidentiality and COI declaration forms, and agree to review and update their declarations regularly or as soon as their interests change.

The requirement for regular review of COI is important for all stakeholders, given that Challenge programs are 7 years in length and individual research projects can last 3 to 4 years. Potential COI with external committee members is an issue faced by many programs, including CSTIP comparator programs. It is particularly true for those within a relatively small research ecosystem. The number of unique committees required by the CSTIP exacerbates this effect, as it rapidly depletes the pool of available candidates.

Process effectiveness: opportunity to integrate processes

Processes generally effective, improved over time

CSTIP processes are generally effective in the implementation of Challenge programs, such that:

• grants and contributions are being issued
• transfer payments are managed appropriately

G&C recipients generally gave positive feedback on CSTIP processes:

• program management were engaged and accommodating
• IIAs were easy to communicate with and provided support
• satisfied with level of collaboration with NRC researchers

Internal stakeholders noted improvements made by NPO, including the streamlining and simplification of processes (e.g., agreement amendments). The NRC achieved this in part by bringing leadership with significant G&C experience to NPO.

“We have established processes. We’ve been building processes as we go. We’re constantly improving in this first 5 years. We started with nothing and no people.”

Process integration valuable but a major investment

NPO internal stakeholders identified the need for an integrated client relationship management (CRM) system. Business tools and processes are not currently integrated, as different program and project phases use different software platforms (i.e., project management by a mix of email, spreadsheets, other software).

A CRM system or other integrated system could support program staff, G&Cs recipients, and other program stakeholders in effectively implementing processes. Integration would demand a significant level of effort due to the current disparate state of systems. Successful integration may also require an overall framework that comprehensively maps CSTIP processes, which is currently lacking. Thus, NPO’s level of readiness to implement a CRM, as well as the cost that would be associated, could be prohibitive.

Canadian, international programs integrate systems

ICI, ARPA-E and EIC use web portals to integrate processes and facilitate information exchange between program staff, peer reviewers and external funding recipients. For example, they can be used to support document and data processing, project application, development and selection, project reporting and monitoring, and communications.

Process effectiveness: opportunities to strengthen model

Cross-RC collaboration

There are no formal processes or structures to support intra-RC collaboration. Program directors suggested that NPO act as a facilitator, which could be possible through direct engagement or maintenance of shared tools. Formal support of cross-collaboration may increase awareness of Challenge programs and research centre capabilities, thus reducing barriers to intra-NRC collaboration. Note that this would add responsibilities for NPO. 

Intellectual property and technology transfer

The structure necessary to commercialize solutions stemming from Challenge program projects, namely processes for intellectual property (IP) management and technology transfer, was underdeveloped.

IP processes were sufficient for lower TRL projects where near-term commercialization was unlikely. However, they were not well-defined for higher TRL projects or more complex situations. Ownership of co-developed IP was negotiated on an ad hoc basis via IP agreements separate from project collaboration agreements.

Technology transfer mechanisms were not well-developed, and clear plans were not in place to advance research outcomes beyond program lifecycles.

Project risk assessments

Project risk assessments appear to have been conducted subjectively by IIAs (i.e., without a standardized method), and implementation varies. A standardized approach to project risk assessments with objective ranking criteria would strengthen due diligence by NPO.

Additionally, these assessments only consider funding risks (i.e., likelihood recipient will deliver on project), and are conducted post project-selection. It is unclear how project proposals are assessed for technical risk in order to identify high-risk, high-reward research in the spirit of the CSTIP model.

Challenge identification

Identification process responsive, albeit unstructured

The NRC has identified challenges in response to federal priorities as they have emerged. This approach has enabled CSTIP to:

• align existing division and RC capabilities with other federal agencies and initiatives
• leverage funding sources for additional G&C and O&M in addition to the original CSTIP funding in Budget 2019

Following Budget 2021 for instance, the NRC identified the AQC challenge to align with the National Quantum Strategy, and secured additional funding for its pre-existing quantum program, QSP. This approach is continuing for programs in development as of FY 2023 (e.g., Critical Minerals Strategy, Greener Homes Initiative).

This approach, however, has been unstructured, lacking standard processes to identify challenges or determine how many programs may be launched simultaneously. Some DGs did not know how they themselves could identify a challenge and recommend a new program. Providing some structure to the identification process may enable RCs to find alignment with emerging federal priorities and recommend new Challenge programs.

Standard processes to develop programs

Processes are more formal and consistent after challenge identification and for program conceptualization. Before any new Challenge program can be launched, it must undertake standard activities such as external peer assessment and presentation to FRPC. Given that the FRPC was put in place recently, it is to soon to assess if this structure will streamline the development and approval of program proposals. Similarly, it is not clear which role the PSCs to be created will play in this process.

External stakeholders are then engaged to assist in program development through various activities, including one-on-one meetings and interviews, presentations, workshops, and surveys. This helps ensure that programs are not designed in isolation.

Most projects aligned with Challenge program aims

Approaches to program pillar development have been inconsistent, which may resultin different approaches to project portfolio management and engagement strategies. However, MTRs of the programs found that pillars and projects are generally well aligned with their respective external challenges.

Challenge function

Challenge function not thoroughly enacted

Foundational CSTIP program documents connect the Challenge function to the DARPA model, where it is described as oversight on project milestones and deliverables to achieve the program's ultimate objectives. This is distinct from the societal “challenge” a given program aims to address. Rather, it is an internal function to “challenge” the programs themselves. Within CSTIP, the function tends to be dispersed between IIAs, EPRCs, PACs, and MTRs (which include peer review committees). However, committees, namely the PACs, do not seem to play a role beyond project intake and selection.

With the function diffused and inconsistently understood, it is not evident that research activity is being challenged. Case studies found most project proposals were embedded within Challenge program proposals, and thus were collectively reviewed and approved along with their respective program. Information was not available to determine what proportion of projects, if any, were terminated because they no longer aligned with the program or were not performing as expected.

Innovation investment advisors (IIA), directors differ on ‘challenge’ roles

IIAs reported a need to distinguish between a 'true' Challenge function (i.e., challenging project selection and scientific portfolio management) and the administrative function of handling G&Cs (i.e., ensuring program alignment, and adherence to T&Cs). Based on their experience with industry and research program management, some IIAs believe they should be providing both. This has caused frustration for some IIAs who feel their capabilities are underutilized. In comparison, IIAs have similar skillsets to NRC IRAP’s industrial technology advisors (ITAs) but have less influence over G&C direction and research activity.

Program directors consider the IIAs' Challenge function to be mainly around G&C funding, aligning with CSTIP T&Cs, and monitoring project milestones and deliverables. Over time, this has increasingly been the case. As CSTIP expanded grant eligibility, more grants were awarded relative to contributions. This resulted in less opportunity for IIAs to challenge the ongoing relevance and direction of projects as there are fewer opportunities to do so with a grant than with a contribution agreement.

Challenge function for portfolio management

EIC Pathfinder uses a 2-stage project selection process to challenge research. Proposals are first peer reviewed for scientific excellence. Then, programs holistically review proposals that cleared stage 1 to create a smaller, complementary project portfolio.

Process efficiency

Some evidence of increased efficiency

Both internal and external stakeholders found processes had become more efficient over CSTIP’s first 5 years, and highlighted specific improvements:

• financial monitoring (similar to NRC IRAP) with integrated risk ratings was established, helping to streamline payments
• greater signing authority was granted to IIAs and a risk assessment matrix was introduced, simplifying agreement approvals

G&C recipients tended to be content with CSTIP processes overall, but contribution recipients were more likely to have issues with processes. This aligns with the views of internal staff who found contributions more demanding to administer than grants. Internal and external stakeholders also noted that there remain areas for improvement, including:

• complexity of processes for contribution agreements and claims payments
• volume and clarity of reporting requirements
• complexity of collaboration agreements and associated processes for development

As of spring 2023, NPO was mapping and reviewing its claims payment process with an aim to speed up delivery.

Figure 6 – Text version

Bar chart showing the percentage of grant and contribution recipients who reported positive satisfaction with the CSTIP process. The percentage of contribution recipients is lower than the percentage of grant recipients.

Project close-out reports (PCRs) question	Percent of grant recipients (n=30) who provided positive response	Percent of contribution recipients (n=14) who provided positive response
Funds received in a timely manner	93	57
Processes understood and efficient	87	43
Reporting not too heavy	80	50

Source: Round 1 project close-out reports (PCRs) (n=44); percent agreeing somewhat or strongly.

Significant data gaps inhibit efficiency tracking

It was not possible to assess process efficiency throughout program and project lifecycles due to 3 important limitations:

1. NPO lacks a detailed framework and process map to define and track its processes
2. insufficient capture of administrative information to permit analysis and quantify process efficiency
3. there are currently no service standards or efficiency baselines against which to compare

NPO reported it may be difficult to establish service standards due to the variety of programs and funding streams under CSTIP, which may be further exacerbated by T&C changes in FY 2024, and the lack of an integrated system for information management.

Gender-based Analysis Plus (GBA Plus)

Gender-based Analysis Plus and equity, diversity and inclusion (EDI) considered in program and project design

CSTIP demonstrated consideration of GBA Plus and EDI in its overall design, at the Challenge program level, and at the project level.

• CSTIP level: TB submissions made some GBA Plus commitments (e.g., requiring committees to assess program and project proposals with an intersectional lens), but most commitments were around EDI (e.g., hiring targets for program hosts and collaborators). T&Cs amendments in FY 2021 did promote the inclusion of Indigenous persons by expanding eligible activities and types of recipients.
• Program level: GBA Plus was considered in development of Challenge programs, though to varying extents. 2 of the 3 case study programs explored how they may uniquely impact specific populations. The AIP program, notably, convened an “experts by experience” panel for adults 65 and older or their caregivers to advise on program direction and participate in research. EDI was considered less, with only 1 program proposing how it would target specific equity groups for advertising and employment.
• Project level: There was some evidence of GBA Plus principles being integrated at the project level. For example, a project using sensors to measure oxygen in the brain tests people of diverse skin pigmentations, as this can affect the sensor's accuracy. For the Arctic and Northern Challenge Program, all projects require collaborators from Arctic and northern populations.

Limits on data collection

Information was not collected on NRC workforce diversity at the CSTIP level, namely whether or not researchers working on the program reflected the NRC’s EDI targets. Due to the complexity of attributing new RC hires to CSTIP funding specifically, NPO is planning to develop new metrics for GBA Plus achievements at the CSTIP level (rather than EDI).

For each Challenge program, NPO mainly tracks progress towards GBA Plus-related indicators via 2 project-level instruments: annual performance reports and PCRs. Within these instruments, there is a strong overlap of GBA Plus and EDI, with the focus on the latter. Round 1 program PCRs show that most projects (82%) included EDI considerations for their team, but less than half (39%) considered GBA Plus in project design.

Program directors felt that they have little ability to require GBA Plus or EDI elements from recipients, and that there are challenges around assessing progress because of limits on collecting data.

Information management

Program management has sufficient information to administer CSTIP. NPO and programs are tracking program implementation and project activity, generally, but there are significant gaps for tracking efficiency and progress towards outcomes. Some key performance indicators tracked at the program or project level are not integrated, which limits overall performance monitoring. MTRs provide necessary checks on program performance, but round 1 MTRs were limited in their ability to inform adjustments and assess success to date. Importantly, there is no plan for the assessment of individual programs longer-term outcomes (i.e., beyond the midpoint). As such, program management and NRC executives lack information on the extent to which programs are addressing their respective challenges.

Data collection planning: performance monitoring

Monitoring divided between NPO, programs

The CSTIP model divides performance monitoring between the funder (NPO) and performers (program hosts). Generally, NPO focusses at the CSTIP level (e.g., G&C accountability, overall achievements), while performers focus on progress towards challenge-specific outcomes. NPO needs to aggregate program-level key performance indicators (KPIs) to assess CSTIP overall, which can be challenging. In practice, there are gaps and inconsistencies at both the CSTIP and program level.

Resourcing information management

According to NPO and Challenge program staff, incremental resources are needed for performance monitoring. With demands to implement new programs and limited O&M funding, NPO and programs believe they lack the resources needed to fully monitor performance as planned.

Systems not fully implemented at NPO

NPO has yet to fully implement its monitoring system as initially envisioned. In an effort to reduce reporting burden on programs, NPO has not requested that all KPIs be tracked. It is collecting and reporting descriptive statistics and anecdotal outcomes. As of FY 2023, not all data collected was being aggregated and reported.

NPO has revised tools to systematically collect and aggregate data. For example, they are using project close-out reports (PCRs) to capture KPIs directly from G&C recipients (e.g., satisfaction, TRL progression, student involvement) but only began aggregating data in FY 2024. Moreover, though required to do so, only half of G&C recipients (51%) submitted PCRs within a year of completing their project.

Data collection varies between programs

Programs use modified, or in some cases unique, templates to capture KPIs that relate to their subject matter. However, this flexibility makes it difficult for NPO and programs to aggregate data. This is further complicated by the divided reporting responsibilities of the CSTIP model.

A working group has been struck to bring NPO and programs together to address reporting responsibilities. As of FY 2023, programs were not routinely sharing outcomes data with NPO, instead providing some data upon request. Program staff are working with IIAs to determine how their own project reviews may be utilized at the CSTIP level to avoid duplication of efforts or burdening internal and external PIs with information requests.

Alignment of performance tools

Need for service standards

Performance monitoring is inadequate to meaningfully assess CSTIP processes and support continuous improvement. At the NPO level, there is a lack of service standards or baselines to compare performance against, and a mismatch of indicators for performance attributes.

For instance, while NPO intends to measure efficiency with an overall administrative-cost ratio, it would be more useful to assess NPO service against set service standards—such as how quickly it responds to grant proposals or fulfills contribution claims. As of March 2023, NPO had begun mapping out and measuring its claims process so that it could in fact assess its efficiency in a meaningful and constructive way.

Lessons learned in information management

Comparator programs highlighted the importance of integrated data systems for data collection, performance monitoring and reporting, with an emphasis on accessibility of information to both internal and external stakeholders.

Need for clear outcomes and consistent monitoring

For outcomes metrics, planned indicators are sufficient for corporate reporting needs (i.e., the NRC’s departmental results and reporting to TBS) and related to specific goals (e.g., business innovation) but insufficient for assessing outcomes. In some cases, NPO has deferred the measurement and assessment of outcomes to future CSTIP evaluations, which are not likely to assess outcomes of individual Challenge programs. In other cases, NPO relies on program hosts to monitor and track progress towards outcomes as part of regular program management.

However, even in cases where relevant outcome data is being collected, it is not being centrally aggregated. This undermines the ability of NPO to monitor program performance. The lack of aggregation and monitoring exposes the program to the risk that monitoring regimes differ across Challenge programs thus undermining access to performance information.

“Trying to measure impacts is hard, and is going to require resources in order to have meaningful measures. Other organizations have whole offices just do to this.”

Assessment: mid-term reviews and final assessments

Reviews are opportunities to adjust course

MTRs of Challenge program performance and progress towards outcomes is essential to ensure that programs address their respective challenges. However, their efficacy may be limited by late timing within program lifecycles, insufficient performance data, and unclear plans for taking action following the MTR.

• Round 1 program MTRs were facilitated by NPO and conducted by an external consultant. These MTRs included external peer review committees, and were conducted in year 4 of their 7-year lifecycles, 1 year later than planned.
• The MTRs found programs to be relevant (e.g., program strategies and projects aligned with challenges) but could not fully assess progress towards outcomes, due to limited performance data and information from external collaborators. For example, one MTR recommended an entirely new set of KPIs to assess project suitability for continuation, scale up or termination. Another MTR could not assess progress towards outcomes whatsoever based on the data they were provided.
• Programs will develop management response and action plans (MRAPs) to address MTR recommendations. However, expectations for follow-up on implementation of actions were unclear, including timing and involvement of oversight bodies such as program steering committees.

Unclear plan for assessing program outcomes

At this time there is no clear plan to assess whether the Challenge programs achieved their intended outcomes and the extent to which the NRC has successfully responded to the Minister's priority challenges. CSTIP must be evaluated overall at least once every 5 years, with the next evaluation by NRC OAE scheduled for FY 2028. This evaluation would not assess results of individual Challenge programs. There are no plans to conduct Challenge program-specific assessments at the end of the program.

Information availability

Mixed perceptions on information availability

NPO and Challenge program staff were divided on whether available information is sufficient. Nearly all IIAs and program advisors said they have the information they need to do their jobs. Program directors were split, with some expressing concerns about data being unstructured and out of date, and others being generally satisfied. Those who felt data was not readily available did note they could access it through specific requests to NPO. NPO management considers the available information to be inadequate, due to an inability to aggregate CSTIP program-level information.

Externally, PAC members considered their access to information as only somewhat sufficient. Notably, they wanted more information on program and project progress, including clear baselines and expected timelines, by which to monitor performance and provide guidance.

Need to improve information availability, integration

At both the CSTIP and Challenge program level, the lack of an integrated data collection system inhibits the availability of information. With information spread out among a mix of independent forms, emails, and standalone databases, key stakeholders do not readily have the information they need to make data-informed decisions. For program management, this is particularly relevant for strategic planning.

While NPO consistently tracks G&C-related project information, the data is limited, consisting mainly of high-level descriptive statistics and immediate outcomes information (see appendix B). Information that is not collected or readily available includes:

• CSTIP- and program-level process efficiency information, limiting continuous improvement and resource management
• progress towards EDI targets, and GBA Plus considerations, limiting the assessment of impacts on different populations
• NRC human resource and labour information, namely who is participating in the Challenge programs, limiting assessment of intra-NRC collaboration and actual costs to RCs
• medium-to-long term outcomes information (e.g., TRL progression), limiting decisions on program direction and project portfolio management (e.g., project continuation, scale-up, or termination)

Ultimately, information availability could be strengthened by integrating CSTIP- and program-level tracking and reporting.

Recommendations and Management response and action plan (MRAP)

Supporting rationale and recommendations

Priorities, expectations and organizational culture:

• The Challenge program model (herein “the model”) is designed to support transformative research that spans the TRL spectrum from disruptive or foundational research to knowledge mobilization and pre-commercialization.
• The model is flexible by design, but the expectations are unclear for the type of research and the path for outcomes when a program ends. Research funded to date may not be generally characterized as transformative, high-risk and high-reward. There is insufficient structure for managing the advancement of project portfolios (i.e., project maturation and scale-up or termination), developing technology with industry or other partners, and defining successful, end-user adoption.
• The model provides the NRC with a platform for its 14 RCs and external partners to collaborate on priority challenges for the benefits of Canadians. Most funding is for G&Cs, and RCs have been asked to contribute to an increasing number of programs using their base operational budget. VPs and DGs were critical about available resources in light of perceived conflicting priorities. In order to support an NRC-wide culture change toward collaborative and mission-oriented research, organizational priorities and expectations would need to be clarified.

Accountability, ownership and oversight:

• The governance structure for the collaborative R&D initiatives has been effective overall. Individual Challenge programs are accountable for their actual R&D outcomes. However, the accountability and ownership of the Challenge program model overall is not clearly established. Given its novel and flexible nature, clarity is needed as to whether NPO, or another entity, is accountable for the model (i.e., entity responsible to assess the model and to make adjustments over time based on lessons learned, MTRs and evaluations).
• Programs are answerable to various internal and external committees, which have been implemented differently across programs. Together, this has resulted in reporting burden and inconsistent expectations for what programs report, and how they respond, to oversight bodies. In particular, there is an opportunity to better define the roles and responsibilities of the PSC in relation to those of other committees and management's response to the MTR process.

Program assessment:

• There are no plans to conduct program-specific assessments beyond their mid-term. It has yet to be determined how the NRC will assess if individual programs achieved their intended results by the end of their life cycle.
• There is a need to clarify who is accountable to report on the extent to which the NRC has successfully responded to the Minister's priority challenges for the benefits of Canadians.

Roles and responsibilities for the Challenge function:

• It is not consistently defined or understood how expert advice is used to “challenge” project selection towards effective portfolio management across project phases of programs. In practice, the Challenge function is diffused between IIAs, oversight committees (i.e., EPRC and PAC) and MTRs.
• To ensure funded research aligns with desired program outcomes, it is necessary to clarify the "Challenge function" within the program model, specifically in regards to the role of the IIA.

Sustainability and scalability:

• NPO was established as a new independent function to enable the delivery of G&C funding to NRC collaborators while complying with the PTP. In the 5 years since, NPO has provided oversight, support, and G&C management to an increasing number of programs and funding agreements. To its credit, it has done so with a relatively flat budget and staff, and without an overarching framework to support consistency and efficiency.
• In turn, NPO has been playing catch up. As of FY 2023 it was still implementing improvements recommended by a policy and process review from FY 2021. Elements of this review are included in the CSTIP Toolkit Roadmap, which NPO plans to implement in FY 2024. Moving forward, NPO resources will be further strained if the number of CSTIP programs and funding streams continues to grow.

System integration and outcome-based management:

• NPO’s tools and processes are spread across multiple systems and software platforms, resulting in inefficiencies. There are also data gaps and a need for better automation of data collection to improve timely analysis of key processes. Integrating key CSTIP processes for program and project development, G&C administration and client relationship management would support more effective and efficient administration of CSTIP, especially if the program continues to grow. This may provide a platform to integrate processes and facilitate information exchange between program staff, peer reviewers and G&C recipients.
• There are gaps in the information that is critical for effective project portfolio management and the assessment of progress toward intended outcomes. Performance measurement tools and processes are not yet fully implemented and, importantly, outcome-focussed indicators are either not being collected or sufficiently aggregated. Roles and responsibilities for data collection are at times unclear as both NPO and program hosts have performance monitoring obligations. CSTIP requires a set of performance and outcome-focussed key indicators, including appropriate service standards, with clear accountability for data collection established. Integrating these indicators into a common system could improve outcome-based management.
• NPO has made policy and process improvements by reviewing and streamlining specific tools and processes. As part of its CSTIP Toolkit Roadmap, NPO plans to further review its business tools and processesso that they can be easily adapted and integrated into 1 system. Systems integration will require the completion of ongoing streamlining initiatives, including the development of an overall framework that comprehensively maps CSTIP processes.

Management response and action plan

Recommendation 1: priorities, expectations and organizational culture

The VP-BPS should further clarify expectations for the Challenge program model with respect to:

• advancing transformative, high-risk, high-reward research
• structuring the development of technology and policy solutions for end-user adoption
• prioritizing research centres participation and intra-NRC collaboration

Risk level: moderate

Recommendation 2: accountability, ownership and oversight

The VP-BPS should clarify accountability and ownership for the overall CSTIP to ensure that the Challenge program model achieves its goals, including whether oversight committees could be consolidated.

Risk level: moderate

Recommendation 3: program assessment

The VP-BPS should clarify with senior management the need to assess whether individual Challenge programs have achieved their intended outcomes and, if so, how this will be done.

Risk level: moderate

Recommendation 4: roles and responsibilities for the Challenge function

The VP-BPS should clarify the:

1. “Challenge function” within the Challenge program model, particularly as it relates to program development and direction, project selection and portfolio management
2. roles and responsibilities of IIAs outside of their due diligence function for G&Cs

Risk level: moderate

Recommendation 5: sustainability and scalability

The VP-BPS should examine the sufficiency of resources (human, financial and information technology) to complete ongoing continuous improvement initiatives to enable the success of the Challenge program model and CSTIP overall, and develop a plan for moving forward.

Recommendation 6: system integration and outcome-based management

The VP-BPS should develop and implement an integrated system for key CSTIP processes, including a platform to facilitate interaction and information sharing among key stakeholders. This system should also integrate strengthened performance information to promote process effectiveness, efficiency and outcome-based management.

Risk level: high

Appendices

Appendix A – detailed methods

Document review

Internal documents were reviewed to provide context, to assess the program management practices against the NRC and TB policies, and to complement other lines of evidence in assessing the effectiveness of program model and governance, processes and controls, including performance monitoring systems and reviews. Procedures were used to identify and review key business processes and procedures, tools and templates, guidelines and quality assurance approaches and methodologies.

Internal documents included CSTIP terms and conditions, strategic, operational and HR plans, program governance, model and profiles, process maps, conflict of interest forms, field guides, terms of reference and other key documents of various committees and panels, job descriptions, performance and oversight reporting material, management toolkits, and MTRs.

External documents were mainly reviewed as part of the comparative analysis study to provide context on Canadian and international mission-oriented innovation policies and initiatives.

Data review

Corporate data was sourced from nBoss and SAP systems, G&Cs and O&M funding tracking tools, and also included CSTIP administrative data, information on projects and funding recipients, program project tracking tools and program and project management workbooks, and financial and human resources data.

Targeted data analytics were used to assess and validate program management assertions. Performance monitoring data were also reviewed including key performance indicators at CSTIP and Challenge program level.

Available Challenge programs project close-out reports were also reviewed (n=44) to obtain feedback from funded external collaborators.

Internal and external interviews

Individual and group interviews were conducted to collect personal experiences, opinions and expert knowledge related to the effectiveness of the Challenge program model and governance, processes and controls and the use of information to inform decisions.

A total of 73 stakeholders were interviewed between November 2023 and March 2024. 76% were internal staff and 24% were external to the NRC:

• 56 internal staff: senior management, VPs and DGs, NPO and NPBS management and staff, business development directors, Challenge program directors and managers, Cluster Support program directors, team leads and co-leads, and NRC IRAP ITAs
• 17 external stakeholders: funded and unfunded external R&D collaborators and members of program advisory committees and other committees

Case study interviews are included in this total (29 people). Interviews and consultations carried out as part of the comparative analysis are not included in this total.

Case studies

The case studies focused on documenting and understanding the experience of 3 Challenge programs as they evolved within CSTIP governance and processes:

1. HTSN: RC-hosted; round 1 program
2. PRCP: division-hosted; emergency response program
3. AIP: division-hosted; round 2 program

These programs were selected to include one round 1 program, one round 2 program and a mix of sponsorship types (division- or RC-hosted programs). Planning consultations recommended the inclusion of the PRCP due to its unique nature.

Case studies allowed for in-depth investigation and the collection of context-specific evidence to support the broader evaluation findings. The case study approach included the review of program documents, data and interviews.

• In addition to program-level information sources, a review of a non-random sample of projects (n=11) was completed to examine project-level documents and data.
• Interviews were conducted with program management staff (n=4), research team leads and co-leads (n=6), business development directors (n=3), chairs of PACs and other committees (n=4), NRC IRAP ITAs (n=4) and with external funded (and unfunded) collaborators (n=12).

Collaborators and sample of projects were selected to include a range of project values (high, medium and low), funding and project types.

Comparative analysis

Selection process

Identifying comparator programs was difficult due to the unique collaboration and funding model of the NRC's Challenge programs. It was not possible to find comparator programs that included both government researchers and external organizations funded to work collaboratively on R&D projects.

From extensive online searches and multiple consultations with NPO and NPBS, 2 domestic and 2 international initiatives supporting mission-oriented research were selected based on goals, funding model, and program structure and other features. See details on comparators programs and rational for selection in the Profile section.

Rationale for selection of comparator programs

• ICI Cleantech Challenges: to explore the strong similarities in federal policy drivers and requirements, program objectives, as well as oversight and delivery structures
• EIP: to leverage the experiences and best practices of NRCan in managing transfer payments for R&D projects, as well as to examine similarities and differences in processes and governance structures
• ARPA-E: to further investigate the strengths and weaknesses of the Defense Advanced Research Projects Agency (DARPA) program model (after which ARPA-E was designed), including lessons learned in achieving high-risk, high-reward research objectives
• EIC Pathfinder: to explore the similarities and differences in oversight and delivery structures under the EIC program model (inspired by features of ARPA-E) and others

Data collection methods

Data collection and analysis consisted of a review of documents and data. There was some information on comparators based on publicly available documentation (program websites, past audit and evaluation reports). An interview guide and information request checklist were developed based on key indicators for comparison and other elements that could not be found in the public domain. Representatives from all 4 comparator programs were interviewed: 2 individual or group interviews (groups of 2 to 3 individuals) relative to each of the 4 comparator programs. Additional documents were shared by interviewees and follow-ups were completed to fill in information gaps and to validate information.

Appendix B – limitations and mitigation strategies

Although the evaluation encountered some challenges, methodological limitations were mitigated, where possible, through the use of multiple lines of evidence and the triangulation of data. This approach was taken in order to establish the reliability and validity of the findings and to ensure that conclusions and recommendations were based on objective and documented evidence. Details on limitations and their associated mitigation strategies are described below.

Lack of data on NRC contributions

It was not possible to quantify the contributions of individual RCs and the materiality of Challenge programs (i.e., total value of projects and programs) as the planned contributions in project proposals are not systematically or equally tracked across the organization. This prevented the effective assessment of the:

• actual level of effort of RCs to Challenge programs
• effects of the R&D collaborative model at changing collaborative behaviours between RCs over time
• scalability (and ultimately the viability) of the R&D collaborative model given that RCs are solicited to contribute to an increasing number of Challenge programs over the same period of time, with a similar level of operating funds and priorities for mandated and revenue generating activities

Mitigation:

To examine RC’s contributions to Challenge programs, OAE:

• interviewed internal stakeholders, including case study interviewees who could comment on such contributions
• reviewed program data and documentation that referenced RC involvement
• used planned NRC contributions for the Profile section

Limited perspective from external stakeholders

Given the scope of this first evaluation of the CSTIP framework and processes, the evaluation team relied as much as possible on available data to avoid increasing the burden on external funding recipients. Concurrent to the evaluation, there were 4 MTRs involving interviews with external recipients and committee members. Funded collaborators were also asked to provide feedback on their experience of working with the NRC on processes and reporting requirements for recently completed projects.

The next CSTIP evaluations should include wider consultations with external stakeholders and take advantage of more extensive feedback and related performance information collected, reported and analyzed by NPO.

Mitigation: OAE and NPO closely worked together to:

1. avoid overlap between evaluation case study interviews and MTR interviews by sharing contact lists and questionnaires
2. incorporate findings from MTR reports when available (n=4 reports)
3. include external feedback from close-out reports compiled by NPO (n=44 reports)

Variability and imbalance of information in comparative analysis

Information gaps are to be expected in a comparative analysis. Information and interviewee availability varied across comparator programs. Ultimately for 1 comparator was the consultant able to reached the targeted number and categories of key informants (i.e., program support and oversight function and program direction).

Given the scope of this evaluation, which looked in detail at processes and controls, there was an imbalance of information between CSTIP and the comparator programs. As a result, this analysis was intended to primary to compare frameworks and models for funding mission-oriented research, rather than specific processes and controls. Finally, the unique nature of CSTIP (with the NRC as both research funder and performer), limited the identification of best practices and lessons learned, as the comparators fund perform research.

Mitigation: The evaluation team reviewed publicly available documents and data, then followed up with comparator representatives by e-mail to fill information gaps. When some categories of key informants were unavailable, available interviewees provided valuable information on other functions and identified some supplementary documentations. Finally, for some analyses, a mix of quantitative and qualitative information sources was used to provide proxy indicators that can be put into perspective with those of CSTIP, and draw meaningful comparisons.

Appendix C – supplementary information

Summary table of KPIs collected and reported by NPO