Evaluation of the Collaborative Science, Technology and Innovation Program 2023-24

Status: Active
Effective date: December 5, 2023
Prepared by: Office of Audit and Evaluation
Approval: NRC's President
Cat. No.: NR16-434/1-2024E-PDF
ISBN: 978-0-660-69472-6
Related document: Evaluation of the Collaborative Science, Technology and Innovation Program 2023-24 - summary report
Alternate format: Evaluation of the Collaborative Science, Technology and Innovation Program 2023-24 (PDF, 1.61 MB)


Aging in Place
Advanced Research Projects Agency-Energy
Business and Professional Services
Disruptive Technology Solutions for Cell and Gene Therapy
Conflict of interest
Collaborative Science, Technology and Innovation Program
Defense Advanced Research Projects Agency
Director general
Equity, diversity and inclusion
Expert Peer Review Committee
European Innovation Council
Financial Resources and Program Committee
Grants and contributions
GBA Plus
Gender-based Analysis Plus
High-throughput and Secure Networks
Impact Canada Initiative
Innovation investment advisor
Key performance indicator
Materials for Clean Fuels
Mid-term review
National Programs and Business Services
National Program Office
Industrial Research Assistance Program
Operations and maintenance
Program Advisory Committee
Project close-out report
Pandemic Response Challenge Program
Program Steering Committee
Policy on Transfer Payments
Research and development
Research centre
Senior Executive Committee
Terms and conditions
Treasury Board Secretariat
Technology readiness level


The evaluation of the National Research Council’s (NRC) Collaborative Science, Technology and Innovation Program (CSTIP) covers the 5-year period from 2018-19 to 2022-23. It was carried out in accordance with the NRC Departmental Evaluation Plan, the Treasury Board Policy on Results (2016) and in compliance with the Financial Administration Act. This is the first evaluation of CSTIP and covers all fiscal years since the program’s inception.

This report begins by providing a profile of CSTIP. It then presents findings on the extent to which CSTIP’s program model, processes and controls, and information management are adequate to support the achievement of program objectives. Following the findings are recommendations for improvement.

Evaluation approach


This is a process evaluation of CSTIP’s collaborative research and development (R&D) initiatives, focussed particularly on the Challenge programs. As a process evaluation, it assessed the extent to which programs were set up to achieve their intended goals and whether progress to date has been supported by CSTIP’s collaborative R&D framework. It did not assess program outcomes. Focus was placed on the Challenge programs as they have been identified as an area of potential future growth for the NRC.


This evaluation was led by the NRC’s Office of Audit and Evaluation (OAE). It applied a mixed-methods approach, incorporating both qualitative and quantitative data from several lines of evidence. This allowed for the triangulation of data. A Gender-based Analysis Plus (GBA Plus) lens was applied throughout the evaluation.


Lines of evidence and analytical methods that support the evaluation include:

  • document review (internal and external sources)
  • data review (administrative, financial, program, project, and performance data)
  • interviews (n=73 interviewees)
    • NRC staff (n=56 interviewees)
    • external stakeholders (n=17 interviewees)
  • case studies (n=3 Challenge programs), including document and data review, case-study specific interviews (n=29 interviewees, included above) and the review of a non-random sample of projects (n=11)
  • comparative analysis (n=4 comparator programs), including the review of program documents and interviews (n=6 interviewees)

See appendix A for detailed information on the methods and appendix B for limitations.

Evaluation questions

  1. Is the CSTIP collaborative R&D program supportive of the intended goals to foster breakthrough research that will benefit Canada and enhance collaboration with and within the NRC?
    1. Does the program model support intended goals?
    2. Is progress towards the achievement of goals supported by effective and efficient processes and controls?
  2. Does program management have access to information needed to make informed decisions?
  3. How does the Challenge program model compare to similar international and Canadian programs?


The Collaborative Science, Technology and Innovation Program (CSTIP) uses grant and contribution (G&C) funding to bring together the best minds from across the research ecosystem to address Canada’s most pressing public policy challenges and deliver scientific and technological breakthroughs.

Program overview

The National Research Council’s (NRC) Collaborative Science, Technology and Innovation Program (CSTIP) is a transfer payment program established in fiscal year (FY) 2018-19 in an effort to position the NRC at the centre of research excellence and collaboration. With $150 million (M) in federal funding over 5 years and $30M per year ongoing, CSTIP aims to catalyze collaborative, high-risk, high-reward research and deliver tangible scientific and technological breakthroughs that address some of Canada’s most pressing public policy challenges. CSTIP uses G&C funding to convene strategic, highly-dynamic research teams from across academia, industry and government.

CSTIP includes 3 separate components that leverage G&C funding, provided and administered by the NRC’s National Program Office (NPO), to engage with external collaborators:

  1. Collaborative Research and Development (R&D) initiatives ($107M in G&Cs to date): Comprised of the Cluster Support and Challenge programs, these initiatives bring together the NRC’s national network of researchers and facilities with collaborators from industry, academia, and government in order to tackle specific research objectives across a variety of scientific disciplines.
    Cluster Support programs ($32M in G&Cs to date): Support the objectives of Canada's 5 Global Innovation Clusters.
    Challenge programsFootnote 1 ($75M in G&Cs to date): Support the development of disruptive technologies to address social, economic and environmental challenges affecting Canadians.
  2. Ideation Fund ($9.6M in G&Cs to date): The Ideation Fund, which includes the New Beginnings and Small Teams initiatives, supports exploratory, transformative research by NRC scientists collaborating with external partners.
  3. Outreach initiative ($1.2M in G&Cs to date): The Outreach initiative aims to increase the NRC’s outreach, engagement, and support of research excellence across science, technology, engineering and mathematics, with a focus on under-represented diversity groups.

Focus on the Challenge programs

The Challenge programs present a new platform for the NRC to collaborate with external stakeholders—one where it is both a funding agent and an active collaborator. CSTIP has launched 9 Challenge programs since 2018-19, with 3 more currently in development.

The Challenge programs seek to:

  • advance high-risk, high-reward research that is mission-oriented towards transformative outcomes
  • catalyze multi-disciplinary collaborations across NRC research centres (RCs) and with external partners across research sectors and industries, and outside Canada (6% of funded projects were in Europe, Japan or the United States)
  • provide scientific solutions to Canada’s most pressing challenges (current and emerging)

To accomplish this, the Challenge programs are:

  • collaborative by design: developed in consultation with internal and external stakeholders, and strengthened through peer review, during program development and at the mid-term
  • time-limited and staggered: typically 7 years in length, with mid-term reviews (MTRs) in year 4; new programs are generally launched every 3 years
  • funded annually: baseline budget for G&Cs ($24M per year) and NRC-side operations ($4M per year)
  • innovative: advancing projects that span the innovation continuum, from early-stage, foundational research to knowledge mobilization and pre-commercialization (TRL 1-9)

Over 5 years, the Challenge programs provided 219 grants and 44 contributions to collaborators:

  • 85% academia
  • 5% industry
  • 2% provincial and territorial government
  • 9% not-for-profit and others

Note: Total percentage of collaborators adds up to more than 100% due to rounding. Includes all G&C funded projects from April 1, 2018 through February 28, 2023.

Transfer payments at a glance

The Government of Canada’s Policy on Transfer Payments (PTP) defines a “transfer payment” as a monetary payment made by the government to an eligible, third-party recipient. Transfer payments, which include grants and contributions (G&Cs), are primarily used to enable and engage capabilities and resources that exist outside the federal government.

What is a transfer payment program?

Transfer payment programs, like CSTIP, are the mechanism by which the government leverages third-party assets and directs recipients’ activities towards the achievement of specific objectives and outcomes. Per the PTP, programs must:

  • be designed, delivered, and managed with integrity, transparency and accountability
  • clearly demonstrate value for money through tangible contributions to program objectives and departmental results
  • be supported by appropriate administrative, oversight and control systems that are proportionate to the level of risk specific to the program
  • remain recipient-focused, with an emphasis on accessibility, inclusivity and usability

What is the difference between grants and contributions?

  • Grants are subject to pre-established eligibility and other entitlement criteria, but are typically not subject to audit. Under CSTIP, grants are primarily used to engage with academia and not-for-profit organizations.
  • Contributions are subject to stricter performance conditions specified in a funding agreement, and are also subject to recipient audit. Under CSTIP, contributions are typically used for more complex projects with multiple collaborators, to engage with industry and to make foundational investments (i.e., equipment purchases).

As of FY 2023, the funding limit for both grants and contributions, per recipient per project, is $2 million per year. Oversight and reporting requirements for grants remain relatively lower than for contributions.

The need for the National Program Office

The PTP requires transfer payment programs to have clear separation between the organizational unit that issues the transfer payment and the organizational unit that collaborates with the payment recipient in order to ensure an unbiased and transparent process, free from conflicts of interest. This requirement sparked the need for the NRC to establish the National Program Office (NPO), an independent corporate entity to manage all G&C funding under CSTIP and provide administrative oversight to the recipient application process.

The National Program Office (NPO)

As of January 2023, NPO had a total of 36 staff:

  • Annual expenditures: $4.0M per year (FY 2023)
  • NPO Management: 3
  • Operations and governance: 11
  • Ideation program: 5
  • R&D programs: 11 (including innovation investment advisor)
  • Digital and data management: 4
  • Program policy: 3

Roles and responsibilities

NPO is responsible for the day-to-day administration and oversight of CSTIP, including:

  • program development support
  • due diligence and advice on funding requests
  • funding decisions related to G&Cs
  • funding agreements administration and management
  • program policy and operational guidance
  • coordination and oversight of internal and external committees and peer review, including the MTRs
  • monitoring and reporting on progress, performance and achievement of program outcomes

The innovation investment advisor

The innovation investment advisor (IIA) acts as the liaison between NPO and the collaborative R&D programs (including both Challenge and Cluster Support programs). Formerly known as Challenge officers, the IIA plays an impartial advisory role or “Challenge function,” in the strategic oversight and delivery of the programs and individual R&D projects. The Challenge function is also provided by other oversight bodies within the NRC. The IIA’s role includes:

  • advising on NRC resource and funding investments
  • establishing strategic co-funding partnerships
  • supporting the design of programs through stakeholder engagement and the establishment of expert advisory committees
  • performing due diligence and impartial assessment on program and project proposals
  • funding projects in support of collaborative R&D programs, including management of programs’ G&C budgets, competitive funding calls and agreements

Challenge program governance

Figure 1.
Challenge program governance
Figure 1 – Text version

This map represents the governance structure for typical Challenge programs.

Program management bodies:

  • NRC Senior Executive Committee (SEC)
  • NRC Financial Resources Program Committee (FRPC)
  • VP Business and Professional Services (VP-BPS)
  • National Program Office (NPO)
  • Program host (VP or DG Research)
  • Program director and management team

Advisory bodies:

  • Program Steering Committee (PSC)
  • External Peer Review Committee (EPRC)
  • Program Advisory Committee (PAC)
  • Mid-term review (MTR) including Peer Review Committee (PRC)

R&D collaborators:

  • Participating NRC research centres (RCs)
  • External collaborators

SEC and FRPC provide authority and approval to CSTIP Challenge programs.

VP-BPS and NPO provide program oversight, administration and G&C management to CSTIP Challenge programs. NPO issues G&Cs to external collaborators.

Program host, program director and management team provide program delivery and scientific direction to CSTIP Challenge programs.

PSC, PAC and MTR-PRC provide expert advice and guidance to program director and management team.

CSTIP Challenge programs produce R&D projects which involve participating RCs and external collaborators.

Note: The above represents the governance structure for a typical Challenge program. Some variation exists between programs in terms of the number and membership of advisory committees.

Challenge program lifecycle

Internal and external bodies fulfill program management, advisory, and collaborative research functions through a program’s lifecycle. 

Program management

Stakeholders Program concept Program proposal Project selection G&C admin R&D project delivery Performance reporting Mid-term review Closure







Program managementFootnote 2


Advisory bodies

Stakeholders Program concept Program proposal Project selection G&C admin R&D project delivery Performance reporting Mid-term review Closure








MTR PRCFootnote2








PSCFootnote 2






PACFootnote 2








R&D collaborators

Stakeholders Program concept Program proposal Project selection G&C admin R&D project delivery Performance reporting Mid-term review Closure
Participating RCsFootnote 2






External collaboratorsFootnote 2






Round 1: Challenge programs

CSTIP launched 4 Challenge programs between FY 2018-19 and FY 2019-20. All 4 were hosted by an NRC research centre (RC), and underwent MTRs in FY 2022-23.

Program title Focus Program host or lead Projects (as of FY 2023) Academia (%) NFP and other (%) Industry (%)
Artificial Intelligence for Design (AI4D)

Algorithms, methods and datasets for the design and scientific discovery of AI-driven technologies

Disruptive Technologies Research Centre





Disruptive Technology Solutions for Cell and Gene Therapy (CGT)

Technology solutions for cell and gene therapies to significantly improve health outcomes

Human Health Therapeutics Research Centre





High-throughput and Secure Networks (HTSN)Footnote 3

Technologies for delivering secure, affordable and high-speed internet in rural and remote communities

Advanced Electronics and Photonics Research Centre





Materials for Clean Fuels (MCF)

Clean, sustainable energy and chemical industry through materials innovation

Energy, Mining and Environment Research Centre





Round 2: Challenge programs

CSTIP launched 4 Challenge programs between FY 2021-22 and FY 2022-23. Two round 2 programs were hosted at the division level, rather than by an RC.

Program title Focus Program host or lead Projects (as of FY 2023) Academia (%) NFP and other (%) Industry (%)
Aging in Place (AIP)Footnote 4

Innovation for safe and healthy aging, supporting a sustainable model for long-term care.

Life Sciences Division





Internet of Things: Quantum Sensors (QSP)

Advancement of quantum sensing technologies towards adoption and commercialization.

Security and Disruptive Technologies Research Centre





Arctic and Northern (ANCP)

Support strong, sustainable Northern communities through applied technology.

Engineering Division





Applied Quantum Computing (AQC)Footnote 5

Quantum applications and software for the simulation of complex physical systems.

Digital Technologies Research Centre





Pandemic Response Challenge Program

Hosted by the Life Sciences division, the Pandemic Response Challenge Program (PRCP), was a special 2-year program that aimed to fast-track research and development (R&D) for pandemic-related challenges and gaps identified by Canada's health experts. The program was developed under tight timeframes with its own unique governance structure and external advisors for each project. Program leaders took a portfolio approach to risk management by selecting a mix of low to high-risk projects to mitigate overall program risk. By planning and acting quickly, the program was an example of how to structure a short-term Challenge program to respond quickly to a crisis like COVID-19.

Program title Focus Program host or lead Projects (as of FY 2023) Academia (%) NFP and other (%) Industry (%)
Pandemic Response Challenge Program (PRCP)Footnote 6

Public health

Life Sciences Division





Challenge program timeline and financials

CSTIP costs have increased over time with the launch of new Challenge programs and changes to its terms and conditions (T&Cs). Note the spikes in operations and maintenance (O&M) in FY 2020-21 due to extra dedicated funding to support the PRCP, then in FY 2021-22 in response to additional funding from Budget 2021 as part of Canada’s National Quantum Strategy (benefitting AQC and QSP).

FY 2018-19

  • NPO formed
  • Round 1 programs developed and 3 launched (CGT, HTSN, MCF)

FY 2019-20

  • Fourth round 1 program launched (AI4D)
  • T&Cs amended to expand grant eligibility, and enable PRCP

FY 2020-21

  • PRCP launched in response to COVID-19 pandemic
  • T&Cs amended for inclusivity

FY 2021-22

  • Round 2 programs launched (AIP, ANCP, QSP)

FY 2022-23

  • AQC program launched (via Budget 2021)
  FY 2018-19 FY 2019-20 FY 2020-21 FY 2021-22 FY 2022-23

G&C (actuals)






O&M (actuals)






RC contributions (planned)Footnote 7

$0Footnote 8





NPO admin costs (actuals)Footnote 9






Estimated total






Comparator programs

The evaluation methodology included a comparative study of CSTIP’s Challenge program operational framework with that of similar, mission-oriented research programs. The intent of the study was to position the NRC’s overall approach in comparison to other approaches, and to identify best practices. To this end, 2 Canadian and 2 international programs were selected in collaboration with NPO for comparison:

Comparator program Mission statement Total program fundingFootnote 10 Project budgetFootnote 10 Project duration TRL range

Impact Canada Cleantech Challenges (6 programs):

Impact Innovation Unit (IIU), Privy Council Office (PCO) and Natural Resources Canada (NRCan)

Address persistent barriers in cleantech development and adoption by setting ambitious but achievable goals to identify and develop breakthrough solutions.

$75M over 5 years (program funding)

$500K up to $5M

2-3 years


Energy Innovation Program (EIP):

Office of Energy Research and Development (OERD), Natural Resources Canada

Advance clean energy technologies that will help Canada meet its climate change targets, while supporting the transition to a low-carbon economy.

$319M over 7 years (program funding)

$500K up to $7M

Up to 5 years


Advanced Research Projects Agency – Energy (ARPA-E):

United States Department of Energy

Fund innovative, high-risk technologies that have the potential to transform the energy landscape but are not yet mature enough for private sector investment.

$536M in 2021 (program, open, exploratory funding)

$627K up to $12.5M

1-3 years


EIC Pathfinder:

European Innovation Council (EIC)

Support the exploration of bold ideas for radically new technologies, welcoming the high-risk/high gain and interdisciplinary cutting-edge science collaborations that underpin technological breakthroughs.

$504M in 2023 (program and open funding)

Up to $6M

Up to 4 years


Program model

The CSTIP collaborative R&D initiative provides a flexible program model for funding collaborative research towards addressing societal challenges. Its flexibility was by design, which enabled some programs to leverage both in-house resources and external funding opportunities where they align with their specific challenge mission. The model is effective at enhancing collaboration with external researchers, mainly in academia, through the provision of grants and contributions.

As familiarity with CSTIP increased, the model has also been effective at enhancing program-level collaboration across the NRC, with CSTIP acting as a ‘change program’ for the NRC. Amendments to CSTIP’s terms and conditions supported broader and more inclusive collaboration. Programs incorporated external guidance through a number of committees at different stages in their lifecycles. To date, round 1 Challenge programs have attracted collaborators to work with the NRC, co-author publications and generate IP, and feedback from G&C recipients and NRC researchers on collaborations has been positive.

It is not possible to conclude whether the Challenge program model provides an effective structure to advance funded research towards breakthrough outcomes, given that round 1 programs are only passed the midpoint of the 7-year lifecycles in FY 2023. However, the model provides insufficient structure for managing the advancement of project portfolios (i.e., project maturation and scale-up or termination), developing technology with industry or other partners, and defining successful, end-user adoption. More broadly, the model has suffered from a combination of resource allocation challenges at both the CSTIP and program level, unclear roles and responsibilities, mixed expectations, heavy oversight yet inconsistent structure, and possible change fatigue.

Governance structure: flexible program model


The CSTIP collaborative R&D initiative provided a flexible model that centralized G&C administration while affording variation in Challenge program implementation. Each program’s governance was somewhat unique, with variation in how programs were hosted within the NRC, and their committee and reporting requirements.

Stakeholders satisfied with centralized program office

NRC stakeholders,including senior management and program directors, were generally satisfied with the NPO-centralized governance model of Challenge programs. Challenge and Cluster Support program directors agreed that an NPO-like body is essential to support program establishment and implementation, while permitting flexibility in specific program delivery. Program directors and NPO leadership consulted regularly via a community of practice (CoP) to identify issues, share best practices, and recommend changes on program governance.

The NRC Challenge program model is unique

The comparative study found the Challenge program model—in which the NRC is both a funding agent and R&D collaborator—to be internationally unique. While some comparator programs do allow for the participation of national laboratories or other federal agencies as 3rd party R&D performers, the comparator organizations themselves do not directly collaborate with G&C recipients on funded projects.

Flexible models for program hosting within the NRC

Challenge programs may be hosted at the division level or hosted at the RC level. There were advantages associated with each approach:

  • Division-hosted, VP sponsored: more aligned with the mission-oriented intent of CSTIP, tending to support greater intra-NRC collaboration and NRC-wide strategic planning
  • RC-hosted, DG sponsored: more directly connected to both research staff and support capacities (e.g., business development, program management)

Ultimately, model suitability depends on multiple factors such as program scope, the nature of the challenge, and the resources and range of expertise needed. Overall, program directors and senior management agreed that this flexibility is warranted going forward, rather than selecting a single model.

Flexible program direction a best practice

Several comparator programs highlighted the importance of establishing flexible delivery models and authorities that support the program’s unique needs and objectives.

Governance structure: advice and oversight


Committees played important roles in program governance, but there is a need to clearly define their intentions. There may be opportunities to consolidate committees, streamline oversight functions and reduce reporting burden for programs.

Programs supported by advisory committees

The program advisory committees (PACs) provided expert advice and guidance to the Collaborative R&D programs. CSTIP’s flexible model allowed external expert advisory bodies to meet the differing needs of the Challenge programs. For example, the RC-hosted CGT employed its Research Centre Advisory Board (RCAB) to play this role, while the division-hosted PRCP deferred to interdepartmental committees on an ad-hoc basis for scientific advice and direction. HTSN however established its PAC relatively late in the program’s lifecycles, limiting its influence in program direction and project selection.

Role for new steering committees unclear

A new layer of internal governance, the program steering committee (PSC), was added to the program model in FY 2023. It is intended to liaise between programs and the NRC’s executive committees, and to guide mid-term changes. All future programs must have a PSC, but only 2 have one as of March 2023 (AQC, which launched in FY 2023, and HTSN, which piloted a PSC earlier). Existing programs may convene PSCs following their MTRs to support implementation of their subsequent management response and action plans (MRAPs), but this remains optional.

It is not yet possible to comment on PSCs’ effectiveness in supporting program development and implementation. Moreover, it was unclear how PSCs will complement existing committees.

Program oversight excessive

Challenge programs are answerable to various internal and external committees. Committee implementation was inconsistent across programs. Together, this resulted in reporting burden and inconsistent expectations for what programs report, and how they respond, to oversight bodies.

Program directors considered oversight to be excessive. At different points in their lifecycle, programs may be reviewed by or required to report to a combination of internal steering committees and advisory committees, unique external peer review committees for program design (EPRC), project selection committees, MTRs, and RCABs. Implementation has varied between programs, notably with some programs now adding PSCs and different approaches to PACs.

It was difficult to assess how oversight may be streamlined or consolidated as it is not consistently implemented across programs. It may be possible to standardize reporting so that programs present the same progress reports to committees, rather than unique ones for each.

Roles and responsibilities: National Program Office


Program stakeholders appreciated NPO’s central role, and there was interest in expanding NPO’s involvement in facilitating intra-NRC collaboration. The roles and responsibilities of innovation investment advisors (IIAs), who are key program staff, were unclear or inconsistent across programs.

Central office necessary but expectations mixed

The general role of NPO as a central administrative and support function for CSTIP programs was clear, and well-appreciated by program stakeholders. However, the extent to which NPO contributed to the scientific direction of Challenge programs, or advised on funding decisions and partnerships, varied across programs.

There was interest from some internal stakeholders for NPO to act as a facilitator between research centres, branches, and NRC Industrial Research Assistance Program (NRC IRAP) to increase awareness of Challenge programs and enhance intra-NRC collaboration. NPO did this to some extent, such as for the PRCP, but had limited capacity to fulfill this role.

“The roles and responsibilities are getting clearer. The lack of clarity at the beginning was necessary since there is a need for some flexibility within different Challenge programs.”

Program director

“The role of NPO needs to be better defined, whether purely administrative or if there is a Challenge function. I feel strongly that this is the weakness of the system right now.”

Innovation investment advisor

Innovation investment advisor (IIA) roles differ across Challenge programs

As of FY 2023, with regard to the role of the IIA, there remained considerable confusion and difference in experience between programs. There was a lack of consensus among program directors and IIAs on the nature of the IIA role. The core issue is to what extent IIAs were meant to provide a “Challenge function”, which is confounded by a mix of interpretations as to what this function entails. In practice, this function was performed by program directors and oversight committees, in addition to IIAs (see more on the Challenge Function under Processes and controls).

In addition to the necessary administrative due diligence, clarity is needed on the involvement of IIAs in providing advice on program and project direction at both the development and delivery level.

IIA are without comparator

The comparative study found the IIA and its role as the liaison between NPO and R&D programs to be unique to CSTIP, as it is necessitated by CSTIP’s own unique collaborative model.

Roles and responsibilities: committees and corporate services


The roles and responsibilities of committees and business services varied over the evaluation period and across the 9 Challenge programs. Though engaged at times, these bodies external to NPO and Challenge program management could be more consistently and directly engaged.

Influence of advisory committees varies

Program Advisory Committees (PACs) were established as a source of impartial expert advice for each Challenge program. This general role was well communicated and understood by CSTIP stakeholders. PACs include external and internal members, including staff from the NRC IRAP. However, committees were inconsistently implemented across the Challenge programs and at different points within program lifecycles.

For most programs, PACs played an important role in challenging and strengthening their overall strategic direction. In some cases, PACs were involved in the selection of individual projects while others fulfilled a less direct, advisory role. Per case studies, PACs tended to focus either on larger grants or ‘master projects.’ NRC IRAP members brought industry perspectives to programs and connected them with businesses (e.g., they facilitated technology demonstration and testing for AIP stakeholders).

Impartial expert advice as a standard

Like CSTIP, the comparator programs rely on independent external experts and committees to evaluate project proposals for technical merit, provide scientific guidance and strategic direction to program management, as well as assess project performance and program impacts.

Unclear role for the NRC’s business services

Internal stakeholders who provide business development, contracting and other services to NPO and Challenge programs were at times unsure of their roles and responsibilities for CSTIP-funded projects. When developing G&C agreements under CSTIP, business development staff were not always involved, or involved late in the process, creating ambiguity about when and by whom NRC business decisions need to be made. These issues were further complicated by differences between the division and RC-hosted models, the latter of which have their own in-house business development resources.

Bilateral G&C agreements, which constituted the vast majority under CSTIP, were processed internally by IIAs by way of a standard template, as opposed to being routed through NRC contracting services as a typical collaborative research agreement would be. While NPO did seek input from the NRC’s contracting team for more complex agreements (i.e., multi-party agreements, international collaborations, or those requiring amendments), the existence of separate processes with varying degrees of involvement from corporate staff compounded inefficiencies.

There is an opportunity to develop a specific process, with clear roles and responsibilities for the NRC’s business services, that would help ensure appropriate negotiations and checks occur.

Program compliance


NPO ensured program compliance by working closely with TBS throughout the implementation and evolution of CSTIP’s T&Cs. As CSTIP continues to grow and change, this relationship may support continued compliance.

Evolving, flexible program-maintained compliance

When CSTIP first launched in FY 2019, there was no similar program in Canada (i.e., where a federal agency both works on collaborative projects and provides G&Cs to its external collaborators). Treasury Board Secretariat (TBS) allowed the NRC flexibility in its interpretation of the Policy on Transfer Payments (PTP) because of the unique collaborative nature of CSTIP.

In the spirit of continuous improvement, NPO and TBS worked together on T&Cs changes, ensuring oversight and policy compliance throughout the program’s evolution. When CSTIP began, the T&Cs were relatively restrictive, particularly for grant eligibility, and demanded a high level of scrutiny for all payments. This was warranted as CSTIP was a new funding program for the NRC. After 2 years, NPO demonstrated its competence in this area and was able to change the T&Cs to be more flexible, expand grant eligibility, and moderate scrutiny for payments.

Amendments to the T&Cs were also made specifically in response to COVID-19 in order to implement the PRCP, and to increase grant opportunities for Indigenous recipients.

Changes to CSTIP terms and conditions

Year Main changes Intent and impact

FY 2019-20

Grant criteria; rapid response

Greater use of grants, fewer contributions; 2-year PRCP

FY 2020-21


Reduced barriers for Indigenous proponents; expanded cost eligibility

FY 2023-24

New funding streams

Further flexibility (strategic response programs, challenge prizes)

Program changes and growth pose risk

As CSTIP evolves, NPO needs to continue working closely with TBS. To date, CSTIP has complied with federal legislation thanks to sufficient controls and processes for managing G&Cs and financial administration.

There is a risk that NPO will be insufficiently resourced to maintain compliance as the program changes and grows. NPO has been playing “catch up” with Challenge programs over the past 5 years. Their capacity to do so will be stretched thinner as the total number of Challenge programs and CSTIP eligibility criteria continue to grow without additional efficiencies or operations funding.

Model viability and scalability: NPO resources


The long-term viability of the CSTIP Challenge program model may be inhibited as it is scaled up, in terms of new funding streams and total active programs, as NPO may lack the resources to support CSTIP overall.

NPO responsibilities may outweigh resources

Scale-up of CSTIP would likely encounter significant problems with NPO’s existing resource levels and responsibilities. The relative workload for NPO increased over the evaluation period, above and beyond initial expectations for the office. Notably, the NRC initially planned to launch 3 new Challenge programs every 3 years or “round”, but increased that to 4 per round, not counting the PRCP.

Currently, NPO has the human resources necessary to administer day-to-day G&C operations. As of FY 2022, NPO was responsible for roughly 6 times the number of projects as it was in FY 2019 with only twice the staff. NPO struggled to formalize its systems, processes and tools to support the implementation and monitoring of CSTIP programs. Should processes be improved, NPO may achieve the efficiencies needed to manage a larger workload.

New funding streams were approved in FY 2024, adding a strategic response stream with shorter program lifecycles and a new Challenge prize stream. The assignment of new responsibilities to NPO (e.g., serving a potential liaison role between research centres) would exacerbate resource issues should program growth not be paired with additional operations funding or efficiencies.

Figure 2. Increasing number of active CSTIP projects relative to NPO resources
Increasing number of active CSTIP projects relative to NPO resources
Figure 2 – Text version

Stacked column chart showing the number of CSTIP’s active projects in Challenge, Ideation and Cluster Support programs compared to the total number of NPO staff from fiscal year 2019 to fiscal year 2022. The number of CSTIP active projects has increased from 50 to 307. The number of total NPO staff has increased from 17 to 38.

Fiscal year HR (total number of staff) Active Challenge program projects Active Ideation program projects Active Cluster Support program projects Total active CSTIP projects

FY 19






FY 20






FY 21






FY 22






Source: NRC Human Resources and CSTIP project database.

CSTIP manages more projects per resource

The comparative study estimated that the administrative cost ratio for NPO (10%) was within the range of comparator programs (8 to 10%). This ratio calculates total program administration costs relative to total G&Cs per year. However, NPO manages a greater number of total projects while the comparators tend to fund larger but fewer total projects. In addition, CSTIP project agreements involve both internal and external collaborators, which makes the administration complex.

Model viability and scalability: research centre participation


The extent to which RCs have aligned their resources and other existing priorities with Challenge programs has varied across RCs. There is opportunity to enhance the impact by increasing the consistency of RC participation.

Limited funding to host programs, collaborate

Challenge program hosts are provided with limited operations and maintenance (O&M) funding. After deducting program direction and administration costs (i.e., program director and support role salaries), host RCs may have relatively little funding to resource NRC-side collaborators. This issue has been echoed by MTRs, that found round 1 programs lacked the internal resources necessary to achieve their ultimate outcomes.

The extent of this issue differs by program. Some hosts have accessed additional O&M monies through alternative sources, such as through revenue generation, other government department funding, or TB submissions for related national strategies. Overall, programs fell into 3 camps:

  • one third of programs have dedicated funding for administration (e.g., via alternative sources)
  • one third are covered by their host RC
  • one third are using CSTIP O&M funding (up to 60%)

Hosts aligning RC priorities with Challenge programs

Program hosts acknowledged it can be difficult to balance Challenge program participation with other internal priorities (e.g., revenue generation). Where possible, hosts have aligned their Challenge programs with other priorities and projects, including funding sources, and thus buttressed their CSTIP commitments with additional resources.

Just over half of programs (5 of 9) sought out or accessed third-party funding or launched revenue projects (versus G&C projects). For examples, AIP, CGT and ANCP implemented projects where the NRC and external collaborators collectively seek third-party funding (e.g., AIP and CGT are leveraging funding from the Canadian Institutes for Health Research, and ANCP from the Canada-Inuit Nunagat-United Kingdom research programme). Other programs, such as MCF, included research service projects within their Challenge program portfolio.

Non-host RCs need to balance Challenge programs with other priorities

Similarly, non-host RCs must balance Challenge program participation with other priorities, and align them where possible. RCs felt they were required to cut from their regular business and operational budgets in order to support the Challenge programs, which they viewed as butting up against revenue targets and other strategic deliverables. At the same time, it has not been clearly communicated to RCs that Challenge programs are an organizational priority, and thus RCs must budget and plan accordingly.

Model effectiveness: advancing ‘breakthrough’ research


CSTIP has yet to demonstrably advance transformative, high-risk, high-reward research. Research has been mission-oriented (i.e., towards societal challenges) but most funded projects may not be considered high-risk, high-reward research. It is also unclear how the model could be used to support breakthrough outcomes within the 7-year program lifecycle.

Incremental change, not yet ‘breakthrough’

It is unclear how CSTIP prioritizes the most promising research and defines success for each Challenge program. The model did not provide Challenge programs a structure to focus on achieving tangible solutions for Canadians. To date, programs funded a range of projects, each moving incrementally up the technology readiness level (TRL) scale but did not appear to be systematically managing their project portfolio towards planned outcomes. Internal interviewees were divided on whether the model supports breakthrough research, but tended to agree that the low TRL focus would preclude breakthroughs for Canadians within 7-year program lifecycles.

To date, the program model did not appear to be scaling up promising research with near-term commercialization potential and terminating activity without (i.e., the DARPA model). G&C size was relatively consistent over the first 5 years of the round 1 programs, suggesting programs are not focussing resources on a smaller selection of projects post-midpoint. This was affirmed by MTRs which found round 1 program goals were too ambitious given their timeframe, and suggested programs focus on scaling up technology and facilitating industry pull.

Figure 3. External collaborators reported TRL increases
External collaborators reported TRL increases
Figure 3 – Text version

Barbell chart showing the average TRL of projects at launch and the average TRL of completed projects for PRCP, AI4D, CGT, MCF and HTSN programs.

Challenge program Average TRL launch Average TRL completed
















Source: Self-reported TRL progression from round 1 project close-out reports (PCRs) (n=44).

Models for transformative research

International comparators provide an alternate model to achieve breakthrough research. ARPA-E and EIC Pathfinder focus only on early-stage (TRL 1-5), high-risk, high-reward projects, rather than the full TRL continuum, as CSTIP does. Upon completion, the most promising projects are then continued by in-house programs that focus on scale-up (TRL 6-10), in order to bring new technologies to market. The 3-year Pathfinder program, for instance, is linked to a 1-year transition program and a 3-year commercialization program—the same 7-year length as most Challenge programs.

Model effectiveness: enhancing external collaboration


Since its outset, the CSTIP model has been effective at enhancing collaboration between NRC and external academic partners towards research aligned with federal priorities. It was less evident to what extent collaboration with industry or end-users outside academia was enhanced.

Engaging both past and new collaborators

Challenge programs enhanced collaboration with external researchers and feedback from G&C recipients and NRC researchers on collaborations has been positive. Challenge programs partnered with players in their respective fields, leveraging unique and complimentary expertise and facilities towards their goals.

The availability of high-value grants was particularly effective at attracting academic collaborators; most funded projects (85%) were with academic institutions. Within 3 years, the NRC and G&C recipients co-authored over 240 publications, and submitted over 40 patent applications.

Programs have attracted new, external collaborators to work with the NRC. According to project close-out reports (PCRs) for round 1 programs, slightly more than half (55%) of principal investigators (PIs) had not worked previously with the NRC. However, PCRs were only available for two fifths of completed projects, and thus it is not possible to conclude definitively that most PIs were new to working with the NRC.

Figure 4. Round 1 program outputs
Round 1 program outputs
Figure 4 – Text version

Column chart showing round 1 program outputs in co-authored publications, invention disclosures and patent applications.

Fiscal year Co-authored publications Invention disclosures Patent applications

FY 2020




FY 2021




FY 2022




Source: CSTIP program results database (NPO).

Programs tasked to engage industry more

As of FY 2023, only 5% of external collaborators were businesses. Though industry engagement may be expected to be low in the first half of programs’ lifecycles (i.e., low TRL), EIC Pathfinder had 22% industry participation for programs at a similar stage.

Generally, NRC Challenge programs planned to transfer technology to industry, rather than co-develop it. According to PCRs, 47% of G&C recipients felt commercialization within 5-10 years was very likely.

Program oversight bodies have recommended that programs use the latter half of lifecycles to implement models to engage industry. PACs and mid-term reviewers stressed the essential role industry should play in bringing CSTIP-supported technologies to market in order to address programs’ respective challenges. Obstacles to this shift to industry included the COVID-19 pandemic, and the limited funding available to attract larger companies.

Model effectiveness: enhancing internal collaboration


As familiarity with CSTIP increased, its flexible model has begun to prove effective in enhancing collaboration across RCs.

Research centres collaborating on challenges

The model has demonstrated effectiveness at enhancing program-level collaboration across the NRC, with CSTIP acting as a ‘change program’ for the NRC. It has directed RCs to lead projects that contribute towards common challenges. This has necessitated a significant culture shift for RCs and remains an ongoing effort.

  • Round 1 programs demonstrated limited intra-NRC collaboration. On average, each program involved 3.5 RCs, but 85% of projects were led by their host RC. This may be attributed to limited familiarity with the new model (G&Cs for externals without funding for non-hosts) and a lack of early engagement with RC leadership during program design, which in turn hindered the necessary culture shift.
  • Round 2 programs demonstrated significant intra-NRC collaboration and are less dominated by a single RC. On average, each program involved 5.3 RCs, and only half (49%) of projects were led by their host or single RC. This may be attributed to multiple factors, including increased familiarity with the model, the adoption of the division-hosted model by some programs, and the ability of some hosts to secure additional O&M funding (thus reducing costs to participate).
Figure 5. More RCs leading round 2 program projects
More RCs leading round 2 program projects
Figure 5 – Text version

100% stacked column chart showing percent of projects led by host or primary RC compared to percent of projects led by other RCs per Challenge program. More RCs leading program projects in round 2 compared to round 1.

Challenge program Percent of projects led by host or primary RC Percent of projects led by other RCs

























Source: NPO project database (pre-dates launch of AQC).

Continuous improvement


There is a need for more streamlined and standardized processes from NPO. Program efficiency, oversight and information management are undermined by a lack of structure.

Recognized room to improve

NPO and senior leadership recognized that CSTIP remains, in part, a work-in-progress. There is a need for more streamlined and standardized processes as well as for integrated systems at the NPO level. CSTIP was launched as NPO and its processes and tools were still in development. Some issues identified by NPO as part of a policy and process review in FY 2021 remain unaddressed or mitigations remain in development. In FY 2024, NPO established a CSTIP toolkit roadmap to implement outstanding actions within the year.

Ultimately, there is a need to allow flexibility and autonomy at the program level, balanced with common structures and processes to maintain appropriate oversight, resourcing and reporting. Programs have been uniquely governed and funded, resulting in mixed experiences among program leaders.

Leaders sharing what works

Continuous improvement is supported by the program directors’ community of practice (CoP), a platform to identify common problems, share best practices, and respond to NPO and senior executive with a single voice.

As of fall 2022, the CoP, at the President’s direction, has been meeting to discuss challenges in program execution, including program administration and funding, internal NRC collaboration, and program oversight. The discussions have carried over into division and research centres’ strategic planning in spring 2023. No formal plans or recommendations have been made by the CoP and on some matters, namely oversight, they are waiting on the results of this evaluation.

PRCP a rapid response ‘test case’

PRCP went through the complete program life-cycle from inception to delivery and completion in 2 years. According to a lessons-learned report prepared by PRCP leaders and teams, programs developed in tight timeframes and under extenuating circumstances benefit from:

  • a multidisciplinary program launch tiger team, and the integration of NPO and NRC governance (i.e., including IIA in program management committee)
  • taking a portfolio approach to program and risk management (have a portfolio of low to high-risk projects to mitigate program-level risk)

In FY 2023, NPO applied some of these lessons as part of changes to CSTIP’s T&Cs, formally adding a rapid response model as an alternative to the standard 7-year program.

Processes and controls

CSTIP processes and controls have been adequate to support program implementation, but are not necessarily as efficient or as effective as they could be. Overall, CSTIP was launched before processes and controls could be fully developed by NPO. Therefore, NPO has been playing ‘catch up’ over the past 5 years, developing the structure necessary to support CSTIP collaborative R&D’s goals, while also responding to the roughly biannual changes in CSTIP’s terms and conditions and to the growth in the total number of Challenge programs.

Process consistency


Processes are generally consistent at the NPO level, albeit unstructured by an overarching framework. At the program level, processes are often inconsistent in areas such as project ranking, review and approval, and research pillar development. 5 years into CSTIP’s existence, NPO- and program-level field guides remain in development, tools that would help IIAs and program directors, and promote process consistency.

Program-level inconsistency may inhibit model

Programs have implemented their own processes, with differing amounts of documentation and data collected. As flexibility is a feature of the model, some inconsistency may be expected. However, processes and tools could be standardized to support program and information management. As of FY 2023, a field guide for program directors remained in development. This would provide much-needed standardized guidance for both current and future program staff, while still permitting some flexibility.

The design and development of individual Challenge programs follows a series of common steps. Differences, however, were observed in terms of sub-theme or “pillar” development among case study programs. The earlier programs studied, HTSN and PRCP, built their research pillars around the NRC’s in-house capabilities. This ran the risk of perpetuating existing approaches to research activities, though for PRCP this was justified by the need to act quickly. AIP, a round 2 program, built its pillars thematically around the “challenge” itself, which better aligned with CSTIP’s intent of encouraging collaboration within and outside of the NRC.

NPO-level processes need framework, field guide

NPO-level processes are implemented in a generally consistent manner (e.g., G&C process, payments) and were found to be aligned with the Policy on Transfer Payments (PTP). NPO mapped some of its processes in recent years, notably its financial controls and claims process, to ensure program compliance with the PTP and improve efficiency. However, NPO is generally using forms and templates to drive processes, rather than an overarching framework. Moreover, as already stated a field guide specifically for IIAs would better define that positions’ roles and responsibilities.

An overarching CSTIP framework may balance structure with flexibility. For example, a framework could provide guiding principles, and map and define the intent of core processes, while permitting flexible implementation so long as programs adhered to the principles. This would reduce inconsistencies in program implementation, management and oversight, but still allow programs to implement processes in a risk-appropriate manner suited to their unique needs. A program framework would also provide an anchor for a robust IIA field guide.

Internal controls


Internal controls are adequate but not necessarily proportional, and may inhibit the use of contribution agreements. There are opportunities to calibrate controls based on project attributes, as well as to further leverage NRC IRAP processes, as was done for financial controls.

Opportunity to calibrate controls

Internal controls for project due diligence, approval, and monitoring are in place. NPO interviewees reported that due diligence and internal controls are generally appropriate, but sometimes excessive as the same level of controls are required regardless of the amount of funding. NPO leadership contended that heavy controls were warranted at CSTIP’s outset as it was a new program administered by a new office.

Having proven its competence, both internally and with TBS, there are opportunities for NPO to calibrate controls based on project attributes (e.g., size, grant vs. contribution, risk rating). Internal interviewees noted the relationship between due diligence and risk, where processes could improve if risk-based tools were standardized. At first, CSTIP did not leverage NRC IRAP’s existing processes, long used for contribution agreements, to the extent possible. To address this, NPO recently hired an NRC IRAP employee with PTP expertise to guide revisions of their funding instruments, processes, and risk-model framework.

“It’s the same level of rigour for any amount. In some cases, it’s the right amount but sometimes it is overkill.”

Innovation investment advisor

CSTIP choosing grants over contributions

To date, NPO has favoured the use of grants over contributions. Interviewees indicated that programs are using grants for academic partners, mainly for highly qualified personnel. As for contributions, NPO staff reported that contribution agreements are used for more complex projects. Contribution agreements take more effort in terms of documentation and milestone-based claim requirements, for both agreement development and implementation. T&Cs changes made in FY 2021 enabled CSTIP to disburse funding almost entirely by grants (96% of all agreements in FY 2023, up from 78% in FY 2020).

Building on the NRC’s past G&C experience

CSTIP avoided some of the issues experienced by the NRC’s former Canada Accelerator and Incubator Program (CAIP), as identified by that program’s evaluation in 2019. CAIP supported a broad range of activities and costs, beyond simply overhead and salaries, which in practice demanded regular line-by-line reviews of invoices. For CSTIP, grant eligibility is determined up front thus requiring relatively little follow-up, and contributions are generally directed towards salaries, thus avoiding demanding reviews.

Conflict of interest


Targeted efforts have been made to strengthen conflict of interest management. This has mitigated CSTIP’s risk exposure, which has increased due to CSTIP having taken on a greater and more varied selection of funded collaborators.

NPO has identified and managed conflicts of interest (COI)

NPO has made sustained efforts to improve the identification and management of COI by maturing their processes, educating CSTIP stakeholders (primarily NRC staff), and including COI forms at specific program and project milestones. This will be especially important if CSTIP continues to grow, both adding new programs and expanding eligibility criteria, which will increase risk exposure.

NPO worked cooperatively with the NRC’s Value and Ethics team to strengthen COI management. In FY 2021, NPO implemented the “Conflict of interest attestation for NRC principal investigators (PIs)” in an effort to identify and manage issues of COI at the project level. There is an opportunity for CSTIP to further strengthen its process and ensure regular review of the NRC PI attestation, similar to what is required for external participants. As there are multiple potential sources for real or perceived COIs, it may be prudent for NRC staff to re-examine their situation whenever they sign onto a new collaboration agreement.

Potential sources of real or perceived COI:
  • existing relationships between the NRC and a potential recipient
  • NRC PIs who hold a position at recipient organizations (academia especially)
  • committee members who are actual or potential recipients
  • evolving relationships with recipients (e.g., student hiring, spin-offs companies)

Committee COI risks unique, but well-managed

COI for external EPRC and PAC members is well-managed. Upon selection, members are required to sign confidentiality and COI declaration forms, and agree to review and update their declarations regularly or as soon as their interests change.

The requirement for regular review of COI is important for all stakeholders, given that Challenge programs are 7 years in length and individual research projects can last 3 to 4 years. Potential COI with external committee members is an issue faced by many programs, including CSTIP comparator programs. It is particularly true for those within a relatively small research ecosystem. The number of unique committees required by the CSTIP exacerbates this effect, as it rapidly depletes the pool of available candidates.

Process effectiveness: opportunity to integrate processes


Processes are generally effective, and viewed positively by collaborators, but may be strengthened overall if integrated.

Processes generally effective, improved over time

CSTIP processes are generally effective in the implementation of Challenge programs, such that:

  • grants and contributions are being issued
  • transfer payments are managed appropriately

G&C recipients generally gave positive feedback on CSTIP processes:

  • program management were engaged and accommodating
  • IIAs were easy to communicate with and provided support
  • satisfied with level of collaboration with NRC researchers

Internal stakeholders noted improvements made by NPO, including the streamlining and simplification of processes (e.g., agreement amendments). The NRC achieved this in part by bringing leadership with significant G&C experience to NPO.

“We have established processes. We’ve been building processes as we go. We’re constantly improving in this first 5 years. We started with nothing and no people.”

Program advisor

Process integration valuable but a major investment

NPO internal stakeholders identified the need for an integrated client relationship management (CRM) system. Business tools and processes are not currently integrated, as different program and project phases use different software platforms (i.e., project management by a mix of email, spreadsheets, other software).

A CRM system or other integrated system could support program staff, G&Cs recipients, and other program stakeholders in effectively implementing processes. Integration would demand a significant level of effort due to the current disparate state of systems. Successful integration may also require an overall framework that comprehensively maps CSTIP processes, which is currently lacking. Thus, NPO’s level of readiness to implement a CRM, as well as the cost that would be associated, could be prohibitive.

Canadian, international programs integrate systems

ICI, ARPA-E and EIC use web portals to integrate processes and facilitate information exchange between program staff, peer reviewers and external funding recipients. For example, they can be used to support document and data processing, project application, development and selection, project reporting and monitoring, and communications.

Process effectiveness: opportunities to strengthen model


Intra-NRC collaboration, tech transfer, and risk-assessment could be strengthened to enhance CSTIP impact.

Cross-RC collaboration

There are no formal processes or structures to support intra-RC collaboration. Program directors suggested that NPO act as a facilitator, which could be possible through direct engagement or maintenance of shared tools. Formal support of cross-collaboration may increase awareness of Challenge programs and research centre capabilities, thus reducing barriers to intra-NRC collaboration. Note that this would add responsibilities for NPO. 

Intellectual property and technology transfer

The structure necessary to commercialize solutions stemming from Challenge program projects, namely processes for intellectual property (IP) management and technology transfer, was underdeveloped.

IP processes were sufficient for lower TRL projects where near-term commercialization was unlikely. However, they were not well-defined for higher TRL projects or more complex situations. Ownership of co-developed IP was negotiated on an ad hoc basis via IP agreements separate from project collaboration agreements.

Technology transfer mechanisms were not well-developed, and clear plans were not in place to advance research outcomes beyond program lifecycles.

Project risk assessments

Project risk assessments appear to have been conducted subjectively by IIAs (i.e., without a standardized method), and implementation varies. A standardized approach to project risk assessments with objective ranking criteria would strengthen due diligence by NPO.

Additionally, these assessments only consider funding risks (i.e., likelihood recipient will deliver on project), and are conducted post project-selection. It is unclear how project proposals are assessed for technical risk in order to identify high-risk, high-reward research in the spirit of the CSTIP model.

Challenge identification


Challenge identification has been generally driven by government priorities. This responsive approach has been effective in aligning NRC with national initiatives involving other federal agencies and in securing additional funding sources. However, there are opportunities to clarify and increase awareness of how Challenge programs are initiated.

Identification process responsive, albeit unstructured

The NRC has identified challenges in response to federal priorities as they have emerged. This approach has enabled CSTIP to:

  • align existing division and RC capabilities with other federal agencies and initiatives
  • leverage funding sources for additional G&C and O&M in addition to the original CSTIP funding in Budget 2019

Following Budget 2021 for instance, the NRC identified the AQC challenge to align with the National Quantum Strategy, and secured additional funding for its pre-existing quantum program, QSP. This approach is continuing for programs in development as of FY 2023 (e.g., Critical Minerals Strategy, Greener Homes Initiative).

This approach, however, has been unstructured, lacking standard processes to identify challenges or determine how many programs may be launched simultaneously. Some DGs did not know how they themselves could identify a challenge and recommend a new program. Providing some structure to the identification process may enable RCs to find alignment with emerging federal priorities and recommend new Challenge programs.

Standard processes to develop programs

Processes are more formal and consistent after challenge identification and for program conceptualization. Before any new Challenge program can be launched, it must undertake standard activities such as external peer assessment and presentation to FRPC. Given that the FRPC was put in place recently, it is to soon to assess if this structure will streamline the development and approval of program proposals. Similarly, it is not clear which role the PSCs to be created will play in this process.

External stakeholders are then engaged to assist in program development through various activities, including one-on-one meetings and interviews, presentations, workshops, and surveys. This helps ensure that programs are not designed in isolation.

Most projects aligned with Challenge program aims

Approaches to program pillar development have been inconsistent, which may resultin different approaches to project portfolio management and engagement strategies. However, MTRs of the programs found that pillars and projects are generally well aligned with their respective external challenges.

Challenge function


The Challenge function is only somewhat effective, as it is neither clearly defined nor commonly understood among stakeholders. In practice, this function is dispersed between IIAs, oversight committees and MTRs. It is unclear to what extent programs are challenged in selecting projects for scale-up or termination.

Challenge function not thoroughly enacted

Foundational CSTIP program documents connect the Challenge function to the DARPA model, where it is described as oversight on project milestones and deliverables to achieve the program's ultimate objectives. This is distinct from the societal “challenge” a given program aims to address. Rather, it is an internal function to “challenge” the programs themselves. Within CSTIP, the function tends to be dispersed between IIAs, EPRCs, PACs, and MTRs (which include peer review committees). However, committees, namely the PACs, do not seem to play a role beyond project intake and selection.

With the function diffused and inconsistently understood, it is not evident that research activity is being challenged. Case studies found most project proposals were embedded within Challenge program proposals, and thus were collectively reviewed and approved along with their respective program. Information was not available to determine what proportion of projects, if any, were terminated because they no longer aligned with the program or were not performing as expected.

Innovation investment advisors (IIA), directors differ on ‘challenge’ roles

IIAs reported a need to distinguish between a 'true' Challenge function (i.e., challenging project selection and scientific portfolio management) and the administrative function of handling G&Cs (i.e., ensuring program alignment, and adherence to T&Cs). Based on their experience with industry and research program management, some IIAs believe they should be providing both. This has caused frustration for some IIAs who feel their capabilities are underutilized. In comparison, IIAs have similar skillsets to NRC IRAP’s industrial technology advisors (ITAs) but have less influence over G&C direction and research activity.

Program directors consider the IIAs' Challenge function to be mainly around G&C funding, aligning with CSTIP T&Cs, and monitoring project milestones and deliverables. Over time, this has increasingly been the case. As CSTIP expanded grant eligibility, more grants were awarded relative to contributions. This resulted in less opportunity for IIAs to challenge the ongoing relevance and direction of projects as there are fewer opportunities to do so with a grant than with a contribution agreement.

Challenge function for portfolio management

EIC Pathfinder uses a 2-stage project selection process to challenge research. Proposals are first peer reviewed for scientific excellence. Then, programs holistically review proposals that cleared stage 1 to create a smaller, complementary project portfolio.

Process efficiency


CSTIP processes have improved incrementally over the life of the program, though there remain areas to improve. Efficiency could not be assessed due to significant gaps in data.

Some evidence of increased efficiency

Both internal and external stakeholders found processes had become more efficient over CSTIP’s first 5 years, and highlighted specific improvements:

  • financial monitoring (similar to NRC IRAP) with integrated risk ratings was established, helping to streamline payments
  • greater signing authority was granted to IIAs and a risk assessment matrix was introduced, simplifying agreement approvals

G&C recipients tended to be content with CSTIP processes overall, but contribution recipients were more likely to have issues with processes. This aligns with the views of internal staff who found contributions more demanding to administer than grants. Internal and external stakeholders also noted that there remain areas for improvement, including:

  • complexity of processes for contribution agreements and claims payments
  • volume and clarity of reporting requirements
  • complexity of collaboration agreements and associated processes for development

As of spring 2023, NPO was mapping and reviewing its claims payment process with an aim to speed up delivery.

Figure 6. Satisfaction with processes
Satisfaction with processes
Figure 6 – Text version

Bar chart showing the percentage of grant and contribution recipients who reported positive satisfaction with the CSTIP process. The percentage of contribution recipients is lower than the percentage of grant recipients.

Project close-out reports (PCRs) question Percent of grant recipients (n=30) who provided positive response Percent of contribution recipients (n=14) who provided positive response

Funds received in a timely manner



Processes understood and efficient



Reporting not too heavy



Source: Round 1 project close-out reports (PCRs) (n=44); percent agreeing somewhat or strongly.

Significant data gaps inhibit efficiency tracking

It was not possible to assess process efficiency throughout program and project lifecycles due to 3 important limitations:

  1. NPO lacks a detailed framework and process map to define and track its processes
  2. insufficient capture of administrative information to permit analysis and quantify process efficiency
  3. there are currently no service standards or efficiency baselines against which to compare

NPO reported it may be difficult to establish service standards due to the variety of programs and funding streams under CSTIP, which may be further exacerbated by T&C changes in FY 2024, and the lack of an integrated system for information management.

Gender-based Analysis Plus (GBA Plus)


GBA Plus considerations are required in program and project proposals, and CSTIP’s T&Cs were updated to be more inclusive. However, insufficient tracking prevented an assessment of the engagement of diverse populations, the consideration of GBA Plus in research activities, or advancements in employment equity.

Gender-based Analysis Plus and equity, diversity and inclusion (EDI) considered in program and project design

CSTIP demonstrated consideration of GBA Plus and EDI in its overall design, at the Challenge program level, and at the project level.

  • CSTIP level: TB submissions made some GBA Plus commitments (e.g., requiring committees to assess program and project proposals with an intersectional lens), but most commitments were around EDI (e.g., hiring targets for program hosts and collaborators). T&Cs amendments in FY 2021 did promote the inclusion of Indigenous persons by expanding eligible activities and types of recipients.
  • Program level: GBA Plus was considered in development of Challenge programs, though to varying extents. 2 of the 3 case study programs explored how they may uniquely impact specific populations. The AIP program, notably, convened an “experts by experience” panel for adults 65 and older or their caregivers to advise on program direction and participate in research. EDI was considered less, with only 1 program proposing how it would target specific equity groups for advertising and employment.
  • Project level: There was some evidence of GBA Plus principles being integrated at the project level. For example, a project using sensors to measure oxygen in the brain tests people of diverse skin pigmentations, as this can affect the sensor's accuracy. For the Arctic and Northern Challenge Program, all projects require collaborators from Arctic and northern populations.

Limits on data collection

Information was not collected on NRC workforce diversity at the CSTIP level, namely whether or not researchers working on the program reflected the NRC’s EDI targets. Due to the complexity of attributing new RC hires to CSTIP funding specifically, NPO is planning to develop new metrics for GBA Plus achievements at the CSTIP level (rather than EDI).

For each Challenge program, NPO mainly tracks progress towards GBA Plus-related indicators via 2 project-level instruments: annual performance reports and PCRs. Within these instruments, there is a strong overlap of GBA Plus and EDI, with the focus on the latter. Round 1 program PCRs show that most projects (82%) included EDI considerations for their team, but less than half (39%) considered GBA Plus in project design.

Program directors felt that they have little ability to require GBA Plus or EDI elements from recipients, and that there are challenges around assessing progress because of limits on collecting data.

Information management

Program management has sufficient information to administer CSTIP. NPO and programs are tracking program implementation and project activity, generally, but there are significant gaps for tracking efficiency and progress towards outcomes. Some key performance indicators tracked at the program or project level are not integrated, which limits overall performance monitoring. MTRs provide necessary checks on program performance, but round 1 MTRs were limited in their ability to inform adjustments and assess success to date. Importantly, there is no plan for the assessment of individual programs longer-term outcomes (i.e., beyond the midpoint). As such, program management and NRC executives lack information on the extent to which programs are addressing their respective challenges.

Data collection planning: performance monitoring


NPO and programs have responsibilities to monitor CSTIP and program-specific performance. However, monitoring systems were not fully or consistently implemented, precluding the ability to track and report on progress towards outcomes.

Monitoring divided between NPO, programs

The CSTIP model divides performance monitoring between the funder (NPO) and performers (program hosts). Generally, NPO focusses at the CSTIP level (e.g., G&C accountability, overall achievements), while performers focus on progress towards challenge-specific outcomes. NPO needs to aggregate program-level key performance indicators (KPIs) to assess CSTIP overall, which can be challenging. In practice, there are gaps and inconsistencies at both the CSTIP and program level.

Resourcing information management

According to NPO and Challenge program staff, incremental resources are needed for performance monitoring. With demands to implement new programs and limited O&M funding, NPO and programs believe they lack the resources needed to fully monitor performance as planned.

Systems not fully implemented at NPO

NPO has yet to fully implement its monitoring system as initially envisioned. In an effort to reduce reporting burden on programs, NPO has not requested that all KPIs be tracked. It is collecting and reporting descriptive statistics and anecdotal outcomes. As of FY 2023, not all data collected was being aggregated and reported.

NPO has revised tools to systematically collect and aggregate data. For example, they are using project close-out reports (PCRs) to capture KPIs directly from G&C recipients (e.g., satisfaction, TRL progression, student involvement) but only began aggregating data in FY 2024. Moreover, though required to do so, only half of G&C recipients (51%) submitted PCRs within a year of completing their project.

Data collection varies between programs

Programs use modified, or in some cases unique, templates to capture KPIs that relate to their subject matter. However, this flexibility makes it difficult for NPO and programs to aggregate data. This is further complicated by the divided reporting responsibilities of the CSTIP model.

A working group has been struck to bring NPO and programs together to address reporting responsibilities. As of FY 2023, programs were not routinely sharing outcomes data with NPO, instead providing some data upon request. Program staff are working with IIAs to determine how their own project reviews may be utilized at the CSTIP level to avoid duplication of efforts or burdening internal and external PIs with information requests.

Alignment of performance tools


Performance indicators and monitoring processes are insufficient to assess whether program administration is efficient and whether longer-term outcomes are being achieved. NPO does not have service standards or other metrics for assessing efficiency, and programs are not consistently monitoring and reporting on outcomes.

Need for service standards

Performance monitoring is inadequate to meaningfully assess CSTIP processes and support continuous improvement. At the NPO level, there is a lack of service standards or baselines to compare performance against, and a mismatch of indicators for performance attributes.

For instance, while NPO intends to measure efficiency with an overall administrative-cost ratio, it would be more useful to assess NPO service against set service standards—such as how quickly it responds to grant proposals or fulfills contribution claims. As of March 2023, NPO had begun mapping out and measuring its claims process so that it could in fact assess its efficiency in a meaningful and constructive way.

Lessons learned in information management

Comparator programs highlighted the importance of integrated data systems for data collection, performance monitoring and reporting, with an emphasis on accessibility of information to both internal and external stakeholders.

Need for clear outcomes and consistent monitoring

For outcomes metrics, planned indicators are sufficient for corporate reporting needs (i.e., the NRC’s departmental results and reporting to TBS) and related to specific goals (e.g., business innovation) but insufficient for assessing outcomes. In some cases, NPO has deferred the measurement and assessment of outcomes to future CSTIP evaluations, which are not likely to assess outcomes of individual Challenge programs. In other cases, NPO relies on program hosts to monitor and track progress towards outcomes as part of regular program management.

However, even in cases where relevant outcome data is being collected, it is not being centrally aggregated. This undermines the ability of NPO to monitor program performance. The lack of aggregation and monitoring exposes the program to the risk that monitoring regimes differ across Challenge programs thus undermining access to performance information.

“Trying to measure impacts is hard, and is going to require resources in order to have meaningful measures. Other organizations have whole offices just do to this.”

Senior manager

Assessment: mid-term reviews and final assessments


MTRs provide necessary checks on program performance to assess progress towards success and inform course adjustments. In round 1 programs, delays, data gaps, and unclear roles limited the usefulness of the MTRs. Importantly, there is no clear plan for the assessment of individual program outcomes at the end of the program.

Reviews are opportunities to adjust course

MTRs of Challenge program performance and progress towards outcomes is essential to ensure that programs address their respective challenges. However, their efficacy may be limited by late timing within program lifecycles, insufficient performance data, and unclear plans for taking action following the MTR.

  • Round 1 program MTRs were facilitated by NPO and conducted by an external consultant. These MTRs included external peer review committees, and were conducted in year 4 of their 7-year lifecycles, 1 year later than planned.
  • The MTRs found programs to be relevant (e.g., program strategies and projects aligned with challenges) but could not fully assess progress towards outcomes, due to limited performance data and information from external collaborators. For example, one MTR recommended an entirely new set of KPIs to assess project suitability for continuation, scale up or termination. Another MTR could not assess progress towards outcomes whatsoever based on the data they were provided.
  • Programs will develop management response and action plans (MRAPs) to address MTR recommendations. However, expectations for follow-up on implementation of actions were unclear, including timing and involvement of oversight bodies such as program steering committees.

MTRs focused on 4 questions

MTRs assessed alignment between:

  1. the external challenge and the program’s strategy
  2. program strategy and the project scope and goals
  3. program plans and planned progress
  4. program’s strategy and its achievements

The MTR methodology included document review, self-assessment by program staff, interviews with program leadership and staff, and an external peer review.

Unclear plan for assessing program outcomes

At this time there is no clear plan to assess whether the Challenge programs achieved their intended outcomes and the extent to which the NRC has successfully responded to the Minister's priority challenges. CSTIP must be evaluated overall at least once every 5 years, with the next evaluation by NRC OAE scheduled for FY 2028. This evaluation would not assess results of individual Challenge programs. There are no plans to conduct Challenge program-specific assessments at the end of the program.

Information availability


Sufficient information is available for day-to-day operations, but is lacking for strategic program planning and direction, as well as for overall performance monitoring. In some instances, key information is not being tracked at all, while in others, it is tracked at the program level but not aggregated and shared upwards.

Mixed perceptions on information availability

NPO and Challenge program staff were divided on whether available information is sufficient. Nearly all IIAs and program advisors said they have the information they need to do their jobs. Program directors were split, with some expressing concerns about data being unstructured and out of date, and others being generally satisfied. Those who felt data was not readily available did note they could access it through specific requests to NPO. NPO management considers the available information to be inadequate, due to an inability to aggregate CSTIP program-level information.

Externally, PAC members considered their access to information as only somewhat sufficient. Notably, they wanted more information on program and project progress, including clear baselines and expected timelines, by which to monitor performance and provide guidance.

Need to improve information availability, integration

At both the CSTIP and Challenge program level, the lack of an integrated data collection system inhibits the availability of information. With information spread out among a mix of independent forms, emails, and standalone databases, key stakeholders do not readily have the information they need to make data-informed decisions. For program management, this is particularly relevant for strategic planning.

While NPO consistently tracks G&C-related project information, the data is limited, consisting mainly of high-level descriptive statistics and immediate outcomes information (see appendix B). Information that is not collected or readily available includes:

  • CSTIP- and program-level process efficiency information, limiting continuous improvement and resource management
  • progress towards EDI targets, and GBA Plus considerations, limiting the assessment of impacts on different populations
  • NRC human resource and labour information, namely who is participating in the Challenge programs, limiting assessment of intra-NRC collaboration and actual costs to RCs
  • medium-to-long term outcomes information (e.g., TRL progression), limiting decisions on program direction and project portfolio management (e.g., project continuation, scale-up, or termination)

Ultimately, information availability could be strengthened by integrating CSTIP- and program-level tracking and reporting.

Recommendations and Management response and action plan (MRAP)

Supporting rationale and recommendations

Priorities, expectations and organizational culture:

  • The Challenge program model (herein “the model”) is designed to support transformative research that spans the TRL spectrum from disruptive or foundational research to knowledge mobilization and pre-commercialization.
  • The model is flexible by design, but the expectations are unclear for the type of research and the path for outcomes when a program ends. Research funded to date may not be generally characterized as transformative, high-risk and high-reward. There is insufficient structure for managing the advancement of project portfolios (i.e., project maturation and scale-up or termination), developing technology with industry or other partners, and defining successful, end-user adoption.
  • The model provides the NRC with a platform for its 14 RCs and external partners to collaborate on priority challenges for the benefits of Canadians. Most funding is for G&Cs, and RCs have been asked to contribute to an increasing number of programs using their base operational budget. VPs and DGs were critical about available resources in light of perceived conflicting priorities. In order to support an NRC-wide culture change toward collaborative and mission-oriented research, organizational priorities and expectations would need to be clarified.

Recommendation 1

The VP-BPS should further clarify expectations for the Challenge program model with respect to:

  1. advancing transformative, high-risk, high-reward research
  2. structuring the development of technology and policy solutions for end-useradoption
  3. prioritizing research centres participation and intra-NRC collaboration

Accountability, ownership and oversight:

  • The governance structure for the collaborative R&D initiatives has been effective overall. Individual Challenge programs are accountable for their actual R&D outcomes. However, the accountability and ownership of the Challenge program model overall is not clearly established. Given its novel and flexible nature, clarity is needed as to whether NPO, or another entity, is accountable for the model (i.e., entity responsible to assess the model and to make adjustments over time based on lessons learned, MTRs and evaluations).
  • Programs are answerable to various internal and external committees, which have been implemented differently across programs. Together, this has resulted in reporting burden and inconsistent expectations for what programs report, and how they respond, to oversight bodies. In particular, there is an opportunity to better define the roles and responsibilities of the PSC in relation to those of other committees and management's response to the MTR process.

Recommendation 2

The VP-BPS should clarify accountability and ownership for the overall CSTIP to ensure that the Challenge program model achieves its goals, including whether oversight committees could be consolidated.

Program assessment:

  • There are no plans to conduct program-specific assessments beyond their mid-term. It has yet to be determined how the NRC will assess if individual programs achieved their intended results by the end of their life cycle.
  • There is a need to clarify who is accountable to report on the extent to which the NRC has successfully responded to the Minister's priority challenges for the benefits of Canadians.

Recommendation 3

The VP-BPS should clarify with senior management the need to assess whether individual Challenge programs have achieved their intended outcomes and, if so, how this will be done.

Roles and responsibilities for the Challenge function:

  • It is not consistently defined or understood how expert advice is used to “challenge” project selection towards effective portfolio management across project phases of programs. In practice, the Challenge function is diffused between IIAs, oversight committees (i.e., EPRC and PAC) and MTRs.
  • To ensure funded research aligns with desired program outcomes, it is necessary to clarify the "Challenge function" within the program model, specifically in regards to the role of the IIA.

Recommendation 4

The VP-BPS should clarify the:

  1. “Challenge function” within the Challenge program model, particularly as it relates to program development and direction, project selection and portfolio management
  2. roles and responsibilities of IIAs outside of their due diligence function for G&Cs

Sustainability and scalability:

  • NPO was established as a new independent function to enable the delivery of G&C funding to NRC collaborators while complying with the PTP. In the 5 years since, NPO has provided oversight, support, and G&C management to an increasing number of programs and funding agreements. To its credit, it has done so with a relatively flat budget and staff, and without an overarching framework to support consistency and efficiency.
  • In turn, NPO has been playing catch up. As of FY 2023 it was still implementing improvements recommended by a policy and process review from FY 2021. Elements of this review are included in the CSTIP Toolkit Roadmap, which NPO plans to implement in FY 2024. Moving forward, NPO resources will be further strained if the number of CSTIP programs and funding streams continues to grow.

Recommendation 5

The VP-BPS should examine the sufficiency of resources (human, financial and information technology) to complete ongoing continuous improvement initiatives to enable the success of the Challenge program model and CSTIP overall, and develop a plan for moving forward.

System integration and outcome-based management:

  • NPO’s tools and processes are spread across multiple systems and software platforms, resulting in inefficiencies. There are also data gaps and a need for better automation of data collection to improve timely analysis of key processes. Integrating key CSTIP processes for program and project development, G&C administration and client relationship management would support more effective and efficient administration of CSTIP, especially if the program continues to grow. This may provide a platform to integrate processes and facilitate information exchange between program staff, peer reviewers and G&C recipients.
  • There are gaps in the information that is critical for effective project portfolio management and the assessment of progress toward intended outcomes. Performance measurement tools and processes are not yet fully implemented and, importantly, outcome-focussed indicators are either not being collected or sufficiently aggregated. Roles and responsibilities for data collection are at times unclear as both NPO and program hosts have performance monitoring obligations. CSTIP requires a set of performance and outcome-focussed key indicators, including appropriate service standards, with clear accountability for data collection established. Integrating these indicators into a common system could improve outcome-based management.
  • NPO has made policy and process improvements by reviewing and streamlining specific tools and processes. As part of its CSTIP Toolkit Roadmap, NPO plans to further review its business tools and processesso that they can be easily adapted and integrated into 1 system. Systems integration will require the completion of ongoing streamlining initiatives, including the development of an overall framework that comprehensively maps CSTIP processes.

Recommendation 6

The VP-BPS should develop and implement an integrated system for key CSTIP processes, including a platform to facilitate interaction and information sharing among key stakeholders. This system should also integrate strengthened performance information to promote process effectiveness, efficiency and outcome-based management.

Management response and action plan

Recommendation 1: priorities, expectations and organizational culture

The VP-BPS should further clarify expectations for the Challenge program model with respect to:

  • advancing transformative, high-risk, high-reward research
  • structuring the development of technology and policy solutions for end-user adoption
  • prioritizing research centres participation and intra-NRC collaboration

Risk level: moderate

Management response Measure of achievements Proposed person(s) responsible Expected date(s) of completion

Response: accepted


1. Clarify and reinforce expectations around the Challenge program model with respect to a, b, and c through improved documentation.

2. Develop and deliver an internal communications plan.

  1. Documentation of decision.
  2. Proof of communication.
  • Melanie Cullins, DG NPBS IIO
  • Rachelle Bruton, Director NPO

1. July 2024

2. Dec 2024

Recommendation 2: accountability, ownership and oversight

The VP-BPS should clarify accountability and ownership for the overall CSTIP to ensure that the Challenge program model achieves its goals, including whether oversight committees could be consolidated.

Risk level: moderate

Management response Measure of achievements Proposed person(s) responsible Expected date(s) of completion

Response: accepted and linked to recommendation 4


1. Clarify accountability of overall CSTIP Challenge program model (duration, requirements, of a program).

2. Review of oversight committees and consider consolidations.

  1. Accountability is documented and communicated.
  2. Review completed and any changes approved by SEC.
  • Melanie Cullins, DG NPBS IIO
  • Rachelle Bruton, Director NPO

1. July 2024

2. November 2024

Recommendation 3: program assessment

The VP-BPS should clarify with senior management the need to assess whether individual Challenge programs have achieved their intended outcomes and, if so, how this will be done.

Risk level: moderate

Management response Measure of achievements Proposed person(s) responsible Expected date(s) of completion

Response: accepted


1. Clarify with SEC on the need to assess program outcomes at the end of the program.

2. If so, determine how this will be accomplished (i.e., who will do it).

  1. Record of discussion and decision to senior management.
  2. Documentation on path forward.
  • Melanie Cullins, DG NPBS IIO
  • Rachelle Bruton, Director NPO

March 2025

Recommendation 4: roles and responsibilities for the Challenge function

The VP-BPS should clarify the:

  1. “Challenge function” within the Challenge program model, particularly as it relates to program development and direction, project selection and portfolio management
  2. roles and responsibilities of IIAs outside of their due diligence function for G&Cs

Risk level: moderate

Management response Measure of achievements Proposed person(s) responsible Expected date(s) of completion

Response: accepted and linked to recommendation 2


1. Clarify “Challenge functions” through a roles and responsibilities review with the primary lens of “Challenge function”.

2. Clarify IIA roles and responsibilities over the lifecycle of a program.

  1. Responsibility assignment matrix identifying roles and responsibility for key decision points throughout program lifecycle, in particular for the Challenge function.
  2. Role of IIA documented.
  • Melanie Cullins, DG NPBS IIO
  • Rachelle Bruton, Director NPO

July 2024

Recommendation 5: sustainability and scalability

The VP-BPS should examine the sufficiency of resources (human, financial and information technology) to complete ongoing continuous improvement initiatives to enable the success of the Challenge program model and CSTIP overall, and develop a plan for moving forward.

Risk level: high
Management response Measure of achievements Proposed person(s) responsible Expected date(s) of completion

Response: accepted


1. Review of NPO resources mapped against workload requirements to inform a plan to either seek additional resources or right-size workload.

2. Further develop b-base program costing calculator to better reflect NRC operational needs.

  1. Review completed and decision on path foreword for NPO.
  2. Revised costing model for new Challenge programs.
  • Melanie Cullins, DG NPBS IIO
  • Rachelle Bruton, Director NPO

1. June 2024

2. September 2024

Recommendation 6: system integration and outcome-based management

The VP-BPS should develop and implement an integrated system for key CSTIP processes, including a platform to facilitate interaction and information sharing among key stakeholders. This system should also integrate strengthened performance information to promote process effectiveness, efficiency and outcome-based management.

Risk level: high

Management response Measure of achievements Proposed person(s) responsible Expected date(s) of completion

Response: accepted


1. Consultation with senior management to identify feasibility as part of an enterprise solution or other solution for establishment of a program delivery system (e.g., CRM).

2. Develop a roadmap and get ready for eventual CRM implementation (CSTIP toolkit, framework, and process mapping).

  1. Completed consultations.
  2. Provide CRM project lead with CSTIP process, case studies and project requirements to ensure key milestones and approvals tracked with CRM.
  • Melanie Cullins, DG NPBS IIO
  • Rachelle Bruton, Director NPO

1. June 2024

2. September 2024


Appendix A – detailed methods

Document review

Internal documents were reviewed to provide context, to assess the program management practices against the NRC and TB policies, and to complement other lines of evidence in assessing the effectiveness of program model and governance, processes and controls, including performance monitoring systems and reviews. Procedures were used to identify and review key business processes and procedures, tools and templates, guidelines and quality assurance approaches and methodologies.

Internal documents included CSTIP terms and conditions, strategic, operational and HR plans, program governance, model and profiles, process maps, conflict of interest forms, field guides, terms of reference and other key documents of various committees and panels, job descriptions, performance and oversight reporting material, management toolkits, and MTRs.

External documents were mainly reviewed as part of the comparative analysis study to provide context on Canadian and international mission-oriented innovation policies and initiatives.

Data review

Corporate data was sourced from nBoss and SAP systems, G&Cs and O&M funding tracking tools, and also included CSTIP administrative data, information on projects and funding recipients, program project tracking tools and program and project management workbooks, and financial and human resources data.

Targeted data analytics were used to assess and validate program management assertions. Performance monitoring data were also reviewed including key performance indicators at CSTIP and Challenge program level.

Available Challenge programs project close-out reports were also reviewed (n=44) to obtain feedback from funded external collaborators.

Internal and external interviews

Individual and group interviews were conducted to collect personal experiences, opinions and expert knowledge related to the effectiveness of the Challenge program model and governance, processes and controls and the use of information to inform decisions.

A total of 73 stakeholders were interviewed between November 2023 and March 2024. 76% were internal staff and 24% were external to the NRC:

  • 56 internal staff: senior management, VPs and DGs, NPO and NPBS management and staff, business development directors, Challenge program directors and managers, Cluster Support program directors, team leads and co-leads, and NRC IRAP ITAs
  • 17 external stakeholders: funded and unfunded external R&D collaborators and members of program advisory committees and other committees

Case study interviews are included in this total (29 people). Interviews and consultations carried out as part of the comparative analysis are not included in this total.

42% of interviewees were women

In addition to the representation across the NRC, positions supporting programs, categories and regional distribution of external stakeholders, efforts were made to include diversity of perspective, including gender balance.

Case studies

The case studies focused on documenting and understanding the experience of 3 Challenge programs as they evolved within CSTIP governance and processes:

  1. HTSN: RC-hosted; round 1 program
  2. PRCP: division-hosted; emergency response program
  3. AIP: division-hosted; round 2 program

These programs were selected to include one round 1 program, one round 2 program and a mix of sponsorship types (division- or RC-hosted programs). Planning consultations recommended the inclusion of the PRCP due to its unique nature.

Case studies allowed for in-depth investigation and the collection of context-specific evidence to support the broader evaluation findings. The case study approach included the review of program documents, data and interviews.

  • In addition to program-level information sources, a review of a non-random sample of projects (n=11) was completed to examine project-level documents and data.
  • Interviews were conducted with program management staff (n=4), research team leads and co-leads (n=6), business development directors (n=3), chairs of PACs and other committees (n=4), NRC IRAP ITAs (n=4) and with external funded (and unfunded) collaborators (n=12).

Collaborators and sample of projects were selected to include a range of project values (high, medium and low), funding and project types.

Comparative analysis

Selection process

Identifying comparator programs was difficult due to the unique collaboration and funding model of the NRC's Challenge programs. It was not possible to find comparator programs that included both government researchers and external organizations funded to work collaboratively on R&D projects.

From extensive online searches and multiple consultations with NPO and NPBS, 2 domestic and 2 international initiatives supporting mission-oriented research were selected based on goals, funding model, and program structure and other features. See details on comparators programs and rational for selection in the Profile section.

Rationale for selection of comparator programs
  • ICI Cleantech Challenges: to explore the strong similarities in federal policy drivers and requirements, program objectives, as well as oversight and delivery structures
  • EIP: to leverage the experiences and best practices of NRCan in managing transfer payments for R&D projects, as well as to examine similarities and differences in processes and governance structures
  • ARPA-E: to further investigate the strengths and weaknesses of the Defense Advanced Research Projects Agency (DARPA) program model (after which ARPA-E was designed), including lessons learned in achieving high-risk, high-reward research objectives
  • EIC Pathfinder: to explore the similarities and differences in oversight and delivery structures under the EIC program model (inspired by features of ARPA-E) and others
Data collection methods

Data collection and analysis consisted of a review of documents and data. There was some information on comparators based on publicly available documentation (program websites, past audit and evaluation reports). An interview guide and information request checklist were developed based on key indicators for comparison and other elements that could not be found in the public domain. Representatives from all 4 comparator programs were interviewed: 2 individual or group interviews (groups of 2 to 3 individuals) relative to each of the 4 comparator programs. Additional documents were shared by interviewees and follow-ups were completed to fill in information gaps and to validate information.

Appendix B – limitations and mitigation strategies

Although the evaluation encountered some challenges, methodological limitations were mitigated, where possible, through the use of multiple lines of evidence and the triangulation of data. This approach was taken in order to establish the reliability and validity of the findings and to ensure that conclusions and recommendations were based on objective and documented evidence. Details on limitations and their associated mitigation strategies are described below.

Lack of data on NRC contributions

It was not possible to quantify the contributions of individual RCs and the materiality of Challenge programs (i.e., total value of projects and programs) as the planned contributions in project proposals are not systematically or equally tracked across the organization. This prevented the effective assessment of the:

  • actual level of effort of RCs to Challenge programs
  • effects of the R&D collaborative model at changing collaborative behaviours between RCs over time
  • scalability (and ultimately the viability) of the R&D collaborative model given that RCs are solicited to contribute to an increasing number of Challenge programs over the same period of time, with a similar level of operating funds and priorities for mandated and revenue generating activities


To examine RC’s contributions to Challenge programs, OAE:

  • interviewed internal stakeholders, including case study interviewees who could comment on such contributions
  • reviewed program data and documentation that referenced RC involvement
  • used planned NRC contributions for the Profile section

Limited perspective from external stakeholders

Given the scope of this first evaluation of the CSTIP framework and processes, the evaluation team relied as much as possible on available data to avoid increasing the burden on external funding recipients. Concurrent to the evaluation, there were 4 MTRs involving interviews with external recipients and committee members. Funded collaborators were also asked to provide feedback on their experience of working with the NRC on processes and reporting requirements for recently completed projects.

The next CSTIP evaluations should include wider consultations with external stakeholders and take advantage of more extensive feedback and related performance information collected, reported and analyzed by NPO.

Mitigation: OAE and NPO closely worked together to:

  1. avoid overlap between evaluation case study interviews and MTR interviews by sharing contact lists and questionnaires
  2. incorporate findings from MTR reports when available (n=4 reports)
  3. include external feedback from close-out reports compiled by NPO (n=44 reports)

Variability and imbalance of information in comparative analysis

Information gaps are to be expected in a comparative analysis. Information and interviewee availability varied across comparator programs. Ultimately for 1 comparator was the consultant able to reached the targeted number and categories of key informants (i.e., program support and oversight function and program direction).

Given the scope of this evaluation, which looked in detail at processes and controls, there was an imbalance of information between CSTIP and the comparator programs. As a result, this analysis was intended to primary to compare frameworks and models for funding mission-oriented research, rather than specific processes and controls. Finally, the unique nature of CSTIP (with the NRC as both research funder and performer), limited the identification of best practices and lessons learned, as the comparators fund perform research.

Mitigation: The evaluation team reviewed publicly available documents and data, then followed up with comparator representatives by e-mail to fill information gaps. When some categories of key informants were unavailable, available interviewees provided valuable information on other functions and identified some supplementary documentations. Finally, for some analyses, a mix of quantitative and qualitative information sources was used to provide proxy indicators that can be put into perspective with those of CSTIP, and draw meaningful comparisons.

Appendix C – supplementary information

Summary table of KPIs collected and reported by NPO

Program or project aspect Specific KPIs Data collected Data reported
Enhancing collaboration

Number of collaborators, collaborator type, organization, size


Number of peer reviewed publications, citation scores


Invention disclosures, patent applications, number of technologies developed

Highly qualified personnel (HQP)

Count of students, technicians and other HQP



G&C recipient satisfaction


Perceived impact

Likelihood of commercialization, other impacts


Advancing breakthrough research

TRL progression (self-reported)


Business growth

Industry collaborators reporting growth




Administrative cost ratio




Percentage of new hires from underrepresented groups



EDI – external collaborators

Diversity of researchers participating in projects