Evaluation of the NRC’s Industrial Research Assistance Program

Abbreviations and acronyms

AGS

Accelerated growth service

ARP

Accelerated review process

CAGR

Compound annual growth rate

CEA

Client Engagement Advisor

CIIP

Canadian International Innovation Program

CTO

Contribution to organization

EDI

Equity, diversity and inclusion

FY

Fiscal year

GBA Plus

Gender-based analysis plus

IAP

Innovation Assistance Program

ICT

Information and communication technology

IIP

IRAP innovation portal

IRAP

Industrial Research Assistance Program

ISC

Innovative Solutions Canada

ITA

Industrial Technology Advisor

LVC

Large value contribution

NRC

National Research Council Canada

OAE

Office of Audit and Evaluation

OGD

Other government department

PBCA

Partial benefit cost analysis

PDAC

Program Delivery Advisory Committee

PSES

Public service engagement survey

R&D

Research and Development

RCAO

Regional Contribution Agreement Officer

SIF

Strategic Innovation Fund

SLT

Senior leadership team

SME

Small and medium-sized enterprise

TAC

Technology Access Centre

Introduction

The evaluation of the National Research Council's Industrial Research Assistance Program (NRC IRAP) covered the period of FY 2017-18 to FY 2021-22. It assessed NRC IRAP's reach, effectiveness, efficiency and relevance. Data collection and analysis was conducted during FY 2021-22. The evaluation was carried out in accordance with the NRC's approved evaluation plan, Treasury Board Secretariat's Policy on Results (2016) and the requirements of the Financial Administration Act. The program was last evaluated in FY 2016-17.

A comprehensive mixed-methods approach was used in the evaluation, including data and document reviews, internal and external interviews, case studies, a survey, focus group discussions, journey mapping, an international comparison study and a partial benefit cost analysis (PBCA). This report provides an overview of the key findings, conclusions, and recommendations.

Evaluation approach

Approach

The evaluation was led by the NRC's Office of Audit and Evaluation (OAE). It applied a mixed-methods approach, incorporating both qualitative and quantitative data from several lines of evidence. This allowed for the triangulation of data across diverse lines of evidence and ensured that consequent evaluation findings were robust and reliable. A gender-based analysis plus lens was also applied in order, to assess how diverse groups of people may be experiencing or differentially impacted by the program.

Scope

The evaluation covered the period from FY 2017-18 to FY 2021-22. The evaluation considered all NRC IRAP activities and expenditures, except for its collaborative programs with Global Affairs Canada (i.e. CanExport and Canadian International Innovation Program (CIIP)) and the Innovation Assistance Program (IAP). IRAP International, though included in the evaluation to some extent, was not a primary focus.

Evaluation questions

• Reach: To what extent have the NRC IRAP's client engagement efforts been effective at reaching intended (i.e. high potential and innovative) clients?
• Effectiveness: To what extent has the NRC IRAP been able to contribute to SME growth through innovation and sustainable economic growth within Canada?
• Efficiency: To what extent has the NRC IRAP been designed and delivered in an effective and efficient manner?
• Relevance: To what extent is the NRC IRAP responsive to Government of Canada priorities and the need for innovation support among Canadian SMEs?

Methods

Lines of evidence that support the evaluation include:

• data review (5 years+ of administrative and performance data)
• document review
• Industrial Technology Advisor (ITA) survey (n=125 responses; response rate=41%)
• internal and external interviews (n=86 interviewees)
• partial benefit cost analysis (17 firms)
• ITA focus groups (5 focus groups, n=36 participants)
• contributions to organizations (CTO) case studies (4 organizations)
• international comparison study (4 comparator programs)
• journey mapping of unfunded clients (2 unfunded firms)

For more detailed information on the evaluation methodology, including limitations, refer to Appendix A .

Profile

NRC IRAP was established in 1947 and has a mandate to stimulate wealth creation in Canada through innovation. The program offers a suite of innovation support services, including business and technical advisory services, financial assistance and linkages to national and international research and development (R&D) opportunities. In so doing, it aims to accelerate growth among Canadian small and medium-sized enterprises (SMEs) in various science and technology fields.

NRC IRAP Overview

Source(s): Document review and data review

NRC IRAP is an innovation assistance program designed to help Canadian SMEs grow through innovation and technology

Established in 1947, the program is one of four flagship platforms through which the federal government has streamlined the delivery of innovation programming across Canada since 2018. The other three platforms are the Strategic Innovation Fund (SIF), Canadian Trade Commissioner Service, and Regional Development Agencies.

Mandate

Stimulate wealth creation in Canada through innovation.

Mission

Accelerate the growth of small and medium-sized enterprises (SMEs) by providing them with a comprehensive suite of innovative services and with funding.

Vision

Be the most impactful program of its kind in the world, where Canadian firms go first to transform their ideas into commercial success.

Strategic Objectives

• help small and medium-sized businesses in Canada develop and commercialize technologies
• collaborate with regional and national organizations on initiatives that support the development and commercialization of technologies by small and medium-sized businesses

NRC IRAP helps firms build their innovation capacities and take their ideas to market

The program supports SMEs through advisory services, financial assistance and fostering linkages to appropriate business and research and development (R&D) expertise in Canada and beyond. NRC IRAP clients currently stem from all sectors of the Canadian economy.

29% of clients come to the program from the information and communication technology (ICT) or digital economy sector, representing the highest concentration of clients in any sector.

NRC IRAP is delivered to clients by five regional teams. Distribution of clients who received advisory services and/or funding between 2018 and 2022:

• 10% Atlantic & Nunavut
• 19% Quebec
• 32% Ontario
• 19% Prairies and Northwest Territories
• 18% Pacific and Yukon

Program structure

Source(s): Document review and data review

The Vice-President (VP) of NRC IRAP reports to the NRC's President and has overall managerial accountability for the program. The NRC IRAP VP is responsible for making program-wide decisions, informed by advice from members of the senior leadership team (SLT). In addition to the NRC IRAP VP, the SLT is comprised of seven Directors General (DGs) and the executive director of the International Innovation Office. A Board of Advisors helps ensure the program continues to offer the support Canadian businesses need to innovate and grow. They offer perspectives and advice to guide the program's strategic direction and management. Advisory Board members are appointed by the NRC IRAP VP and represent business and innovation stakeholders from across Canada.

Figure 1 – Text version

This illustrates the National Research Council's Industrial Research Assistance Program's (NRC IRAP) structure in a three-tiered hierarchy.

At the top tier is the President of the National Research Council of Canada.

The second tier is the Vice-President of NRC IRAP and the NRC IRAP Advisory Board composed of businesses and innovation stakeholders, which advises the Vice-President of NRC IRAP.

The final tier is the Direction General: Division Services, Director General: Client Enablement, Five Director Generals: Regional Directors General, and the Executive Director: International Innovation Office overseeing NRC IRAP International. NRC IRAP International was not a primary focus of the evaluation.

Director General, Division Services

Oversees the planning and development of operational policies, tools, and procedures.

Director General, Client Enablement

Ensures that NRC IRAP's client base is sufficiently supported by program delivery and connected to appropriate resources.

Regional Directors General

Responsible for the implementation of the program in their respective locales. Industrial Technology Advisors (ITAs) in each region are the primary holders of relationships with clients and partners. NRC IRAP has a total of 267 ITAs located across the country.

NRC IRAP International ^{Footnote *}

Connects Canadian SMEs with the funding, advisory, export and innovation services they need to access new markets and global value chains.

Innovation support services

Source(s): Document review, interviews

Contribution funding

NRC IRAP contributes funding to eligible firms and organizations through three primary streams. Regular IRAP is the main funding stream supporting clients with awards between $50,000 and $1M. The accelerated review process (ARP) supports small projects below $50,000. Larger projects (i.e. $1M up to $10M) are supported through large value contributions (LVC). These together make up core IRAP.

White label services

NRC IRAP supports other government departments (OGDs) and non-government organizations (NGOs) that champion innovation among Canadian SMEs. The exact suite of services is still being evolved by the program. However, for the moment, it primarily involves conducting business and technical assessments of funding proposals submitted by SMEs and providing advice to support the development of new and existing innovation programs. Importantly, clients are charged a service fee to ensure the partial or full recovery of costs associated with service delivery.

Advisory services

The main mechanism through which NRC IRAP engages SMEs is through advisory services. Skilled ITAs manage relationships with firms and provide business and technical advice as well as critical referrals to R&D and business/innovation expertise. All firms that engage with the program receive advisory services but not all are subsequently funded.

International co-innovation support

NRC IRAP funds collaborative industrial research and innovation projects, with high potential for commercialization, between Canadian SMEs and international counterparts and/or partners. This is achieved through NRC IRAP's partnerships with innovation networks, primarily in Europe, Asia and (to a lesser extent) South America.

Linkages to Canadian expertise

NRC IRAP enhances linkages between SMEs and R&D services and expertise. Through programs such as Certificate (on hiatus), NRC Researcher Engagement and Contributions to Organizations (CTO) eligible firms are provided access to unique research facilities as well as to skilled experts.

Additional support

NRC IRAP provides support through numerous other temporary and ongoing initiatives. The most relevant for the present evaluation include:

• the Youth Employment Program (YEP) through which SMEs receive funding to hire qualified post-secondary graduates as interns
• the global talent stream and dedicated service channel through which firms are supported to access qualified international talent
• the Accelerated Growth Service (AGS) through which SMEs are supported to access key government services needed to expand their businesses

Financial resources

Source(s): Document review and data review

The federal government continued to increase its investments in NRC IRAP over the evaluation period. Budget 2021 proposed $500 million over five years, starting in FY 2021-22, to expand NRC IRAP support to more Canadian businesses. This was in addition to the $700M already committed as part of Budget 2018. As a result, NRC IRAP distributed at least $431 million more in contribution funding to Canadian SMEs and organizations in the present evaluation period than it did previously.

NRC IRAP disbursed more than $1.3 billion to clients between 2018 and 2022.

At the time of reporting, the total financial support provided by the program over the evaluation period was more than $1.3 billion. However, the actual total is expected to exceed $1.8 billion once disbursements slated for the third and fourth quarters of FY 2021-22 are tallied. Importantly, this figure does not include about $450 million in contribution funding that NRC IRAP distributed to clients through other non-core programming delivered during the same timeframe (e.g. through the Innovation Assistance Program).

Figure 2 – Text version

A horizontal line chart representing the NRC IRAP total expenditures by fiscal year, broken into Grants and Contributions, and operating expenditures between the years FY 2017-18 and FY 2021-22. Over the 5 year span, total expenditures increased from $277M to $686M (+148%), with grants and contributions increasing from $216M to $582M (+169%), and operating expenditures increasing from $61M to $104M (+71%).

Figure 3 – Text version

A horizontal line chart with a single line representing actual costs recovered through white label services for the NRC IRAP. The costs recovered remained relatively constant over the 5 year span, going from $2.1M in FY 2017-18 to $1.5M in FY 2021-22. FY 2021-22 is based on budgeted amounts because reporting on actuals had not been completed at the time of the evaluation.

Human resources

Source(s): Document and data review

NRC IRAP staff are classified into three main categories: management, administrative, and technical

There are a total of 454 full-time staff presently employed by the program. The majority (~70%) of the workforce occupy technical roles, while about 20% are in administrative positions and less than 10% are in management roles. This overall distribution remained relatively stable throughout the evaluation period. Technical positions are classified as Research Council Officers (RCO) and include ITAs and Client Engagement Advisors (CEAs) among other policy/program advisory roles. Administrative or corporate positions include Regional Contribution Agreement Officers (RCAOs) and other administrative support staff. Management includes the VP, DGs and Directors.

The NRC IRAP workforce grew; however, the number of full-time staff fell below program expectations

In response to program delivery expansions proposed as part of Budget 2018, the NRC IRAP committed to increasing its workforce by 99 full-time-equivalents (FTEs), including 51 ITAs. The program planned to have 50% of these staff in place by FY 2018-19 while the balance were to be engaged by the end of FY 2019-20. During the evaluation period, the overall number of FTEs increased from 382 at the beginning of FY 2017-18 to 454 at the end of FY 2020-21 (latest available data). However, this meant that only 72 of the targeted 99 full-time positions were filled. ITAs made up only 19 of the 72 new FTE positions compared to the target of 51.

Figure 4 – Text version

An overlapping horizontal bar chart showcasing workforce representation compared to labour market availability from historically underrepresented groups for the NRC IRAP. The groups represented include Women, Visible minorities, Persons with disabilities, and Indigenous persons. The graph indicates that the NRC IRAP has not yet met labour market availability for staff from the aforementioned groups (Women: -9%; Visible minorities: -5%; Persons with disabilities: -7%), with the exception being Indigenous peoples who account for 2% labour market availability.

Figure 5 – Text version

A horizontal line chart representing the change in the amount of women at the NRC IRAP who occupy management, and scientific and technical roles. The figure indicates that women occupy fewer positions than their counterparts, although the overall number of women in these roles increased from 30% (management) and 22% (scientific and technical) in FY 2017-18 to 41% (management) and 27% (scientific and technical) in FY 2021-22.

Operational context

Source(s): Document review, interviews and data review

NRC IRAP and its clients spent two years, during the five year evaluation period, navigating the COVID-19 pandemic

As with most of the world, NRC IRAP staff, clients and partners were operating within a context that was challenging and highly uncertain. SMEs have been particularly vulnerable to the impacts of the pandemic, as they have been affected by issues such as staff cutbacks, supply chain and market disruptions, cancelled contracts and risk aversion from investors. Program staff played pivotal roles in the implementation of several federal government pandemic-related temporary programs and services, and provided over 3,000 pandemic specific advisory services to SMEs within the first year alone. In addition, the program delivered the (now closed) emergency wage subsidy Innovation Assistance Program (IAP) and the conduct of technical assessments for Innovative Solutions Canada (ISC) COVID-19 Testing and Challenge streams. These activities added to staff workloads and affected regular program delivery.

The pandemic-induced switch to a remote work environment brought with it some challenges

For SME clients, challenges included difficulties accessing innovation testing facilities and expertise, work stoppages particularly on international collaboration projects and reduced availability of funding for R&D in most sectors. For staff, adverse impacts included challenges with engaging new clients (See figure 18 in Appendix B) and monitoring project activities. At the onset of the pandemic, the program was reportedly able to leverage social capital that had been established through years of relationship building to execute these tasks. However, staff indicate that, as the pandemic progresses, this critical social capital is being rapidly erased by unprecedented levels of attrition within key stakeholder organizations.

The program is in the early stages of its GBA Plus integration journey

The program has been making deliberate strides towards the full integration of GBA Plus considerations into its operations for several years. During the evaluation period, its efforts have been primarily in alignment with the federal Women Entrepreneurship Strategy outlined in Budget 2018. The universal goal of the strategy is to improve the participation of underrepresented groups, including women entrepreneurs, in the innovation economy. In addition to implementing an inclusiveness strategy to support achievement of this goal, NRC IRAP committed to collecting more robust client equity, diversity and inclusion (EDI) data, and establishing a baseline for the participation of women and other underrepresented groups in the program by the end of FY 2019-20.

Client EDI data collection activities were launched recently and are still ongoing

NRC IRAP launched client-related EDI data collection activities in FY 2020-21. Data were actively being collected during the conduct of the evaluation and at the time of reporting were available for 47%, or 3,688 out of 7,915 clients that were funded in the evaluation period. The program did not have data available for unfunded clients. This is due to the fact that the current phase of data collection is focused on gathering data through the program portal to which, for the most part, only presently funded clients have access. Furthermore, EDI data were being provided by clients on a voluntary basis. Consequently, the evaluation was only able to partially assess program performance on GBA Plus based on available information.

Reach

NRC IRAP is reaching high-potential and innovative clients as intended. Client selection processes meet the needs of the program. The introduction of Client Engagement Advisors (CEAs) and the continued implementation of the portfolio model positively impacted the quality of clients reached during the evaluation period. The program placed more youth, and specifically more female youth, than ever before on client projects. The reach of business and technical assessment services provided by ITAs to government partners also continued to expand, as did the volume of assessments conducted. Overall, NRC IRAP reached more SME clients than it did in the previous evaluation period. Strategies to identify and reach clients from historically underrepresented groups are still in the process of being operationalized. The program is making strides towards collecting more robust EDI data on funded clients; however, the data are currently not available for the majority of unfunded clients. As a result, gender-based analysis of the program's reach could not be fully assessed as part of the evaluation.

Reach among intended clients increased

Source(s): Document and data review

The total program reach increased between the two evaluation periods

The total reach, as measured by the total number of unique firms receiving support from NRC IRAP (advisory services and/or funding), increased by about 20% from 18,732 to 22,458 between the two evaluation periods.

The sector distribution of clients were similar between the evaluation periods of FY 2018-2022 and FY 2013-2017.

Annual targets for total reach were achieved

In FY 2017-18, the program set an annual reach target of 8,000 SMEs. This was revised to 10,000 SMEs per year as part of program commitments following Budget 2021 and a target achievement date of March 31, 2022 was established. The program exceeded annual targets except for FY 2021-22.

Figure 6 – Text version

An overlapping vertical bar chart representing total reach targets of clients who received advisory services (Including IAP support) and funding from FY 2017-18 to FY 2021-22. The total client reach targets were exceeded from FY 2017-18 to FY 2020-21, but fell short by 922 its newly set target in FY 2021-22.

More clients received financial support

Source(s): Document and data review

The number of funded firms increased slightly since the last evaluation

The vast majority of clients during the evaluation period received funding for regular IRAP, Youth and ARP projects. Only 40 clients in the evaluation period (compared to the targeted 44 per year) had an LVC project. However, it should be noted that NRC IRAP made a decision to delay the ramp up of the LVC program in FY 2020-21 due to the pandemic.

Since the previous evaluation, the total number of unique funded clients increased by 4%, from 7,456 (during the FY 2013-2017 period) to 7,717 (during the FY 2018-2022 period). In addition, the average annual number funded firms increased by 2% between periods, from 3,295 firms per year on to 3,362 firms per year.

The FY 2021-22 program target for funded clients was met

An annual reach target of 3,500 funded firms was established in FY 2020-21 for achievement by the end of FY 2021-22. At the time of analysis, the program had exceeded its target, having recorded a reach of 3,657 funded firms. Prior to this, a target was only set for overall reach (i.e. unfunded/advisory service only and funded clients).

Figure 7 – Text version

A vertical bar chart representing the number of funded firms by the NRC IRAP from FY 2017-18 to FY 2021-22. Yearly targets were not set for FY 2017-18 to FY 2020-21. A target of 3,500 funded firms was set for FY 2021-22, which the NRC IRAP exceeded by 157 funded firms.

Improved reach of underrepresented client groups

Source(s): Document and data review

NRC IRAP funded proportionally more SMEs owned by individuals from historically underrepresented groups than were operating in the Canadian economy

NRC IRAP data indicate that, among the clients funded during the evaluation period, 28% are owned or led by women, another approximately 23% are owned or led by individuals identifying as Visible minorities, about 2% by Indigenous peoples and about 1% by Persons with disabilities.

Available national data (i.e. 2017 Statistics Canada information) on business ownership specifically, indicate that NRC IRAP's client population has a higher proportional representation of underrepresented groups than the general SME population in Canada (See Figure 9).

Staff are uncertain about the implementation of the EDI strategies

More than half of NRC IRAP staff, who were consulted as part of the evaluation, expressed a desire to continue to improve program outreach among underrepresented groups. However, there was a genuine lack of awareness about how they could operationalize the program inclusiveness strategy in the execution of their duties. Staff were nonetheless able to propose some recommendations for consideration by management:

• expand the number and diversity of ITAs in order to improve the diversity of reach – present workforce representation for women and Visible minorities is below labour market availability (See Figure 17 in Appendix B);
• collect client EDI data more routinely; and
• design and market more programs specifically targeted to underrepresented groups to improve appeal/awareness of the program.

Figure 8 – Text version

An overlapping horizontal bar chart representing the proportion of business owners that are from underrepresented groups among NRC IRAP clients. The proportion of business owners from underrepresented groups among NRC IRAP clients was found to be higher than that among all Canadian SMEs: Women (+2.0%), Visible minorities (+5.3%), Indigenous peoples (+0.2%), and Persons with disabilities (+0.3%).

NRC IRAP recorded more unfunded clients

Source(s): Data review, interviews

The number of unfunded firms recorded by the program increased between the two evaluation periods

Unfunded clients are typically SMEs that are not yet ready to implement an NRC IRAP-funded project but have received, or are in the process of receiving advisory services from ITAs or other third-party advisory service providers engaged by the program.

Since the last evaluation, the documented number of unique unfunded firms that had received advisory services increased by 21%, from 11,276 to 13,699.

Firms and types of advice are concentrated in specific areas

The majority of unfunded firms were from the Ontario, Quebec and Pacific regions (where the majority of potential clients are located) and from the ICT (digital economy), manufacturing & materials, and agriculture & food sectors. Technical advice was the top service provided followed by technical information and technical partnership. Business information was the top type of business advice followed by business referrals and business partnership advice.

Figure 9 – Text version

A horizontal line chart representing the annual number of unfunded NRC IRAP clients from FY 2017-18 to FY 2021-22. The annual number of unfunded clients remained relatively stable during the 5 year span, with the exception of a spike of 6,294 unfunded clients in FY 2020-21.

Youth reach increased

Source(s): Data and document review

Youth reach almost doubled between the two evaluation periods

Reflecting the increase in available funding, the number of youth participating in the YEP increased from 4,878 in the previous evaluation period to about 7,232, representing about a 50% increase in the number of unique youth participating in the program.

Most youth participants came from Ontario, Quebec and Pacific. The vast majority of youth placements were in the ICT sector, followed by manufacturing and materials, and energy and resources.

The program is reaching more female youth

In response to federal government commitments to support the career advancement of women in STEM, NRC IRAP increased the proportion of female youth engaged in the program over the last decade. The growth in number of female youth reached between FY 2013-14 (earliest available data) and FY 2021-22 was about 262% while that of male youth was about 57%.

The proportion of youth in the program who are female increased compared to the previous evaluation period (38% in FY 2018-2022 vs 32% in FY 2014-2017). During the evaluation period, there was also a steady increase in the proportion of female youth annually engaged by the program from 31% in FY 2017-18 to 54% in FY 2021-22.

Figure 10 – Text version

A horizontal segmented line chart representing the amount of youth from underrepresented groups who took part in the Youth Employment Program, part of the NRC IRAP. The proportion of youth were compared from the two 4-year periods, 2014-2017 and 2018-2021. The figure shows that there were increases in every underrepresented youth group, with the largest increases in proportion coming from Women (+6%) and Visible minorities (other than Indigenous peoples]) (+5%).

More other government departments were supported

Source(s): Document review

NRC IRAP has supported more other government departments

The number of partner programs that have been seeking NRC IRAP support for technical assessments has been increasing gradually year over year. To date, a total of 30+ partner programs have benefited from NRC IRAP support. These have included programs run by OGDs, provincial governments, regional development agencies and non-profits. Since the last evaluation period, the number of active other government department clients has increased from 8 in FY 2017-18 to 21 in FY 2021-22.

The number of technical assessments conducted has also increased

The number of partners seeking support has increased, as has the volume of assessments being conducted. Between FY 2012-13 and FY 2021-22, more than 9,700 assessments were conducted. The average annual number of assessments completed more than doubled in the current evaluation period (i.e. 1,402 per year) compared to the previous evaluation period (i.e. 547 per year).

In FY 2020-21 the number of assessments spiked to 2,730 in response to an increase in demand for support from OGDs running pandemic related programs. This was about 1,300 more than the highest number of assessments the program ever previously recorded (i.e. 1,394 in FY 2017-18).

Figure 11 – Text version

A horizontal line chart representing the amount of technical assessments performed by the NRC IRAP for other government departments (OGDs) from FY 2012-13 to FY 2021-22. There have been significantly more technical assessments performed during 2018-21 period than there was from 2013-17 period, with the largest increase being from FY 2019-20 (769) to FY 2020-21 (2,730). FY 2021-22 was not completed at the time of analysis with 966 assessments completed for 21 different active OGD partners.

Improved quality of program reach

Source(s): Interviews, data and document review

Client engagement advisors (CEAs) have had a positive impact on the quality of clients entering the program

Overall perceptions of CEAs are positive. CEAs are noted to have helped improve the program's reach by contributing to focusing resources on high potential firms. CEAs seek out and screen SME clients before passing leads on to ITAs. In doing so, they reportedly play an instrumental role in easing ITA workloads, allowing ITAs to continue working on delivering the program to existing clients while simultaneously engaging with additional innovative and high-potential clients.

The portfolio model contributes to the expansion of program reach to higher growth clients

The majority of ITAs indicated that the portfolio model contributes to focusing resources on high-potential firms. Once client leads have been referred to ITAs, they, along with their regional directors, work together to identify high-potential portfolio clients, a process which can involve multiple years of relationship building, advising, funding and meeting milestones. The selection process under the portfolio model is more refined and interviewees state that it works well to identify high-potential clients. This is also evidenced by the 96% increase in the proportion of firms who were considered to be high-potential clients over the course of the two evaluation periods (i.e. 15% in FY 2012-13 to >40% in FY 2021-22).

Efforts to diversify program reach improved

Source(s): Document review, data review, case studies and interviewees

The SONAR4d pilot yielded positive results for the diversification of program reach

During the evaluation period, NRC IRAP piloted a new platform, known as SONAR4d, that combines data from multiple organizational sources into one interface to help ITAs and CEAs better identify, research and select SME clients. Though still limited in terms of deployment in Quebec and Ontario, the SONAR4d pilot was perceived to be successful at diversifying program reach. The platform has reduced the time and effort needed to identify potential clients of all profiles. It therefore has the potential to be another valuable pipeline of new client leads in an ITA's toolbox. There are currently plans in place to expand the use of the platform to other regions. However, some staff thought that the feasibility of these plans may be hindered by the considerable variations in provincial rules governing the sharing of data between government agencies, something essential to the success of SONAR4d.

The portfolio model may not be compatible with efforts to expand program reach to new clients

Although the majority of ITAs indicated that the portfolio model contributed to focusing resources on high-potential firms, some found the program's approach to implementing the model problematic. This is because, in their view, the approach ascribes higher value to proven high impact firms over potential high impact firms, thus inadvertently nudging them towards making more conservative client selections. Some ITA perceptions that their annual performance reviews were linked to the performance of their portfolio clients were thought by some in senior management to exacerbate this issue. Program data appeared to support the prevalence of these perceptions, as the number of new funded clients fluctuated during the evaluation period (see figure 18 in Appendix B).

Contributions to Organizations (CTO) Case Study

Efforts to increase the diversity of program reach

The Canadian Executive Service Organization's (CESO) mission is to empower people and drive economic development through sharing expertise. NRC IRAP first funded CESO in 2020 for a pilot project to support Indigenous businesses and communities. Participating SMEs received 50 hours of mentorship mainly for guidance on how to develop their innovative products with coaching on developing strategic, operational and HR plans, and other early stage needs. CESO stated that NRC IRAP support is:

• integral as it allows the organization to continue offering its mentorship services
• a critical success factor for Indigenous SMEs as it helps improve general capacities and accelerates the development of their innovation projects

CESO support is expected to increase the proportion of Indigenous businesses supported by NRC IRAP and to ultimately lead to significant impacts for jobs, revenue and wealth creation within Indigenous communities.

"We felt so good about going this path with CESO which ended with tangible documents. We have no certainty about the future success of our business, but we feel at least that we have the right approach and tools to go forward"

Challenges impacting program reach

Source(s): Interviews, data review, document review

Staff involved in outreach activities feel under-resourced

Staff perceived that their abilities to effectively reach more clients were impacted by a lack of sufficient resources (human, technical and financial). In urban centers, the sheer volume of clients seeking support make it difficult for small teams to properly execute their duties. On the other hand, staff in less-densely populated areas reportedly have to develop outreach strategies in order to reach new and diverse clients. However, they cited a lack resources to address multiple government priorities, and a potential opportunity for the program to review resource allocations to ensure the right balance is being achieved based on geography and the types of outreach services required.

Staff lack some information required to make decisions

The program made great strides over the evaluation period to improve performance monitoring of reach data. However, there was still some opportunity for further improvements, particularly related to how performance monitoring data are communicated to staff with decision-making authority. For example, although some initial reach targets were determined, they were not systematically included in monthly reports, annual datasheets and operational plans used by regional staff to plan their outreach activities. In addition, although data quality has been improving, a large proportion of unfunded client sector data are still unknown (e.g. about 30% are tagged as "N/A" or "Other"). This limits the program's ability to accurately track the sector distribution of its reach of unfunded clients and thus make better informed decisions about resource allocations.

Some aspects of client selection processes hinder reach expansion

As previously mentioned, NRC IRAP made many improvements to the client selection process over the course of the evaluation period. However, the selection process, for large value contribution (LVC) clients in particular, was still perceived by most interviewees to be onerous and a hindrance to the expansion of reach among that segment of clients. Although senior staff indicate that the process is presently not any more bureaucratic than similar government programs, there may be an opportunity to further review and refine project development and approval processes.

Interviewees additionally reported that regular program rules around client eligibility can hinder expanded reach within Indigenous communities. This is because many Indigenous SMEs tend to use a different legal structure from that which government typically deems as eligible for funding. Even though this was viewed as a barrier that was beyond the entire control of the program, interviewees still thought there was an opportunity for the program to push to evolve the rules.

Effectiveness

NRC IRAP has been successful in supporting innovation and sustainable economic growth within Canada. The program supported SME clients to consistently achieve the majority of their project objectives and register marked growth post program engagement. The resulting growth has facilitated the expansion of supply chains, supported the creation of jobs and the generation of new capital within and beyond Canada. NRC IRAP clients also made contributions to Canadian efforts to combat the COVID-19 pandemic. The program contributed to this by providing advisory support and funding to assist clients in the development of new innovations, scaling up production of existing products and the adaptation of facilities to meet pandemic-related needs. The program has stimulated growth in the economy and supported the creation of wealth in Canada.

Increased SME access to an ecosystem of support

Source(s): Document review, case studies and interviewees

NRC IRAP facilitated increased access of SMEs to an ecosystem of support

In addition to providing support through its regular program, NRC IRAP expanded its service offerings to include large value contributions (LVC) and more international opportunities. SMEs were also supported by ITAs to access external funding (private, OGD and international), make connections with strategic partners and test their innovations. This support was made available through NRC research centres, contributions to organizations (CTO), the accelerated growth service (AGS), IRAP International and other relationships that the program had established with key actors in the Canadian and global innovation support ecosystem.

The complementarity of NRC IRAP service offerings with externally available support makes the program more effective

NRC IRAP was perceived by clients to be instrumental in supporting the timely achievement of their respective innovation objectives. Importantly, working with the program de-risked SME clients, making them more attractive to other funders and innovation service providers. Service offerings were reportedly complementary to available external support, which together generally provided the robust and all-encompassing support that firms needed to be successful.

Contributions to Organizations (CTO) Case Study

NRC IRAP supported SMEs to access a network of innovation expertise and technology testing facilities

Tech-Access Canada has a network of 60 Technology Access Centres (TACs) across Canada and has been funded by NRC IRAP since 2016. The funding gives SMEs 20 hours of interactive visits which could include access to equipment, facilities, expertise or advice. The visits focus on getting to know the company, scoping their challenges, demystifying the innovation ecosystem, validating a technical solution they are exploring, giving them objective advice or making suggestions for future activities.

The result of a visit is often a plan that describes next steps which might be another project with a TAC or something the SME does on their own. For SMEs that are at a very early stage, these visits are low risk with a lot of potential. They help them move towards their goals and ultimately create a wider pool of SMEs who might be ready for direct IRAP support.

Enhanced client innovation capacity and productivity

Source(s): Interviews, data and document review

More R&D clients than targeted derive positive benefits by working with NRC IRAP

The proportion of surveyed clients who reported an increase in jobs, sales, R&D expenditures or other positive benefits as a result of services received increased slightly. According to data reported by NRC IRAP, in the FY 2017-18 through to FY 2020-21 NRC Departmental Results Reports, on average, 89% of all R&D clients reported some positive benefits of working with the program. This is compared to an average of 82% reported between FY 2013-14 (earliest available data) and FY 2016-17, and the NRC target of 86% for the current evaluation period.

Clients value the linkages the program provides to NRC experts and facilities

Clients indicate that NRC researcher interactions and the certificate program, in addition to NRC IRAP funding and advisory services, were critical contributors to improvements in SMEs' innovation capacities and the commercialization of their ideas. The certificate program was a pilot and the program did not obtain funds beyond the pilot phase. However, the closure of the certificate program was a source of disappointment for clients and NRC researchers alike. It was perceived to have offered great value to all SMEs, but particularly to early stage firms, on their journeys toward increased innovation capacity, productivity and profitability.

Figure 12 – Text version

A figure representing the change in proportion of benefits reported by clients funded by the NRC IRAP between two five periods: 2013-2017 and 2018-2021. The figure shows that there was a significant increase in all impact categories, which include: Increased technical capability (+40%), Increased competitiveness (+39%), Enhanced ability to conduct R&D (+39%), Increased productivity (+30%), Improved network (+29%) and Increased business capability (+28%).

Increased business growth

Source(s): Data and document review

The program supported SMEs to grow; certain profiles of firms grew at a faster rate than others

NRC IRAP clients exceeded compound annual growth rate (CAGR) targets for both revenue and employment for the evaluation period. Portfolio clients registered higher growth than non-portfolio clients (i.e. 36% versus 25% for revenue growth and 15% versus 11% for employment growth). Although there was no difference in revenue growth rates, NRC IRAP International program clients grew slightly more than regular program clients in terms of employment (i.e. 17% versus 14%).

Firms in the health and life sciences sector grew the most in both revenue (65% CAGR) and employment (22% CAGR). LVC clients also registered high growth rates for both revenue (45%) and employment (21%). However, the LVC clients sample size was small (~25).

Figure 13 – Text version

An overlapping horizontal bar chart representing the compound annual growth rate of firms in revenue and employment of NRC IRAP clients for the evaluation period. The figure shows that the compound annual growth rate target for both revenue and employment were exceeded for the evaluation period by 12% and 3% respectively.

Enhanced creation of wealth through innovation

Source(s): Interviews, data and document review

The program is making strong contributions to the sustainable creation of wealth in Canada

Stakeholders perceive the program to be making strong contributions to the sustainable creation of wealth in Canada. NRC IRAP is viewed as an accelerator that helps catalyze the growth of young companies. In so doing, the program supports the creation and expansion of new supply chains that feed SME innovation and facilitate wider benefits for other actors operating within the Canadian economy.

The majority of NRC IRAP funded SME clients are profitable following project completion

NRC IRAP clients are required to provide the program with snapshot data (including profit and loss information) for five years after the completion of their funded project. Each fiscal year since FY 2014-15, a larger proportion of NRC IRAP funded SME clients reported making a profit. During the evaluation period, the percentage of clients reporting a profit each fiscal year increased from 48% in FY 2017-18 to 64% in FY 2020-21 (latest available data).

NRC IRAP supported about 62,000 jobs during the evaluation period

Program funding was used to support 62,076 high quality jobs within client firms during the evaluation period. This included job placements for 7,232 youth. At the time of reporting, 5,823 of the youth placements had been completed and 4,121 (i.e. 71%) of youth continued to be employed post program engagement. Overall, the program supported 9,512 more jobs than in the previous evaluation period when a total of 52,564 jobs were recorded.

Figure 14 – Text version

A vertical overlapping bar chart representing the amount of jobs supported as well as the target amount of jobs supported by the NRC IRAP from FY 2017-18 to FY 2021-22. The figure shows that the targets were exceeded each year. In FY 2021-22, 16,160 jobs were supported exceeding the target (9,539 jobs) by 6,621 jobs.

Impacts on the Canadian economy are positive

Source(s): PBCA study, interviews, data review, survey, focus group discussions

The net benefit of the program to the Canadian economy is greater than the program costs

PBCA net present values and benefit/cost ratios have increased from the FY 2016-17 PBCA (even if 2016 dollars are inflated to 2021).The results of the FY 2021-22 PBCA show that even under the most conservative assumptions and highest discount rate, the net benefits produced by the NRC IRAP in the Canadian economy are 2.8 times greater than the program costs. This is a slight improvement over the FY 2016-17 PBCA results that showed net benefits were about 2.5 times greater than program costs.

• FY 2021-22 PBCA net present value: $16 billion
• FY 2021-22 PBCA benefit-cost ratio: 5.34 : 1

Under more "reasonable" assumptions (i.e. taking the average value of the lower and upper bounds, at a 5% discount rate) the net benefit of NRC IRAP to the Canadian economy is calculated to be 5.3 times or at least $16 billion greater than the program's costs. This is higher, in contrast to the findings of the FY 2016-17 PBCA that showed that net benefits of the program to the economy were 4.9 times or at least $10 billion higher than the program's cost.

Current NRC IRAP clients are more high-growth and innovative than they were in FY 2016-17

The positive results demonstrate that the strength of NRC IRAP client selection processes, funding decisions, advisory services and general support continue to be critically important to the viability and success of clients and the program as a whole. In fact, IRAP is working with more high-growth, innovative firms since FY 2016-17:

• three firms in FY 2016-17 with lower bound net benefits >$1B; in FY 2021-22 there are five
• two firms in FY 2016-17 with lower bound net benefits <$1B but >$500M; in FY 2021-22 there are seven
• funding and ongoing ITA advice is considered critical to getting to market faster, de-risking research, increasing innovation capacity, and keeping businesses in Canada

Benefits to the Canadian economy improved

Source(s): PBCA study

Cost/benefit analysis: case study examples

Source(s): PBCA study

Kraken Robotics is a marine technology company engaged in the design, development and marketing of advanced sonar and laser sensors and systems and subsea power solutions for Unmanned Underwater Vehicles for military and commercial applications. NRC IRAP supported the firm from a pre-revenue start up to becoming a publicly listed company.

Arctic Spas manufactures hot tubs engineered for harsh winter climates. Their products are manufactured in Canada and sold worldwide. They have developed a niche due to their products' energy efficiency, as well as advanced technological applications. NRC IRAP projects have resulted in practical applications, increased employment and sales.

Questrade is an independent, online investment brokerage that offers Canadian citizens an alternative to trading and investing with big banks. Trading for a variety of asset classes can be done via a desktop or mobile app. NRC IRAP funding has supported the development of technology. The most recent project, starting in 2020, has helped with the development of an online mortgage platform. The firm was also part of the previous PBCA study in FY 2016-17.

Pavemetrics specializes in 3D vision systems for the automated inspection of transportation infrastructure. Their systems are used in 45 countries around the world to automatically inspect roads, runways, railways and tunnels. IRAP projects have helped the company in its product research and development.

Innovative Automation designs and develops automated assembly machines for the automotive industry, consumer goods and medical devices. IRAP funding contributed to product development in addition to services provided, increasing firm revenues and creating jobs. A patent application was filed for its RoboTape automatic tape system.

Semios helps growers use technology to sustainably produce more food. Through their farming platform, they provide crop management tools for specialty crops. Using a wireless network, machine learning and big data analytics, they help growers manage pests and disease, and weather-related risks in real-time. Semios recently acquired Agworld, Altrac and Centricity to further its commitment to simplifying the growers' experience. IRAP projects have supported research and development.

Advisory services are as impactful as financial support

Source(s): Advisory services client journey mapping

Building relationships leads to multiple impacts

The subject of this journey mapping exercise has been in business for over 9 years and sells an environmentally friendly culvert system. With the aid of an ITA, the firm made a connection with the local college and completed a collaborative applied research project.

The project validated the firm's technology and filled an information gap on impacts to fish habitat and behaviours at stream crossings. As a result, sales increased with installations made across Canada and the United States. A demonstration project organized with the City of Lethbridge showcased the technology's environmental benefits and cost-saving features. The city was able to purchase the culvert with considerable savings and install it with no streambed disruption. In addition to preserving the habitat quality for fish, the research project also kick-started the growth of the applied research program at the college, which now ranks on Canada's Top 50 research colleges, and has led to an ongoing relationship between the college and the City of Lethbridge.

The firm maintained their relationship with the ITA who introduced them to at least five other organizations, and continued to provide advice and referrals for business and technical information. One referral led to a distribution agreement in Ontario and Quebec, and the culvert system was sold for use in a wind farm, in an agricultural application, and in the forest industry.

2015

• ITA connections

2016

• technology validated

2017

• increased commercial success
• preserved habitat quality

2018+

• ITA referrals
• continued growth

Clients are supporting the fight against COVID-19

Source(s): Document review and interviews

NRC IRAP is supporting the development of COVID-19 vaccines and therapeutics

The program has provided advisory services and more than $37 million (and plans to provide an additional $113 million over the next 3 years) in funding to support 13 clients working towards research and development of made-in-Canada vaccines and therapeutics for the COVID-19 pandemic.

When the R&D phase projects are completed, NRC IRAP will invite the companies who received funding to submit proposals to undertake a second round of projects to advance the most promising candidates to the next phase of clinical development.

NRC IRAP funded clients used their facilities and technologies to combat the pandemic

During the pandemic, NRC IRAP actively supported and assisted firms in the development of innovations needed in the fight against COVID-19.

As a result, multiple clients supported by the program were able to contribute to national efforts to combat the pandemic by producing personal protective equipment (PPE), disinfection and sanitization solutions and medical innovations for improved diagnostics, testing and treatment of COVID-19. In several cases, NRC IRAP clients scaled up production of existing products and/or adapted them to meet pandemic-related needs.

Advisory services helped clients fight against COVID-19

Source(s): Advisory services client journey mapping

Canada-made personal protective equipment (PPE)

The subject of this advisory services journey mapping exercise is a start-up that wanted to contribute to Canada's pandemic response. The firm's journey with the program started when they were referred to NRC IRAP by a contact. Once connected to the program, the firm worked with NRC IRAP to navigate regulatory hurdles in the development of medical devices.

With the advice and help from two ITAs, the firm achieved an ISO 13485 certification for quality management of medical devices. Upon receiving further advice from an ITA on how to navigate regulatory requirements, Health Canada approved the firm's product: the first N95 respirator manufactured in Canada. This was followed by approval from the National Institute of Occupational Health and Safety (NIOHS), which allowed the firm to expand their market into the United States. The firm continues to work with IRAP and is exploring further development and expansion of their products. They credit ITA advice for speeding up the commercialization process.

2020

• ITA advice
• ISO certification

August 2020

• product approved

2021

• expanding market

2022+

• positive future outlook

Opportunities to maximize outcomes exist

Source(s): Case studies, interviews, international comparison study

Strengthen connections within the ecosystem to more efficiently bridge the gap between R&D and commercialization

Clients and staff perceived the program to be more geared towards supporting firms in the early stages of their R&D journey. Some clients indicated that there was an opportunity for the program to improve support (including access to a network of supports) for SMEs seeking to bridge the gap between the prototyping and commercialization phases of their innovation processes. For example, it was suggested that the program encourage ITA engagement with broad sector partners (e.g. accelerator and incubator organizations) to optimize development of SMEs. In their view, this would help ensure their long-term success.

Although the program is already well-connected within the Canadian and international innovation support ecosystem, some additional lessons could be learned from international comparator programs which rely on their respective ecosystems to bridge this gap. Business Finland leverages services through their active participation in the Team Finland network advisory services to provide clients with advice on commercialization (e.g. concrete market entry). Following the provision of R&D financial support and information, Vinnova connects their clients with Business Sweden, whose strengths are to connect Swedish companies with international collaborators or investors, especially companies at the commercialization phase.

Engage more organizations to support service delivery

Stakeholders suggested that the NRC IRAP increase its outreach to CTO clients to ensure SME clients can be holistically supported to innovate/evolve their innovations. Case studies confirmed that CTOs supported by NRC IRAP are offering an important suite of services that are filling a gap for many SME clients by, for example, assisting SMEs with small projects, helping scope or solve a technical or business challenge and/or making suggestions for next steps. Many SMEs do not have the scientific expertise nor the larger scale processing equipment or pilot scale infrastructure needed to advance.

Supported CTOs included in the case studies deliver a wide range of services, from technical (e.g. access to technical expertise, mentorship, testing/trials, proofs of concept and access to equipment) to business supports (e.g. business planning, strategic planning, human resources management, work organization), and are typically oversubscribed.

To improve NRC IRAP's impact, organizations could be engaged to specifically focus on SMEs in sectors that require more support or that strongly support Canada's economic priorities (e.g. the environment), and better program promotion with underrepresented groups.

Efficiency

NRC IRAP has evolved over the past 75 years into a key player in the Canadian ecosystem for innovation support. The program is designed and delivered in a manner that is consistent with international comparator programs, and generally meets the needs of intended clients. The vast majority of clients are satisfied with their interactions with the program and with the services that are provided. However, some adjustments could be made to improve overall efficiency and to ensure that the program meets the specific needs of diverse groups.

NRC IRAP is effectively designed to meet client needs

Source(s): Interviews and document reviews, international comparison study

NRC IRAP is flexible, agile and capable of rapidly responding to the changing needs of clients

Staff indicate that the design of the program is fit-for-purpose (i.e. capable of meeting objectives and service standards). In their view, NRC IRAP was agile and generally responsive to the changing needs of clients throughout the evaluation period. This was particularly the case during the pandemic when the program was able to stand-up a fully operational temporary emergency wage subsidy program within a week of its announcement. In addition, the program conducted technical assessments for the ISC COVID-19 Testing and Challenge temporary program streams.

Subject expert teams that were set up in response to the pandemic were an important program change, and served the needs of clients well. These teams allowed the program to more efficiently distribute information on pandemic-related opportunities to clients, and to effectively reserve funds for projects that were deemed to be urgent in the context of the pandemic, all while simultaneously delivering the regular program.

Over the last five years, NRC IRAP effectively designed several new sub-programs in response to client needs, as identified in federal budget announcements. These included two new international programs, a Youth Green program, the AGS, the Certificate program, LVC and the emergency IAP. The program also provided OGDs with administrative support, business and technical assessment services and support in the design and launch of new programming (e.g. ISED with SIF, ISC Challenges and Superclusters, CanExport). It continues to be a key platform for the federal government to launch business innovation services and programs.

Client-centric adjustments were made to the design of the program

The NRC IRAP project development process is generally perceived by staff and clients to be effective and efficient (at least compared to other government processes). Improvements made to the core program over the evaluation period include:

• the introduction of new sub-programs (as previously mentioned) to respond to clients' needs and Government priorities;
• the establishment of the client enablement team to meet evolving program delivery requirements, including EDI and GBA Plus commitments;
• an ever-expanding and better-evolved suite of white label services;
• a streamlined project review and approval process (including a revised RCAO checklist); and
• refinements to the IRAP Innovation Portal (IIP) to, among other objectives, allow clients to have a better experience when developing and submitting project proposals.

Clients are satisfied with program delivery

Source(s): Interviews, data and document reviews

Clients are satisfied with NRC IRAP service delivery

Clients are satisfied with services they receive from NRC IRAP. Most clients were satisfied with the knowledge and professionalism of staff, and the overall level of support rendered.

• 91% of client firms had positive sentiments about the program
• 88% of CTO clients had positive sentiments about the services they received

Clients identifying as belonging to a historically under-represented group (i.e. women, Visible minorities, Indigenous peoples and Persons with disabilities) are just as satisfied (~92%) with their interactions with the program, and would recommend it to their professional networks.

"They have filled an important need for us. I am an engineer and neither I nor my partner are business people."

Technical assessment services are delivered in an effective manner

A review of white label services (specifically, the technical assessments) found that OGD clients perceive delivery to be in accordance with service standards outlined in their respective memoranda of understanding.

NRC IRAP was adept at communicating any significant changes in agreed upon service standards. OGDs reported being highly satisfied (90-95% of the time) with the overall technical assessment process and general quality of service received from NRC IRAP.

Reasons provided for high levels of satisfaction included the:

• NRC IRAP Partnership and Assessment team's professionalism
• effectiveness of communication
• flexibility of services offered
• program's commitment to honoring service standards
• unmatched combination of technical and business expertise possessed by the ITAs

Some services are not well integrated into program

Source(s): Interviews, international comparison study

ARP and Youth components are perceived to be more integrated than LVC and International programs

The integration of various components of the program into core IRAP has been improving over time. The ARP and Youth funding streams were considered to be the best integrated within the core program. The roll-out of the LVC funding stream has been progressing well and its integration has improved, although challenges in ensuring eligible clients' progression from regular IRAP to LVC projects persist.

IRAP International was viewed as the most challenging sub-program to integrate because of the fundamental differences in the delivery mechanisms (i.e. an ITA discretionary delivery model vs. a traditional call for proposals). Some ITAs within the regular program do not have positive views on the international project development process, and this hampers willingness to participate and integration with core programming.

The program introduced a three-year rotation program for regular ITAs to serve as International ITAs in an effort to increase buy-in for the program. Other solutions to improve integration are still needed but may be limited by the large variety of international partners and distinct processes that would need to be considered.

Stakeholders' levels of awareness of NRC IRAP services varies

External stakeholders had mixed levels of awareness of various components of the program (based on a small sample size ~30). Awareness of main funding streams (ARP, Youth, IRAP), advisory services, Certificate and researcher interaction programs and the ISC challenges were generally high among all stakeholders. Fewer stakeholders were familiar with IRAP International, LVC and talent acquisition programs i.e. Dedicated Service Channel; Global Talent Stream.

Staff job satisfaction is high but some HR issues persist

Source(s): Interviews, data and document reviews

NRC IRAP staff are generally satisfied with their jobs

Responses to the public service engagement survey (PSES) during the evaluation period indicate that staff at NRC IRAP are at least as satisfied or more satisfied with their jobs as staff in the broader NRC. According to the latest available information from FY 2020-21, about 90% indicated that they liked their job overall. No PSES was conducted in FY 2021-22.

Some segments of staff are overextended

Although data show improvements since FY 2017-18, the FY 2020-21 PSES results still showed NRC IRAP staff were stressed by heavy workloads (51%) and a lack of enough resources on their team (38%). Similarly, the vast majority of internal interviewees, consulted as part of the evaluation, stated that staff are stretched too thin. RCAO, CEA and ITA positions were viewed to be the most understaffed. Staff with these positions were perceived to be particularly vulnerable to burnout as the program continues to grow.

Due to an increasing number of sub-programs and perceived complexities in processes, many staff reported a capacity to address only immediate needs, with little or no time to address clients' strategic needs. They described being overwhelmed by pandemic-related tasks that disproportionately impacted their abilities to efficiently deliver the regular program. Although staff viewed, with a sense of pride, the delivery of IAP as an important way for them to contribute to the combat of the pandemic, they also cited it as a significant contributor to their elevated stress levels during the short timeframe under which it had to be delivered.

HR challenges make the program delivery model difficult to effectively scale-up

As with many other government units, the program consistently faces challenges in the recruitment, hiring and retention of staff and, in particular, qualified ITAs. Recruitment and hiring are impacted by long hiring cycles (~150 days) and salary budget constraints (- viewed by some as below market value). Meanwhile, retention is impacted by significant numbers of retirements. Qualified ITAs are often highly skilled technical professionals in the later stage of their careers.

Figure 15 – Text version

A projected line chart of the Industrial Research Assistance Program Research Council Officer (RCO) level staff's eligibility to retire by 2027. The figure compares NRC IRAP RCO staff with the NRC staff that will have the same eligibility to retire by 2027, finding that 37% of NRC IRAP RCO level staff will be eligible to retire by 2027, compared to the NRC average of 27%.

Program delivery tools are being improved

Source(s): Interviews and document review

Performance measurement tools and approaches are consistent with international good practice

NRC IRAP's performance measurement framework is for the most part aligned with international comparator programs. Indicators tracked (employment, R&D investments, growth) are similar. Like NRC IRAP, comparator programs align performance indicators with United Nations Sustainable Development Goals to assess sustainability of impacts on the wider economy. Periodic follow-ups with clients to assess durability of business are also conducted. NRC IRAP is currently collecting EDI data in accordance with its Inclusiveness Strategy.

However, there is future opportunity for the program to revisit its FY 2017-18 plans to systematically collect voluntary EDI data at the onset of their engagements with clients thus increasing the likelihood that data are available for more clients. Similar practices have been successfully adopted by international comparator programs.

Tools are more consistently adapted

Although the consistent and timely adaptation of program delivery tools has improved over time, there continue to be opportunities for more adjustment. For instance, the vast majority of staff interviewed stated that, despite efforts made to regularly update the field manual (a critical delivery tool), it still did not adequately reflect the numerous evolutions that the program has undergone.

Digital tools being employed are adequate for program needs; however, staff view some as inefficient

Most digital tools being utilized for program delivery are perceived to be adequate for the purpose for which they are employed. SONAR is the exception; the majority of staff interviewed stated that the tool is not ideal for all program delivery and impact assessment needs. SONAR continues to be the primary data repository for field staff, and persistent issues with data extraction and quality make it a challenge to work with for administrative staff. SONAR improvements are underway; however, communications to staff regarding the benefits of the revamped SONAR 360 system could be improved.

SAP Business Objects reportedly works well for NRC IRAP's core analytics team and has enabled the group to produce regular performance reports to support management decisions. However, the tool is also perceived by some staff to be an obstacle to good performance monitoring and reporting. The tool is reportedly clunky, not user-friendly and requires significant training to master. Power BI has been adopted as a replacement tool by the analytics team, but has not as yet been rolled out throughout the program.

Overall, the primary challenge with digital tools stems from the perception among staff that some tools currently in use across the program are not well-suited for the tasks to which they are applied. This can make performance reporting at different levels of the program (e.g. within regional and partnership and assessment teams) a laborious undertaking as data sometimes needs to be extracted from different systems in order to produce a simple data report.

Opportunities exist to improve program efficiency

Source(s): Document review, interviews, survey, focus group discussions

Communication with staff on their involvement in program design

Many improvements made to the program were supported by the Program Delivery Advisory Committee (PDAC). Senior staff indicate that the PDAC has been revamped to include a wider cross-section of representatives based on their geographic location, program experience and subject matter expertise. However, the vast majority of staff consulted were not aware of the existence of the PDAC. Furthermore, staff stated that there was an opportunity for their operational knowledge of the program to be better leveraged by senior management in design discussions. This suggests a need to better communicate to staff about the role of the PDAC in supporting program improvements, and how they might be able to contribute to related decision-making processes.

Program delivery tools

As previously noted, program delivery tools generally serve their intended purpose. However, consultations with staff also revealed some inefficiencies. This is specifically the case for tools that support program operations and performance reporting. Staff also identified a need for management to ensure that they have access to required tools and resources to allow the program to continue to effectively serve clients even as pandemic-related restrictions on travel and physical distancing persist. There is therefore an opportunity for the program to review operational tools such as the program field manual and digital platforms and ensure they are enhanced to meet the needs of users.

Implementation of the portfolio approach

Although the portfolio model was found to be effective at achieving its objective of expanding program reach among higher growth clients, the manner with which it was implemented introduced other issues. Staff consequently had mixed views about whether newly refined program delivery mechanisms (i.e. related to the portfolio model) better served the needs of clients. On the one hand, ITAs reported that portfolios allow them to focus resources on high potential firms, customize the time they spend with high-potential clients and find ways to push companies along their growth journey. Some though, were critical of the approach, categorizing it as bureaucratic, adding an administrative burden every few months and restricting their ability to focus on other categories of clients such as those belonging to EDI groups or aligned with other government priorities such as climate change. Only 46% of ITAs surveyed stated that implementing the portfolio approach helped increase the efficiency of their work.

Project development and approval processes

Although staff and clients perceive project development and approval processes to be generally effective, they indicate that opportunities to create more efficiencies exist. Potential improvements include:

• shortening the length of the processes (particularly for elements of the International program that are within NRC IRAP control)
• ensuring consistency in updating staff guidance (particularly for ITAs) in the field manual and operational policy documents
• further refining the contents of the RCAO checklist to improve clarity and applicability for all staff who need to refer to it

Human resource challenges

The program is actively working to address the human resource issues identified earlier by exploring more creative ways to recruit and retain qualified technical staff. These ways include:

• leveraging professional platforms like LinkedIn to proactively engage potential candidates, including those from diverse backgrounds
• enhancing the level of flexibility in retirement options for technical staff. For example, some retired ITAs are offered the opportunity to support OGD technical assessments on a contractual basis

However, there is still room to explore more creative options. One such option which may warrant consideration is engaging technical staff from within the NRC to support existing ITAs in the execution of some of their duties. Some NRC staff, particularly researchers and NRC IRAP CEAs, expressed interest in taking on tasks traditionally assigned to the ITA role. In their view, they have the industry experience and technical expertise required to be successful. Leveraging the support of other NRC staff would help NRC IRAP reduce the workloads of ITAs and allow them to focus their efforts on addressing clients' strategic needs.

Another potential solution is to review decision-making processes around the allocation of tasks to staff. Interviewees suggested that such a review begin with the premise that the program will always be slightly understaffed, and this should factor into any plans to take on large projects. The ultimate aim would be to ensure a better match between the volume of tasks assigned and the available resources.

White label services

The idea of white label services was still evolving as the evaluation was being conducted. As a result, not all staff were aware of what (beyond the conduct of technical assessments) the services entail. With respect to technical assessments, client perceptions were mostly positive, as mentioned earlier.

However, there are still some opportunities for improvement in efficiency (including data collection, ITA matching to and training on assessments, appropriateness of tools, etc.). Inefficiencies were exacerbated by a lack of formal standard operating procedures to guide the technical assessment process. This made it challenging, but not entirely impossible for ITAs to provide consistent assessments on OGD program client bids.

There is an opportunity for NRC IRAP to play a more active role in ensuring that some formal standard operating procedures are put in place to enhance OGD clients' individual processes. Clients interviewed were open to receiving this sort of guidance from the program.

Relevance

NRC IRAP is overall well aligned with Government of Canada priorities focused on supporting innovation. The program specifically aligns with the Government's Inclusive Innovation Agenda and the six areas identified for action. During the evaluation period, NRC IRAP proved to be a key player in Canada's innovation ecosystem, and remained uniquely positioned to support government efforts to achieve its vision of building Canada as a global centre for innovation. Program progress towards full alignment with Government priorities around GBA Plus has been positive. The program continues to evolve from data collection to the actual use of data for program management. As such, there are opportunities for continuous refinement over the next few years. There is a shared perception among clients and stakeholders that NRC IRAP is unique among Government innovation support programs in the combination and quality of services that it offers. The program has been increasingly more relevant as it fills a significant gap in the innovation support ecosystem for small and medium-sized enterprises as well as other government departments.

Entrepreneurship and science excellence supported

Source(s): Document reviews, interviews

NRC IRAP supported Government efforts to foster an entrepreneurial and creative society

The program aligns with government priorities aimed at fostering economic growth through SME innovation. NRC IRAP ITAs are unique in the Canadian innovation support ecosystem in their ability to provide both technical and business advisory services to program clients. The long term ITA-client relationship was perceived to be integral to encouraging and supporting Canadian entrepreneurs, allowing them to adopt and maintain a mindset that fosters creativity, risk-taking and ambition.

NRC IRAP supported government efforts to enhance global science excellence

The program facilitated SME client access to world class scientific expertise and infrastructure. Through the Contribution to Organizations (CTOs), researcher interaction and certificate programs, NRC IRAP provided clients an opportunity to experiment, explore and strengthen their basic and applied research capabilities. In this way, the program promoted pathways to support the transformation of new ideas into practical solutions Canada could leverage to support economic growth.

Business innovation and competition supported

Source(s): Document review

NRC IRAP supported Government efforts to build world-leading clusters of innovation

In alignment with the Innovation Agenda's call for more innovation-focused partnerships among businesses, research institutes and governments, NRC IRAP partnered with other government departments (OGDs) to create the Accelerated Growth Service (AGS). Through the AGS, the program was able to identify more than 200 high potential firms and support their growth. In addition, NRC IRAP has entered into memoranda of understanding with OGDs and not-for-profit organizations to support their respective SME innovation programs by contributing to program design and conducting technical assessments.

NRC IRAP supported Government efforts to increase the ease of doing business with Canada

Through its core and international programs, NRC IRAP provided an avenue for SME clients to co-innovate with foreign partners, commercialize their ideas and gain access to global markets/contribute to global value chains. As an example, during the evaluation period, NRC IRAP continued to invest in the Eureka program (a network of 40+ partners' economies, primarily in Europe) and the Canada International Innovation Program (CIIP) in Brazil, India, China and South Korea. The program additionally engaged with two new programs in Europe (i.e. Eureka Eurostars and 2+2 Germany) to support market oriented R&D collaboration. Consequently, the number of unique NRC IRAP clients engaging in the international program increased by 210% from 123 in the previous evaluation period to 381 in the current period. The number of international projects between the two periods also increased substantially (by 292%) from 158 to 620.

NRC IRAP supported Government efforts to improve Canada's ability to compete in a digital world

During the evaluation period, the program supported Canadian SME clients through advisory services and funding to leverage the benefits of current and emerging digital technologies for their products. This helped firms produce more innovative products, offer customers more choice and increase their competitive edge both nationally and internationally.

This support builds upon the 2011-2014 NRC IRAP delivery of the Digital Technology Adoption Pilot Program (DTAPP). The objective of DTAPP was to accelerate the adoption of digital technologies in SMEs operating in Canada so they can boost their productivity and create economic growth and opportunity.

Business innovation and competition supported

Source(s): Document and data review

NRC IRAP supported Government efforts to grow companies

During the evaluation period, NRC IRAP continued to focus on supporting SMEs to start-up and scale-up in alignment with its mandate and the Government's Innovation Agenda. The program ensured that clients were supported with funding and advice through the regular IRAP, Youth employment program (YEP), Accelerated Review Process (ARP) and Large Value Contributions (LVC) newly introduced to encourage transformational growth.

NRC IRAP supported Government efforts to accelerate clean growth

NRC IRAP also contributed to the acceleration of clean growth by performing well against clean technology targets set out in the NRC Departmental Sustainable Development Strategy. The program spent between $15M and $73M per year on 194 to 412 clean tech projects, compared to the targeted $15M for 125 projects per fiscal year.

Out of 14,499 projects funded in the evaluation period, 1,579 or 11% were clean technology projects, and roughly 10% of clients had a clean technology project. The top categories were emissions control, clean energy and smart grid.

Figure 16 – Text version

Two vertical bar charts representing the number of clean tech projects and contributions made by the NRC IRAP from FY 2017-18 to FY 2020-21. The NRC IRAP exceeded its targets on total number of both clean tech projects and financial contributions to clean tech projects during the 5 year period.

Increased relevance and alignment with GBA Plus

Source(s): Document review and interviews

NRC IRAP became an increasingly relevant player in the Canadian ecosystem for innovation support

As previously mentioned, in FY 2017-18 the program became one of four flagship platforms through which the federal government is streamlining the delivery of innovation programming across Canada. During the evaluation period, the Government dedicated increasingly more financial resources to support NRC IRAP programs.

In Budget 2021, the federal government committed an additional $500M to the program to be rolled out over the next five years in equal installments of $100M per annum. In addition, over the course of the last five years, an increasing number of OGD partners have recognized value in and have sought out the expertise of NRC IRAP staff. The program provided OGD programs with back-end administrative support (e.g. Can Export) as well as business and technical assessments. As a result, the program was able to increase costs recovered each fiscal year.

NRC IRAP made progress towards alignment with GBA Plus priority

NRC IRAP has taken steps towards greater equity, diversity, and inclusion (EDI) among staff and clients served. The establishment of the Client Enablement team in FY 2019-20 has been one of the most critical steps. The program has implemented some initiatives to increase staff diversity, ensuring that it at least meets labour market availability, and to begin the process of systematically collecting critical EDI data on clients.

However, EDI information is not yet available on the majority of unfunded clients and about half of the funded clients during the present evaluation period.

Importantly, most staff interviewed (including senior NRC IRAP officials) were not aware of the EDI client-related initiatives that were being integrated or considered for integration into programming.

There is, therefore, an opportunity for the program to better communicate about the progress made in monitoring client EDI information (as well as actual EDI data) and how the GBA Plus lens will be integrated in decision-making and the development of programming.

The program meets a need in the ecosystem

Source(s): Interviews, case studies

NRC IRAP is unique among government innovation support programs and meets a need in the ecosystem for SME and OGD clients

Stakeholders consulted indicated that NRC IRAP meets a need in the ecosystem for innovation support. The program is considered to be unique because it operates at the national level, is sector agnostic, and offers advisory services combined with direct funding. Unlike other programs, NRC IRAP targets SMEs of all sizes and sectors, and offers non-repayable contribution funding rather than repayable loans. The program also has access to a team of highly technical- and business-savvy ITAs that are able to maintain long-term relationships with SMEs and provide them with invaluable advice.

The program also meets a need in the ecosystem for innovation support services required by other government departments. It does this by continuing to conduct technical assessments and by contributing to new program design. OGDs state that they would have a difficult time finding an alternative service provider if NRC IRAP no longer provided technical assessments. Affiliation with ITAs, who are respected within industry, is seen to add a level of rigour/credibility to partners' processes that can stand up to intense public scrutiny around the disbursement of large-value awards for innovation.

Services offered by organizations are perceived as valuable by SME clients

NRC IRAP contributions to organizations (CTOs) facilitate access to an important suite of services that fill a gap for many SME clients. Importantly, CTO services are complementary to the support provided by ITAs. CTOs assist SMEs with small projects, providing them with highly specialised support, helping scope or solve a technical or business challenge, and/or making suggestions for next steps.

Many SMEs do not have the scientific expertise on staff nor the larger-scale processing equipment or pilot-scale infrastructure needed to advance their innovations. For them, the relationship with organizations is critical.

"We went to market sooner and were able to hire people thanks to the project."

Recommendations

Recommendations and supporting rationale

Human resources

Supporting rationale

The program did not meet its hiring target for ITAs (i.e. its primary workforce) during the reporting period. Hiring has been and will continue to be a challenge due to the hard-to-fill nature of the ITA position that typically requires highly experienced technically savvy and late-career industry professionals. This constitutes a significant risk for the program, as HR data show that almost 40% of technical staff, of which ITAs make up the majority, will be eligible for retirement by 2027. This would be 10% more than the NRC overall.

Furthermore, the latest PSES results show that NRC IRAP staff were stressed by heavy workloads (51%) and not enough resources on their teams (38%). The vast majority of internal interviewees, consulted as part of the evaluation, also stated that staff are stretched too thin. The imbalance in staff workload was particularly amplified during the implementation of COVID-19 related temporary programs (e.g. the IAP). RCAO, CEA and ITA positions are viewed to be the most understaffed and vulnerable to burnout.

Efficiency

Supporting rationale

Since the last evaluation, the program took concrete steps to streamline and refine project development and approval processes. Staff with authority on the subject indicated that processes were presently more efficient but required some further improvements to become optimal. Opportunities were identified to make further adjustments to the RCAO checklist to ensure its applicability to all (non-RCAO) staff that need to refer to it and to reduce the level of bureaucracy embedded specifically in LVC and IRAP INT project development and approval processes.

Program data show that the originally established target for LVC projects (i.e. 44 per year) was not met, as only 40 projects in total were funded over the evaluation period. Consequently, not all committed funds were spent. The NRC IRAP management decision to slow implementation of LVC during the first year of the pandemic was a contributing factor. However, stringent processes around project development and approvals as well as the program's lack of success in meeting related hiring goals for additional ITAs are perceived to be the primary cause of this situation. With respect to IRAP INT, inefficiencies related to process have deterred ITAs from pursuing international opportunities for their clients, an issue that was later confirmed by IRAP INT staff consulted.

GBA Plus integration

Supporting rationale

The program recently made progress in the collection of client EDI data and several initiatives are currently underway to meet evolving program delivery needs related to GBA Plus commitments. However, there is a need to leverage this momentum to better communicate and integrate GBA Plus considerations into program management and operations.

Staff reported a genuine lack of awareness about ongoing EDI initiatives and how they might be able to operationalize a program's inclusiveness strategy in the execution of their duties. In addition, they do not currently have clear information on program EDI objectives and/or KPIs that they should be working towards. Staff are critical of the portfolio approach, categorizing it as bureaucratic, administratively burdensome and restricting to their ability to focus on other categories of clients including those belonging to underrepresented groups. Interviewees additionally reported that rules around client eligibility can hinder expanded reach within Indigenous communities. This is because many Indigenous SMEs tend to use a different legal structure from that which government typically deems as eligible for funding. There is a risk that these issues could negatively impact client access to and public perceptions of the program.

In order for GBA Plus to be fully integrated into the program, staff need to be supported to understand the hidden barriers faced by diverse clients seeking access to NRC IRAP services. International comparator programs are establishing direct relationships with underrepresented groups. In so doing, they are obtaining guidance related to program design and support in the integration of feedback loops in overall program management processes. This has permitted the development of programs specifically targeted at underrepresented groups. For instance, Innovate UK, an international comparator program, supported in-depth research, outlining barriers, challenges, opportunities and support needs for Visible minority groups and Persons with disabilities to participate in business innovation. Since then, the program has continuously engaged with special interest groups to better understand barriers and support evidence-based solutions in increasing their reach. Given that NRC IRAP has identified a need to expand its reach beyond previously engaged clients, the program may benefit from adopting a similar evidence-based approach to identifying barriers limiting the greater participation of underrepresented groups in its program. Addressing ongoing EDI issues and sustaining solutions in the long-term may also necessitate that NRC IRAP embed regular dialogues with underrepresented groups in program management processes.

Performance measurement

Supporting rationale

The program has existing processes and tools to govern performance measurement. However, these are not always formally documented or clearly outlined in a single repository. This poses a risk to the program in the event of employee attrition or departures. There is specifically a potential for corporate knowledge loss on mechanisms for the development, update and deployment of performance monitoring and measurement tools, the frequency of data collection, data quality assurance, data analysis processes and procedures for internal and external results reporting.

In addition to introducing improved reporting tools (annual datasheets and monthly management reports), the program initiated the development of some integrated data spreadsheets (Excel) to annually compile data on KPIs. More recently, the program also developed an Excel-based management toolkit to monitor progress towards KPI targets. However, there is an opportunity to leverage the new management toolkit to streamline data compilation efforts as well as the number of tools being used to communicate results. While the program has already taken steps to adopt more advanced integrated software (e.g. Power BI), there is an additional opportunity to make use of interactive features of this software to convert elements of the management toolkit into a dashboard. This would make performance data more accessible (to the extent that is appropriate) to operational and management staff based on their varying levels of need for the information (e.g. operations, decision-making and/or reporting).

Additionally, KPIs and targets exist for some core program components but not others. Results reporting, particularly to senior management, is not always framed in the context of objectives and targets that were formally set by the program or envisioned for specific program components. It is good performance monitoring practice to track all indicators in the context of a target and, in cases where none is available, such rationale should be clearly documented within tracking tools. Consequently, there are opportunities to adopt more indicators (for internal tracking purposes) and further leverage data available for existing KPIs to test critical program assumptions (e.g. effectiveness of the portfolio model at reaching and supporting high-potential firms) and inform decision-making.

Management Response and Action Plan

Appendices

Appendix A – Methods

Document review

Internal and external documents were reviewed to provide context and to complement other lines of evidence in assessing relevance and performance. Internal documents included program area documents and strategic business documents. External documents included previous evaluative works, published and grey literature, and Government of Canada publications.

Data review

The data review covered IRAP administrative data on clients and projects, characteristics information on YOUTH program participants, snapshot information on clients, post-project assessments, post-project report data, data on advisory services, and other data regarding budget, human resources and key performance indicators.

ITA survey

A total of 308 ITAs were invited to compete an online survey which included questions related to the value and impact of certain tools on the ITA's work, as well as common barriers to their success.

125 ITAs responded for a response rate of 41%.

Importantly, ITAs were asked to identify potential candidate projects and firms for the PBCA study. They were asked to identify IRAP projects for which Canada has obtained:

• very high economic impacts—and ideally huge impacts
• impacts which can be quantified in dollars ("dollarized")
• impacts that are attributable to IRAP support, at least in part

143 firms, most of which had multiple IRAP projects, were identified through the ITA survey.

Partial benefit cost analysis (PBCA)

A PBCA is an approach to quantifying and comparing the costs of a program with the benefits it generates. With a PBCA, all of the program costs are accounted for, but only some of the benefits are quantified. As a result, the findings of the PBCA can indicate that the benefit-cost ratio is at least as large as estimated, and likely larger. Under this approach, the gross and net benefits of what are believed to be the "highest impact" projects are quantified. These are compared to the total costs of the program thus returning a lower bound estimate of value generated and a benefit-cost ratio.

The study identified 31 high impact firms through indications in the survey sent to ITAs, taking into account the number of projects completed, total value of projects and revenue and profit from FY 2011 to 2021.

In addition to internet searches, 22 firms were interviewed, and 17 were included in the study (9 firms declined or did not respond, and 5 firms dropped due to inability to clearly align with NRC IRAP contributions).

Journey mapping

Journey mapping was conducted with two advisory services-only firms or unfunded firms. This client journey map consists of a diagram that illustrates the process clients go through in engaging with NRC IRAP and the key steps or milestones produced as a result.

Conversations were scheduled with founders that resulted in charts depicting the company's trajectory from their start to their contact with NRC IRAP to where they stand presently. ITAs responsible of those firms were also consulted.

International comparison study

From a preliminary list of NRC IRAP's international counterparts, four institutions were selected based on the structure of programs, services offered, and their financial/resource capacity: Business Finland (Finland); Innovate UK (UK); Vinnova (Sweden); and RVO.nl – Netherlands Enterprise Agency (The Netherlands).

Representatives from all four international comparators were interviewed. An interview guide was developed based on elements that could not be found in NRC IRAP's benchmark report or in the public domain (such as websites). Four interviews with five (n=5) representatives were conducted to fill in information gaps and to validate information that was collected through the document and website review.

CTO case studies

The case studies focused on documenting and understanding the outcomes that SMEs experience as a result of assistance from funded organizations (CTOs), specifically the impacts of this assistance on innovation and commercialization.

The case studies were conducted with four companies (from Ontario, Quebec and Atlantic regions). Interviews were conducted with the responsible ITAs for each company/project, as well as representatives from the companies.

Interviews and focus groups

A total of 64 people were interviewed between July 7th, 2021 and October 28th, 2021. Interviewees included IRAP staff, external stakeholders, and IRAP clients. Interviews conducted in the context of CTO case studies, journey mapping and PBCA mapping are not included in this total.

Five focus groups of approximately two hours were held with ITAs from each of the five IRAP regions. Participants were selected from those who volunteered and those who were nominated by the regional Directors General, totaling 36 participants. A list of discussion topics, as well as a note from the relevant NRC IRAP regional Directors General explaining the purpose of the exercise, was provided in advance. The topics were mainly related to the need for IRAP, the impacts of IRAP advisory and funding for SMEs, and ways in which program delivery and administration could be improved.

Limitations and mitigation strategies

Although the evaluation encountered some challenges, methodological limitations were mitigated, where possible, through the use of multiple lines of evidence and the triangulation of data. Details on limitations and their associated mitigation strategies are described below.

Assumptions underlying the PBCA results

Partial benefit cost analysis relies on a limited number of cases and assumptions about the economic value of benefits which could result in some inaccurate conclusions.

Mitigation: These assumptions have been taken into account in the sensitivity tests (upper and lower bounds; three discount rates). Also, where there are methodological uncertainties (e.g. with regard to attribution to IRAP), the analysis has been based on conservative assumptions.

Limitations of administrative and performance data

While NRC IRAP provided a large quantity of data to the OAE team, extensive data cleaning was required before analysis could be conducted. The extent of this cleaning exceeded the capacities of the digital systems available for the evaluation and thus impacted the ability of OAE to accurately perform some of the data analysis.

Mitigation: OAE and IRAP worked together to fully assess NRC IRAP's impact through creating a shared understanding of program data strengths and limitations. The program's analytics team additionally provided the evaluation team with some pre-analyzed data.

Sample size and bias with interviews

Relative to the selected populations, the number of interviews may not be representative of the total numbers of each group. Also, interviewees who are or were clients of NRC IRAP may be biased to respond either positively or negatively depending on whether they received funding or advice only.

Mitigation: Internal interviewees were carefully selected to obtain a comprehensive and balanced sampling across all five program regions.

Furthermore, 34 more interviews, for a total of 64, were conducted during the evaluation fieldwork than initially planned (n=30) in the evaluation methodology report.

The primary strategy to mitigate potential bias from client interviews was to ensure multiple lines of evidence addressing the same evaluation questions, interviews with funded clients in the context of the PBCA study (n=22), and the journey Mapping (n=2), CTO case studies (n=4) as well as administrative and survey client and projects data collected by the program .The primary strategy to mitigate potential bias from client interviews was to ensure multiple lines of evidence addressing the same evaluation questions, interviews with funded clients in the context of the PBCA study (n=22), and the journey Mapping (n=2), CTO case studies (n=4) as well as administrative and survey client and projects data collected by the program.

Furthermore, the interviews were used to obtain a range of informed and in-depth opinions rather than a consensus.

Appendix B – Supplementary data on program reach