Evaluation of the Innovation Assistance Program

Abbreviations and acronyms

BDC

Business Development Bank of Canada

BIGS

Business Innovation and Growth Support

CRA

Canadian Revenue Agency

CEBA

Canada Emergency Business Account

CEWS

Canada Emergency Wage Subsidy

EDC

Export Development Canada

FY

Fiscal Year

FTE

Full-time Equivalent

GBA Plus

Gender-based Analysis Plus

GDP

Gross Domestic Product

IAP

Innovation Assistance Program

ICT

Information, Communications and Technology

ISED

Innovation, Science and Economic Development Canada

IT

Information Technology

ITA

Industrial Technology Advisor

LoA

Letter of Agreement

NRC IRAP

National Research Council's Industrial Research Assistance Program

NAICS

North American Industry Classification System

NRC

National Research Council of Canada

NSERC

Natural Sciences and Engineering Research Council of Canada

OAE

Office of Audit and Evaluation

OGDs

Other Government Departments and Agencies

OECD

Organisation for Economic Co-operation and Development

R&D

Research and Development

SME

Small- and Medium-sized Enterprise

TB

Treasury Board

Introduction

This evaluation of the Innovation Assistance Program (IAP) covered the fiscal year 2020-21, the period in which the program was delivered. The IAP was launched in April 2020 as an emergency wage subsidy program for innovative Canadian businesses impacted by the COVID-19 pandemic. The program was delivered by the National Research Council's Industrial Research Assistance Program (NRC IRAP). While the program ended in March 2021, at its onset the NRC made a commitment to conduct an evaluation to identify lessons learned from program design and delivery, as well as to assess key program outcomes.

The evaluation was conducted by the NRC Office of Audit and Evaluation (OAE) in accordance with the NRC approved Departmental Evaluation Plan, Treasury Board's Policy on Results (2016) and the requirements of the Financial Administration Act.

This report begins by providing a profile of the Innovation Assistance Program. It then presents evaluation findings on the need for this temporary program, immediate intended and unanticipated outcomes, and lessons learned. Four recommendations for improvement are included at the end of the report.

Evaluation approach

Approach

The evaluation applied a streamlined, mixed-methods approach, incorporating both qualitative and quantitative data from several lines of evidence. This allowed for triangulation of the evaluation findings. A Gender-based Analysis Plus (GBA Plus) lens was applied throughout the conduct of the evaluation.

Scope

The evaluation included small- and medium-sized enterprises (SMEs) that participated in the first round of program funding. SMEs from rounds 2.0 and 2.5 consisted of a subset of first round funding recipients and so were naturally included in the evaluation. As the evaluation used a streamlined approach, unfunded IAP applicants were not included in the scope.

Methods

The evaluation included the following lines of evidence:

• document review (internal and external sources)
• data review (administrative and performance data, data from final reports and a post-funding assessment)
• internal interviews (n=36)
• external interviews (n=19)
• client focus groups (n=3, 15 participants)
• case studies of funded firms (n=15)
• economic impact analysis (input-output simulation model conducted by Statistics Canada)

See appendix A for detailed information on the methodologies used to develop the report, including limitations and mitigation strategies.

Evaluation questions

1. Relevance: To what extent was there a need for this type of program to support small- and medium-sized enterprises (SMEs) during the COVID-19 pandemic?
2. Intended Outcomes: To what extent did the program achieve its intended outcomes (i.e., high-potential firms receive emergency financial support, funded firms survive COVID-19 disruptions, funded firms contribute to Canada's economic recovery)?
3. Unanticipated Outcomes: Were there any unanticipated outcomes from the Innovation Assistance Program funding provided?
4. Program Design and Delivery: What lessons-learned related to design and delivery can be derived from the program?

Return to Top of page

 

Profile

The program provided an emergency wage subsidy for innovative small- and medium-sized enterprises' (SMEs') employees over three rounds of funding from April 1, 2020 to March 13, 2021. Eligibility to participate in the program was predicated on SMEs being ineligible for the Canada Emergency Wage Subsidy (CEWS) and unable to access liquid assets from other sources. There was no cap on the number of employees supported per company, but employees must have met eligibility criteria determined by NRC IRAP. Only those firms that participated in the first round of funding were eligible for rounds 2.0 and 2.5. The maximum benefit per employee gradually declined as the program progressed.

Overview of the program

Source(s): Program documentation

On April 17, 2020, in response to the COVID-19 pandemic, the Government of Canada announced $250 million in emergency wage subsidy for small- and medium-sized enterprises (SMEs), to be allocated through the Innovation Assistance Program and delivered by NRC IRAP.

The program aimed to subsidize payroll costs of innovative early-stage and high growth potential Canadian firms who were seeing a downturn in markets due to the COVID-19 pandemic. The Minister of Innovation announced a target of supporting 1,000 innovative firms.

The program was designed and launched during a period of unprecedented national and global economic uncertainty. It was rolled-out over a period of four business days, from the Prime Minister's announcement on April 17, 2020 at mid-day to the completion of the application portal on the evening of April 21, 2020. The portal was operational for the reception of applications on the morning of April 22, 2020.

On November 6, 2020, approximately 6.5 months later, the Government of Canada committed another $155 million to the Innovation Assistance Program. These additional funds allowed NRC IRAP to extend a second round of support to existing round 1.0 recipients that met round 2.0 eligibility criteria. Once round 2.0 criteria were applied, a balance of funds was still available and the program further adjusted the eligibility criteria to administer round 2.5.

 

The NRC IRAP was instructed by Innovation, Science and Economic Development Canada (ISED) to issue the second round of IAP funding to qualified round 1.0 recipients by invitation only. The funding envelope made available was based on an assumption that the majority of round 1.0 funding recipients would qualify for additional support.

Over the funding period, NRC IRAP disbursed a total of $373.8 million to SMEs

Funding amounts and timeframes for the three rounds of funding are:

• Round 1.0: $246.3 million over 12 weeks from April 1 to June 24, 2020.
• Round 2.0: $94.1 million over 25 weeks from June 25 to December 19, 2020. Round 2.0 firms were among those that received round 1.0 funding.
• Round 2.5: $33.5 million over 14 weeks from December 20, 2020 to March 13, 2021. Round 2.5 firms were among those that had received round 2.0 funding.

Salary was the only eligible cost and there was no cap on the number of employees per company, although employees did need to meet eligibility criteria. For round 1.0, a first, up-front payment of 80% of the funds was issued in May and a second payment for the remaining 20% was issued in June following the provision of information substantiating claims.

The IAP supported 2,230 SMEs

Round 1.0	Round 2.0	Round 2.5
2,230 firms	1,384 firms	1,269 firms
$246.3 million	$94.1 million	$33.5 million

Program delivery by NRC IRAP

Source(s): Program documentation

Delivery of IAP built on IRAP resources and expertise. Existing NRC IRAP staff, 481 full-time equivalents (FTEs), including 269 Industrial Technology Advisors (ITAs), were leveraged to deliver the IAP. The program's application system was built to be as straightforward as possible.

The objective of targeting high-potential firms was operationalized through merit-based criteria. As well as need, selection criteria considered a firm's prospects for future growth and success post-COVID-19. Seven key questions were used to determine eligibility and application scores, based on answers provided by firms and reviews conducted by ITAs. For more detailed information on the qualification questions and associated impact on rating, refer to appendix B.

Through round 1.0, the program provided, for each eligible employee, $847 per week for 12 weeks, totaling a maximum of $10,164 per employee. To receive the maximum allocation, a company's net salary costs, after any federal government support deductions, must have been equal to or greater than the amount claimed under the program. Eligible employees were salaried employees (T4) or employees receiving self-employment income (T4a), typically consultants or contractors.

Round 2.0 focussed on firms that had a formal engineering, research and development (R&D) or new product innovation team. For round 2.5, only firms that had received round 2.0 funding and had sales up to $1 million between January 2020 and December 2020 were invited to apply.

For rounds 2.0 and 2.5, depending on the change in level of staffing since the end of round 1.0 and on how long firms would be able to continue operating without IAP assistance, funding was allocated on declining scale of support per employee per week (e.g., over a period of three months: from $847.50/week down to $451.60/week).

When IAP applicants were receiving funding from other government programs, the firms were required to report on and comply with stacking obligations within their IAP contribution agreement. NRC IRAP staff needed to ensure stacking limits were respected for funds received from other sources (including other government departments and agencies and NRC IRAP R&D project funding) did not exceed 100% of the total salary cost of any employee being claimed. Specific guidance was provided to NRC IRAP staff regarding stacking.

IAP Round 1.0 merit-based qualification question themes

1. Identifies the funding stream most appropriate for the business
2. Business has a formal engineering, R&D or innovation team
3. Business's ability to execute Innovation-based commercialization (sales from new products launched in the past 2 years)
4. Business sales patterns, export focus (Canadian, North American and/or international clients)
5. Business impact on the supply chain of other business
6. Business capacity to adapt to post-COVID 19 market dynamics
7. Business capacity to maintain business operations amidst physical-distancing

 

Stacking of funding occurs when a firm receives funding through multiple government assistance avenues (federal, provincial, territorial and/or municipal) to offset eligible project expenses and there is a risk that, when combined, government assistance could exceed a set percent of eligible project costs.

Profile of program funded firms

Source(s): Document review, data review

Recipients funded by the program, from all rounds, primarily consisted of early stage and small firms. The majority of IAP-supported firms had existing relationships with the NRC IRAP.

• 72% of funding went to firms with 9 employees or fewer
• 65% of funding went to firms with less than $250,000 in annual revenues
• 35% of funded firms were pre-revenue (i.e., had not generated any sales)
• 61% of funded firms had previous interactions with NRC IRAP (i.e., benefited from advisory services and/or project funding between FY 2011 and FY 2020)

 

The information, communications and technology (ICT) sector is a cross-cutting sector, made up of software and computer services, communications services, manufacturing and wholesaling.

The Gross Domestic Product (GDP) is the total value (expenditure-based) of all final goods and services produced annually in a given region (provincial and territorial) or country.

 

Figure 1 – Text version

Pie chart showing the sector distribution of IAP-supported firms.

• 49% supported firms were in the ICT (digital economy) sector
• 11% supported firms were in the health and life sciences sector
• 7% supported firms were in the energy and resources sector
• 6% supported firms were in the manufacturing and materials sector
• 4% supported firms were in the agriculture and food sector
• 23% supported firms were in other sectors

GBA Plus profile of funded firms

Source(s): Document review, data review

GBA Plus was not a consideration in the design of the IAP. After the program was launched, the NRC asked firms to provide (on a voluntary basis) data to allow them to determine which firms were owned or managed by historically underrepresented groups (i.e. women, Indigenous peoples and other minority groups).

NRC IRAP launched data collection for GBA Plus information in FY 2021. NRC IRAP clients (including Innovation Assistance Program firms) were invited to voluntarily update information in NRC IRAP's client portal. The information requested asked firms to identify the percentage of women, members of visible minorities, and Indigenous peoples at three levels in the organization (business ownership, board of directors and executive team).

A total of 626 records were completed by IAP firms, meaning that GBA Plus data is available for about 28% of IAP-funded firms. Based on this partial data, IAP-funded firms were similar to NRC IRAP firms with regard to GBA Plus representation.

Relevance

SMEs faced significant and immediate challenges at the onset of the pandemic when COVID-19 containment measures were being implemented globally and future impacts of the pandemic were unpredictable. Given this context, there was a demonstrated need for the program. Similar programs were being implemented in other international jurisdictions. The program filled a need in Canada to enable innovative, high-potential SMEs to access government emergency support initiatives and worked in tandem with other emergency support financial programs.

Demonstrated need for the Innovation Assistance Program

Source(s): Document review (McKinsey), data Review, external and internal interviews, focus groups, case studies

At the time the program was launched, the need was evident

SMEs are important to the Canadian economy. Small businesses make up 98% of the employer businesses in Canada, and employ 69% of the private sector labour force. This is similar to the 75% share of SME employment on average across Organisation for Economic Co-operation and Development (OECD) countries.

SMEs and start-ups were reported to be among the most affected and most vulnerable organizations in terms of impacts of COVID-19 because they were financially more fragile with smaller cash buffers and weaker supply chain capabilities. Yet immediate public response measures to the pandemic typically did not target early stage firms or start-ups specifically. In addition, many liquidity relief measures were not easily accessible for new ventures because of their eligibility criteria. Immediate government supports often required proof of existence and evidence the organization had been profitable in preceding years.

The pandemic created major challenges and uncertainty for SMEs, especially for early-stage firms affected by issues such as abandonment or reduction of R&D activities, staff cutbacks, market disruptions, cancelled contracts, risk aversion from investors, and impacts on key supply chain partners.

The high number of applications (4,257) to the IAP demonstrates the need for the program. The number of applications was also viewed as an indication of adequate outreach by the program.

The use of wage subsidies was common amongst large economies globally

Wage subsidies were commonly used globally to respond to immediate needs for support at the outset of the COVID-19 pandemic. Reports indicate 90% of countries released measures specifically targeted at SMEs and 95% of high-income countries used wage subsidies as a response to COVID-19 (OECD).

As was the case in Canada, other governments also developed supplementary liquidity measures specifically targeting emerging firms to complement other emergency programs.

"We are a young company, founded in January 2020. When the pandemic broke, we were in the process of raising funds. The pandemic put a stop to that and left us in a fragile position, so IAP was very timely."

"We are a pre-revenue start-up and were really concerned about the ability to generate investments with everything locked down. IAP was a life saver. We had laid-off our staff, but were able to bring everybody back when we got the subsidy."

Source(s): Document review (Statistics Canada, Startup Genome), data review, external and internal interviews, focus groups, case studies

SMEs were hit harder by the pandemic

IAP-funded firms were encountering a number of challenges to maintain operations which led to, or were leading to, defensive measures in order to conserve cash. The most frequent approach to conserving cash was reducing staff complements which subsequently delays R&D and affects competitiveness.

In the third quarter of 2020, 40% of Canadian SMEs reported a revenue decline of 40% or more from April 2019, compared to 28% of businesses with more than 500 employees. 45% of Canadian SMEs were laying off staff, with over half of those laying off more than 50%, compared to 23% among businesses with more than 500 employees.

As of April 2020, when the program was launched, Statistics Canada published data indicating that 26% of SMEs with one to four employees, 40% of SMEs with five to 19 employees, and 35% of SMEs with 20 to 99 employees had to lay off 50% or more of their staff because of the pandemic. In comparison, only 17% percent of large organizations had to lay off 50% or more of their staff.

Globally, statistics from 50 countries indicate:

• Almost three in four start-ups saw their revenues decline and their liquidity position challenged in the immediate aftermath of the pandemic.
• 41% needed to raise capital over the next three months to survive.

IAP-supported SMEs struggled to keep staff employed at the start of the pandemic

Some SMEs had already implemented lay-offs at the time the IAP became available. Without the IAP funding, 62% of round 1.0 firms reported that they would have had to lay off or otherwise reduce staff. Similar results were reported by round 2.0 (67%) and 2.5 (60%) recipients. Furthermore, during rounds 2.0 and 2.5, firms continued to anticipate negative circumstances without program funding.

In addition to IAP, to fully offset pandemic impacts, many firms needed additional sources of funds beyond salary supports. Typically business loans were sought in order to maintain business activities, including being able to meet ongoing operational expenses.

The program filled a gap

Source(s): Document review, data review, case studies, internal and external interviews, focus groups

The IAP filled a gap in the ecosystem

Stakeholders unanimously agreed that the IAP was needed and specifically filled a gap in the innovation support ecosystem for early stage R&D SMEs. The IAP targeted firms that did not qualify for the Canada Emergency Wage Subsidy (CEWS), the government's main emergency wage subsidy program administered by the Canada Revenue Agency (CRA). These firms included early-stage and pre-revenue firms as well as those that could not demonstrate a 30% drop in revenue.

Firms in receipt of program funding needed a combination of supports to weather the crisis

IAP-supported firms needed a combination of supports to maintain activities and sustain their cash flow. During round 1.0, 55% of firms also received another form of government support. The most frequently accessed included the CRA temporary 10% wage subsidy for employers, the Canada Emergency Business Account (CEBA) interest free loan, Business Development Bank of Canada (BDC) Co-lending to support operational cash flow, and the Export Development Canada (EDC) Loan Guarantee to support access to credit. Emergency loan programs helped cover operational expenses, such as maintaining supplies, while the IAP supported salaries.

The IAP was complementary (not duplicative) of other supports due to how the program was designed and implemented so as not to overlap with other programming. For example, funding could not overlap with the CEWS and stacking rules were taken into account.

Needs evolved over the program's delivery period, and the program adjusted

Some firms – especially ICT firms – were able to recover quickly and even grow during the pandemic because of the ubiquitous need to shift activities to virtual platforms. The program adjusted its requirements for rounds 2.0 and 2.5 to focus on smaller firms with ongoing needs and/or a dedicated R&D component, and to implement a declining level of support.

Firms would have been significantly worse off without the program funding

Over the course of the program implementation period, more firms indicated they were pessimistic about their odds of survival. As reported in the final reports after each round of funding, the percentage of firms indicating they 'likely' or 'definitely' would not have survived without program funding increased from 44% in round 1.0, to 58% in round 2.0 and 60% in round 2.5. Also, IAP recipients consulted as part of the evaluation confirmed their situation would have been significantly worse without the funding.

Intended outcomes

The program exceeded its targets in terms of number of firms and jobs supported. The program successfully minimized negative impacts of the pandemic on supported firms. The Innovation Assistance Program provided the bridge funding needed by firms to continue regular operations, pursue planned growth trajectories or to adjust. Some firms were able to seize opportunities created by the pandemic.

Firms and jobs supported

Source(s): Document review, data review

The program allocated almost all of its funding envelope

The program was not designed or delivered to be an entitlement program and had a specific funding envelope within which to manage. The program successfully allocated 100% of its funding envelope for round 1.0 ($250 million), as well as the majority of funding committed for rounds 2.0 and 2.5 ($127 million out of $155 million or 82%).

The objective of targeting emerging firms was also achieved. The characteristics of funded firms were aligned with the objectives of the program. Also, the program funding distribution was regionally balanced both in comparison to the distribution of applications and to the historical distribution of NRC IRAP funding support.

The program exceeded its outcome targets for the number of firms and number of jobs supported

The IAP exceeded its targets in terms of the number of unique firms supported (exceeded by 123%) and of the number of jobs supported (exceeded by 11%). A total of 2,230 unique firms were supported through round 1.0. Subsequent rounds further targeted smaller firms (already included in round 1) with a dedicated R&D component:

• Round 2.0: 1,384 firms; 9,009 individual jobs;
• Round 2.5: 1,269 firms; 7,242 individual jobs.

"IAP helped us to bridge the gap. We were able to complete what we aimed to do [and] carry on as usual."

The program exceeded its expected immediate outcomes

Demonstrated immediate impacts

Source(s): Data review, internal and external interviews, focus groups, case studies

The IAP helped firms avoid the need to implement additional defensive measures in response to COVID

The program helped firms avoid lay-offs, re-hire staff that were temporarily laid-off, and avoid a decline in business long enough for operations to stabilize. Maintaining staff was especially key. Also, re-hiring or recruiting staff to fill highly specialized positions would have been difficult for some firms.

A majority of supported firms reported that the program helped them remain liquid (92%), and 77% agreed that the duration of the funding allowed their company to sustain business operations through the pandemic uncertainty. Over 80% of IAP-supported firms did not need to decrease their staffing levels during the pandemic (53% reported an increase in the number of active employees and 28% reported unchanged staffing levels).

Close to 71% of program recipients who provided final reports gave the program a high score (8, 9, or 10 out of 10) when asked about the extent to which the program allowed their business to maintain its regular business. This feedback was especially positive for firms with less than 10 employees.

93% of firms applaud the 80% up-front claim payment approach, indicating it helped them remain liquid.

The duration of IAP funding was sufficient to support firms through the worst of the crisis, but they continue to face challenges

Firms did not face major consequences when program funding ended, but about one in five continue to face challenges associated with the pandemic. These difficulties include continuing supply chain disruptions, limits on travel, reduced market demand, impacts on partners, and difficulty recruiting and retaining staff.

Additional research would be required to further confirm the survival rate of IAP-supported firms

Amongst the supported firms who participated in the post-funding assessment (a 62% response rate), 98% were still operational as of August 2021. The evaluation did not have a comparison group of non-IAP firms (including unfunded IAP applicants) with similar profiles with which to compare results; further the evaluation could not confirm the status of the remaining 38% of supported clients.

"Since the end of the program, we are slightly more careful — we monitor the situation very carefully. There is still a risk of staff layoff for us. We are doing everything we can, but there is a chance that we'll need to let go of 1 or 2 people."

Firms are contributing to Canada's economic recovery

Source(s): Data review, document review (Statistics Canada, Conference Board Canada), internal interviews, focus groups, case studies

Program funding enabled firms to secure additional funds or investments

Companies were able to pivot to maintain operations over the time they received program funding, so that existing or alternative business plans could be carried out, and in some cases, new product lines developed. Program funding enabled firms to secure additional funds or investments because they were able to sustain the staffing levels required to maintain or grow operations.

59% of firms who responded to the post-funding assessment indicated that their firm's revenue grew between 2020 and 2021.

The program enabled the attraction of private equity and other types of funding (e.g., provincial research and Natural Sciences and Engineering Research Council of Canada [NSERC] program funding). For example, some case study firms were able to demonstrate enough potential value to attract investments.

IAP-supported firms were expected to have the potential to recover and achieve meaningful economic impact. Selection criteria also considered the best prospects for future growth and success post-COVID-19.

Program funding enabled continued innovation

IAP funding had positive impacts on the innovation capacity of firms during the pandemic. 41% of round 1.0 firms and 86% of both round 2.0 and round 2.5 firms were able to continue their R&D activities. This is compared to just 19% of innovative Canadian SMEs who stated the same in response to a national survey by Statistics Canada. IAP-funded firms also significantly outperformed other innovative SMEs in maintenance of overall levels of staff and R&D investments (86% vs. 36%) and in investments into new product development or improvements (72% vs 6%).

"IAP funding allowed the company to take pause, explore other business avenues and strategically plan for a longer-term goal, as opposed to closing down or having to settle for the first direction that would generate revenue."

Source(s): Data review, external and internal interviews, case studies, Input-Output simulation

Most IAP firms, across regions, were able to maintain revenue and active employees

Program funding had the immediate desired effect of allowing firms to retain staff and maintain operations. Over 62% of IAP-supported firms at least maintained their revenues between 2020 and 2021.

Firms supported by the program contributed to Canada's economic recovery

Workers supported by the IAP created additional value within the Canadian economy. Based on Statistics Canada input-output simulations, the total economic value generated by the $373.8 million IAP wage subsidy investment was more than $1.5 billion in goods and services.

This total value generated a total direct and indirect impact of $867.7 million on GDP. For every dollar of wage subsidy provided, $2.32 was contributed in economic value to Canada's GDP. Based on the North American Industry Classification System (NAICS), IAP funded firms in three sectors each contributed more than $100 million to GDP.

• Total impact on GDP $867.7 million
• Ratio of value generated to GDP 1 : 2.32

The professional, scientific and technical services sector generated the most economic impact on Canada's GDP

Firms are contributing to innovation in technology sectors

Source(s): Case studies

Case study examples of firms contributing to the marketing technology and health economic sectors

Interactive Marketing Software

Firm 1 is a small, four person, female-owned firm that was facing significant financial challenges (the entrepreneur was using personal finances) at the time the pandemic shut things down. Spending was halted across the marketing technology industry.

IAP funding allowed the firm to keep staff employed and generate sales, enough that the firm survived to develop a good investment portfolio that attracted potential investors and the firm obtained venture capital funding.

Healthcare Wearable Technology

Firm 2 is a pre-revenue company (no sales) developing smart, wearable computer devices targeted to the health sector.

During the COVID-19 pandemic, product testing was significantly delayed and then put on hiatus since target users were focused on addressing front-line challenges brought on by the pandemic. As a result, the firm experienced significant cash-flow issues.

IAP support allowed the firm to avoid layoffs, continue with R&D and improve their product. By the end of round 1.0, the firm filed a patent, and increased staff from three employees on their original application to nine employees (including summer students).

Healthcare Billing Software

Firm 3 had been in operation for less than a year when the pandemic hit. It did not qualify for other federal government emergency funding. Because this firm was in the process of scaling up and solidifying contracts with clinics and hospitals when the pandemic started, its ability to generate revenue was greatly impacted. All non-essential business in hospitals and clinics was halted so the firm no longer had access to decision-makers.

With IAP funding giving them time to explore other business avenues, the company discovered that a healthcare platform such as theirs was needed in Africa, which would allow medical billing to occur in a safe, controlled manner. Using the same technology, the company pivoted to marketing their software in Nigeria.

Unanticipated outcomes

Most IAP-funded firms already had an existing relationship with NRC IRAP or an ITA; however, 140 IAP-supported firms who had not previously received NRC IRAP funding or advisory services did become NRC IRAP clients following IAP. About 7% of respondent firms were exposed to NRC IRAP for the first time through the Innovation Assistance Program.

Increased NRC IRAP project activity and expectations

Source(s): Data review, external and internal interviews, case studies

Most program recipient firms were already NRC IRAP clients, or at least aware of NRC IRAP

61% of IAP-funded firms had benefited from previous interactions with NRC IRAP post 2011. These firms had either received project funding or benefitted from advisory services. Because of these existing connections, the ITAs were in an excellent position to inform firms about the program and messaging to firms was greatly appreciated.

Although the delivery of IAP was focused on 'getting money out the door quickly', ITAs were able to give additional support to IAP recipients, primarily by sharing information and referring them to other relevant programs.

IAP-supported firms became new NRC IRAP clients

Although not intended, the program did bring new clients to NRC IRAP and enabled ITAs to "get to know" new firms. There were 147 IAP-supported firms that became new IRAP clients in fiscal years 2020-2021 or 2021-2022, receiving either project funding, advisory services, or both. According to the post-funding assessment, 22 firms that indicated they were 'not at all' aware of NRC IRAP before receiving IAP support became NRC IRAP clients.

The program had to manage some expectation for NRC IRAP funding

The IAP may have inadvertently led some firms to believe they could expect NRC IRAP funding. IAP was delivered as a needs-based subsidy program, with the clear objective of providing support to firms in the quickest, most efficient and straightforward way possible. Eligibility for the program was based on a simple demonstration of need and a closed set of criteria. Some firms incorrectly assumed that by receiving IAP funding they were becoming IRAP clients and would also then qualify for NRC IRAP's discretionary funding.

There was some lack of clarity surrounding the differentiation between IAP and regular IRAP, and some firms did not clearly understand that IAP and IRAP were not the same program, and didn't have the same eligibility criteria for funding decisions.

"IAP was not discretionary, it was mechanical. […] We had to fight that IRAP battle again, trying to explain IRAP and moderate expectations about what firms can hope to get from that program. Saying no is that much harder."

Program design and delivery

The program was effectively delivered through clear operational guidance, effective training of staff, good communication, and support provided throughout implementation and delivery. NRC IRAP's existing information technology (IT) systems, operating procedures, funding authorities, and extensive ITA network were all assets in delivering the program and rapidly deploying funds. The dedication of NRC IRAP employees was a key success factor; however, the demand on staff was significant. Human resources were stretched thin to deliver the Innovation Assistance Program on top of other duties, and having to manage stacking issues, in particular, generated unexpected additional work.

Adequate design

Source(s): Data review, document review, external and internal interviews, case studies

The IAP was delivered as intended, within the parameters defined for the program

There were no major differences between design and delivery for the program. Overall, the approach used to select firms in round 1.0 was efficient and targeted the right firms based on IAP's objectives, given the diversity of firm profiles and sectors. An early assessment by the NRC OAE's Internal Audit team indicated that NRC IRAP had established the right controls to ensure due diligence and sound stewardship of funds while delivering at unprecedented speed. Funded firm characteristics are in line with the profile of firms targeted by the program.

Some processes were adjusted between the funding rounds, and delivery became increasingly efficient as staff grew more familiar with terms and expectations. While round 1.0 was a broad and open call, for round 2.0, the funding was further targeted to firms with ongoing need and dedicated R&D activities; round 2.5 introduced a revenue cap.

The invitation approach for round 2.0 was considered efficient, but may have excluded certain firms from receiving needed support

Awarding round 2.0 funding only to recipients of round 1.0 was a decision based on the assumption that the majority of round 1.0 funding recipients would qualify for additional support. Interviewees generally agreed that allocating round 2.0 and 2.5 only to round 1.0 recipients was a sound approach under the circumstances as the program had reached the right clientele with round 1.0.

However, it was recognized that some firms who may have missed round 1.0 for a variety of reasons (e.g., were not aware of the program, lacked the time or capacity to apply at the time) or had their circumstance change between rounds, could have benefited from round 2.0 funding. This remains hypothetical since the evaluation cannot assess the number of firms in need that may not have applied to round 1.0 but the point was raised internally and externally by key informants.

Generally, communication and engagement activities were noted to be effective. However, there was potential for more non-NRC IRAP clients to have potentially been reached for round 1.0 if NRC IRAP had conducted additional promotion and outreach through incubators and other intermediaries (e.g., via communication through accelerators, incubators, associations and intermediaries). Again this was raised internally and externally by key informants.

Successful delivery of the Innovation Assistance Program

Source(s): Document review, data review, external and internal interviews, focus groups, case studies

Delivering the IAP as a contribution program required existing expertise and experience in grants and contributions in order to meet imposed timelines. NRC IRAP was simultaneously responding to pandemic needs internally at the time, ramping up work-from-home capacity and adjusting to a new client service delivery model.

NRC IRAP effectively delivered the program

There was 100% agreement (among both internal and external stakeholders) that NRC IRAP effectively delivered the IAP program. This is especially true considering the pandemic context and point-in-time at which the program was launched. Training, operational guidance, and effective communications supported internal implementation and delivery.

Although the program reported some challenges in managing the volume of enquiries that followed round 1.0 decisions, overall communications to firms were timely and clear. Recipients described the program processes, including application and reporting, as streamlined, efficient and straightforward.

"By far the fastest I've seen government get dollars out the door!"

NRC IRAP's existing structures and tools were key to the rapid and efficient delivery of the program

The program was delivered effectively and efficiently as it leveraged existing NRC IRAP program infrastructures and established protocols. The program leveraged IRAP's existing IT system and NRC IRAP's proven contribution agreement procedures and authorities.

Existing relationships between firms and ITAs, as well as NRC IRAP's previous experience launching as-needed temporary programming also facilitated the rapid deployment of program funds.

Challenges in delivering the Innovation Assistance Program

Source(s): Interviews, case studies, focus groups

The demand on NRC IRAP staff was significant

IRAP staff were proud to have served in their positions during the crisis, and their commitment was praised both internally and externally. However, the delivery of the IAP did generate an unprecedented volume of work that challenged staff work-life balance and wellbeing, particularly in the absence of additional surge capacity. Standing up the program in just four business days, assessing more than 4,000 applications, developing contribution agreements for 2,000 firms and processing as many claims, while simultaneously fielding program inquiries all in a very tight timeframe put significant pressure on staff.

The program disrupted normal NRC IRAP program delivery to some extent

As the program was prioritized across the division, some NRC IRAP projects were delayed and, as a result, there were instances of lapsed NRC IRAP funds. Where IAP recipients were also NRC IRAP clients, firms were encouraged to access the IAP funding for the full eligible amount which led to underspending on NRC IRAP projects (where the same positions were being covered).

Having to manage stacking limits added to staff workload

The existence of a number of pandemic-related programs running concurrently led to an unforeseen volume of work in managing federal stacking limits. IAP-recipient firms required assistance in determining whether they were respecting the stacking limits and each required an individual file examination. Training was conducted and tools were developed to assist with file review given the issues experienced in round 1.0.

Communication around the IAP funding could have been expanded

Following the Ministerial announcements of the program (both for round 1.0 and round 2.0), there were no further public information releases from NRC IRAP. Communications by ITAs were fairly targeted to existing NRC IRAP firms. Informational updates on the program extensions and on the rounds 2.0/2.5 selection criteria was limited. It was mainly communicated to a pre-selected subset of firms from round 1.0. Furthermore, no information was provided on the closure of the program prior to the removal of the IAP web page. Some regional NRC IRAP staff reported that they received information requests and complaints from firms that were not selected for round 2.0 or 2.5. In the end, the program received more than 800 inquires, including requests for more information, feedback and complaints. This volume of inquiries was described by senior staff as representing more than ten times the number received by the regular NRC IRAP program in any given fiscal year.

Government-wide communications and coordination between various emergency programs could have been improved

Communications, timing of announcements and information sources on emergency supports provided by the Government of Canada broadly, were noted to be difficult to navigate. Stakeholders identified a need for timely and coordinated information about available government emergency supports to reduce the stress on businesses, and maximize uptake of programs by those in need.

Lessons learned

Lessons learned from the design and delivery of IAP

Source(s): Data review, document review, external and internal interviews, focus groups, case studies

The following lessons learned were identified during the evaluation of the IAP:

1. An existing program structure that has adaptability and agility is key to supporting the rapid design and delivery of emergency programming.

   Recognizing that existing systems and structures can be used to assist with emergency situations enables faster response to meet critical needs. NRC IRAP's infrastructure and in-house expertise were key to the efficient design and implementation of the IAP. Staff experience with grant and contribution transfer payment programs was particularly beneficial as they were able to be engaged without training, ensuring due diligence of process and good stewardship of funds.

2. Without the ability to engage temporary resources to provide surge capacity, the delivery of emergency programming creates significant pressures on existing staff, challenging mental health, wellbeing and work-life balance.

   Consideration for additional temporary resources is a must to ensure a healthy workplace during the delivery of emergency programming. Under IAP, the workload of NRC IRAP staff was increased by an unforeseen volume of work associated with the management of federal stacking limits for pandemic support programming and the large influx of inquiries about the program received. As a result, regular program delivery was disrupted.

3. Employment of diverse tools allows firms to tailor their own response to an emergency in ways that best respond to individual situations.

   IAP wage support plus other liquidity support options worked together to meet business needs. Many different tools were necessary to address cash flows issues.

4. Limiting flexibility in programming (e.g., not allowing changes to program eligibility criteria) restricts the program's ability to adapt to a changing environment during an emergency.

   Restricting access to a program with a "one time only" opportunity to apply limits the program's ability to meet critical unforeseen needs that surface over time. The pre-defined budget of the program, in contrast to the flexible funding of the CRA CEWS program, limited the ability to plan for additional options to enable firms to access emergency support at a later point in time. The pre-defined budget also limited "second chance" applications with adjustments for adequate entitlement coverage in the event that increased financial support became available.

5. Coordination of communications and dissemination vehicles among government departments and agencies is important for informing targeted audiences.

   Coordinated communications can make a difference in how audiences experience and navigate difficult situations. Without an existing connection into a government department, SMEs found it difficult to determine what programming was available and what programming they might qualify for.

6. Examining the impact of a program on different groups is an important element for informing the design of future temporary programming targeting Canadian firms.

   Without complete information on the sustainability of outcomes for recipients (e.g., survival rate) and in absence of reliable GBA Plus data, it is difficult to identify key barriers to access and lessons learned. Such information is needed to understand how future programs could be designed to better reach and impact firms with differing needs.

7. Administrating an emergency wage subsidy program based on a predetermined funding envelope and set of criteria creates challenges for dispersing maximum support without exceeding the budget.

   Needs-based programming support, based on specific criteria, makes it difficult to ensure maximum support is provided to all eligible firms without going over the fixed budget envelope. Decisions need be taken outside of the initial application eligibility criteria to avoid leaving unassigned funds.

Recommendations and Management Response and Action Plan

Recommendations and supporting rationale

Communications

Supporting rationale

IAP raised expectations for establishing NRC IRAP projects among firms that were not (yet) eligible, creating a temporary need for NRC IRAP to manage expectations vis-à-vis regular NRC IRAP programming with some IAP recipients. Also, IAP did not have a communication plan to inform firms of the multiple rounds (as round 2.0 was only communicated to pre-selected firms from round 1.0) and closure of this temporary program, and to help address these expectations.

A plan on how to better communicate to clients in the event of future emergency or temporary programming is needed. This could be in the form of official program update and wrap-up information, delivered through a formal communications plan and tools, or activities included in the field manual.

Capacity

Supporting rationale

With the delivery of IAP, the need to manage complexities of funding stacking generated unexpected additional work. Under very tight timeframes, both staff and program recipients experienced challenges in managing the stacking rules. NRC IRAP should take advantage of this experience in determining how to best balance support to program recipients with the workload generated by the stacking assessment, and develop guidance and/or training materials and tools for ITAs.

This material could be included in the field manual and be updated on a regular basis by scanning the stacking rules of other complementary innovation support programs, and it could provide timely information that can be used for regular activities or in the context of new programming.

GBA Plus

Supporting rationale

GBA Plus was not an original consideration of the IAP design and as a result only partial GBA Plus-related data was collected. This prevented the NRC from being able to properly categorize IAP supported firms and thus better understand key barriers to accessibility and the differential impact of the program on diverse clients. NRC IRAP recently implemented a GBA Plus initiative in its core programming. IRAP could capitalize on the capabilities developed through this initiative to proactively embed GBA Plus-related data collection into all current and future programming.

Further, with a view to further develop its GBA Plus capabilities, NRC IRAP could examine the best practices, challenges and lessons learned experienced by other federal funding programs for Canadian businesses in capturing and considering GBA Plus variables. This may provide key information that can be used to adjust the NRC's interventions, program designs and communication material moving forward.

Outcomes

Supporting rationale

Limited information is available about IAP-supported firms that are known to have not survived the pandemic. Targeted follow-up with firms who responded to the post-funding assessment as not surviving would be needed to better understand the barriers and identify lessons learned and trends, if any. This information would be useful to NRC IRAP given that the pandemic continues to affect Canadian SMEs and international markets.

Also, a follow-up exercise would be needed to fully capture the outcomes of IAP, including establishing the survival rate of all IAP funded firms (e.g., in two years). This analysis would benefit from including a comparative analysis with control groups composed of unfunded IAP applicants.

Management Response and Action Plan

Appendices

Appendix A – Methods

Data review

The data review covered IAP administrative data (e.g., client data). The data files provided by NRC IRAP were numerous and included many variables for review. In some instances, administrative data files provided by NRC IRAP needed to be linked in order to complete a full analysis. Variables provided ranged from data provided directly by firms, data linked to existing project data, and categorization data developed by NRC IRAP.

IAP final report data (all three rounds), and the results of the post-funding assessment administered by NRC IRAP were also analyzed:

• Final reports: At the end of each round of subsidy, firms were required to complete final reports in order to support program efforts to assess the effectiveness of the IAP, as well as gather feedback on the relevance, timeliness and accuracy of NRC IRAP services. The number of firms submitting a final report was very high for each round, reaching a response rate close to 100% in each instance. The final reports were a significant line of evidence for the evaluation.
• Post-funding assessment: A post-funding assessment was completed by firms on a voluntary basis in August/September 2021 to inform the overall evaluation process. The assessment allowed the evaluation team and NRC IRAP to gather some new information from firms as well as update previously collected final report data. 62% of funded firms completed the post-funding assessment.

Document review

IAP program documents were examined to obtain a clear understanding of how IAP operated over the two (and a half) rounds of funding. The review included program design documentation, operational policy guidance, instructions and guidance for ITAs and recipients, quality assurance documentation, NRC IRAP and IAP-related reporting and the initial audit of IAP conducted by NRC's Office of Audit and Evaluation.

Relevant external documents were also reviewed, including documentation from other government sources and OECD documentation about COVID-19 support to SMEs.

Interviews

A total of 40 interviews were conducted between July 21st, 2021 and September 14th, 2021, including 36 internal NRC IRAP interviews and 4 external interviews.

Of 30 regional interviews, 16 were ITAs interviewed in the context of case studies, but who also commented on the delivery of IAP as a whole. Those case study lead ITAs were asked questions about their respective cases, but also a distinct set of questions about the delivery of IAP in general.

Regional distribution of stakeholders consulted by respondent group

Regional distribution of stakeholders consulted by respondent group

Group	HQ/Nat	ATL	QC	ON	PRA	PAC	Total	%
Interviewees	9	6	7	7	5	6	40	73%
Focus group participants		3	3	5	0	4	15	27%
Total	9	9	10	12	5	10	55	100%
%	16%	16%	18%	22%	9%	18%	100%

Case studies of funded firms

A sample of 15 case studies was developed using a purposive sampling approach in order to achieve a balance across the following criteria:

• Three cases from each region (Atlantic, Ontario, Pacific, Prairies, Quebec);
• One each of high, median, and low dollar value contribution agreements in each region;
• A random mix of recipients that received IAP round 1.0 funding and then a group that continued on to receive round 2.0/2.5; and,
• A random combination of IAP-only recipients and those that were existing NRC IRAP clients.

Clients were grouped by region, then random selections were chosen by using the first company appearing in a regional list, by high, median, and low levels, by funding round, and by NRC IRAP versus non-NRC IRAP clients, until all 15 spots were filled.

Client focus groups

Virtual focus groups were held in September 2021 with three groups of IAP clients, distinguishing between IAP-supported firms that were already NRC IRAP clients and those that were new to NRC IRAP.

For NRC IRAP clients, one group included firms that received only one round of IAP funding (n=4), and another included those that received multiple rounds of funding (n=5). For IAP-only clients, the group was comprised of 6 firms that had received multiple rounds of IAP funding.

Economic analysis

Input-output models are used to simulate economic impacts of a given expenditure and to estimate the change in economic activity caused by an economic event. In this case, the expenditures of the IAP are translated to portray the direct and indirect effect of program expenditures at the macroeconomic level in terms of economic activity and ripple effects throughout the economy. The simulation (a change in inputs to the economy) shows the results of a "shock" of spending.

The Statistics Canada open-model results were used to demonstrate the immediate economic impact of the IAP. The open-model uses direct and indirect impacts to arrive at total impact. Direct and indirect impact measures are defined as:

• Direct impact measures the initial results for an extra dollar's worth of output (money initially spent) of a given industry (e.g., money spent for salaries, supplies, or operating expenses). Associated with this change, there will be direct impacts on GDP, jobs, and imports.
• Indirect impact measures the changes due to inter-industry purchases as they respond to the new demands of the directly affected industries (e.g., additional activity in the local economy based on the initial spending). This is the benefit to those businesses subsequently impacted by increased spending as a result of business-to-business activity.

Business number registration (CRA) were used to match to NAICS codes assigned to IAP recipients by Statistics Canada in order to enable the analysis of impact by sector and sub-sector.

Statistics Canada used the expenditure data provided by NRC IRAP to the Parliamentary Budget Office to run the I-O model (i.e., $373,793,328), produce analysis, and then provided the result to the Office of Audit and Evaluation which in turn was used to produce this report.

Limitations and mitigation strategies

Although the evaluation encountered some challenges, methodological limitations were mitigated, where possible, through the use of multiple lines of evidence and the triangulation of data. This approach was taken to establish the reliability and validity of the findings and to ensure that conclusions and recommendations were based on objective and documented evidence. Details on limitations and their associated mitigation strategies are described below.

Inability to confirm survival of funded firms

Based on the information available, it was not possible to know exactly (without CRA data) how many IAP-supported firms survived COVID-19. The post-funding assessment was distributed to all IAP-supported firms, and received a 62% response rate, with 98% of those firms indicating survival. The evaluation cannot confirm the status of the remaining 38% of firms that did not respond.

Mitigation: With a very high percentage of respondent firms to the post-funding assessment indicating they have survived post IAP funding, statistically, if the entire remaining 38% of non-respondent firms did indicate they did not survive, along with the 2% of post-funding assessment firms that indicated they in fact did not survive, the largest possible portion of firms not surviving would be 39%. This is highly unlikely. NRC IRAP indicates some firms have since reached out to provide input as they missed the deadline for the post-funding assessment. The evaluation is confident that the number of firms that survived is larger than reported in the post-funding assessment, but at minimum 61%.

Difficulty in recruiting IAP-only clients for focus groups

One of the focus group target audiences were firms that had received program funding only, who were not already NRC IRAP clients. It was not possible to recruit participants for a focus group with IAP-only recipients that received only round 1.0 funding, as originally planned due to the random selection of the sample frame and the inaccuracy of the recipient lists provided. Also, the focus group intended for IAP-only recipients (multiple rounds) turned out to include clients who already had contact with NRC IRAP, or were very knowledgeable about NRC IRAP. This means that views from firms without existing NRC IRAP relationships are minimal in the focus groups.

Mitigation: Case study interviews with program recipients did include known IAP-only recipients. The qualitative evidence gathered from the case studies is not different from that collected during the focus groups.

Inherent limitations to I-O model analysis

Generally, there are limitations and considerations to the empirical shortcomings of I-O models and analysis. Key limitations of both the closed and open I-O models include:

• They are theoretical models and are based on consistency analysis (i.e., a linear function). The model does not account for changed conditions.
• They presume that purchases are taken as a given and does not consider available supply.

Mitigation: While there is no mitigation to these inherent limitations, Statistics Canada provided guidance to use the results of the open model 1) to be conservative, and 2) to reduce the likelihood of double counting where induced impacts, in response to consumer expenditures, can tend to be overestimated in the closed model.

Appendix B – Details on the IAP qualification questions

IAP Round 1.0 merit-based qualification questions and impact on rating

Qualification question	Impact on rating
1. Which of the following statement best applies to your business?	Places business in one of the three funding streams, or in an unfunded stream: Funded Stream 1: Business is pre-revenue. Funded Stream 2: Business is generating revenue less than $1 million/year. Funded Stream 3: Business is scaling rapidly and uses sales revenue to hire operational staff. Unfunded streams: No T4 salaried staff; has T4 salaried staff, but not critical to operations; business has significant market traction.
2. Is there a formal engineering, R&D or new product innovation team within your business?	Businesses that rank higher in innovation are more likely to get funded.
3. Describe your business's ability to execute innovation-based commercialization.	Businesses that have demonstrated the capacity to develop and launch new products and generate sales are more likely to be funded.
4. Which of the following best describes your business's sales patterns? (To characterize sales patterns from Canadian, North American and international clients.)	Businesses that have demonstrated an export focus are more likely to be funded.
5. Tell us specifically about your business's supply chain, the role your business plays in the supply chain, and the impact of your activities. Business impact on supply chain.	Businesses that appear to be better positioned to adapt to post-COVID-19 market conditions are more likely to be funded.
6. Tell us specifically about how your business is planning to adapt to post COVID-19 market disruptions or changes.	Businesses that are able to leverage funding and execute on strategy are more likely to be funded.
7. If you receive financial assistance, to what extent would your business be capable of conducting its operations amidst physical-distancing?	Businesses that are most in need of financial support are more likely to be funded.