Evaluation of the Industrial Research Assistance Program (IRAP)

Acronyms

AGS	Accelerate Growth Service
BIAP	Business Innovation Access Program
BMS	Business Management Support
CAGR	Compound annual growth rate
CAIP	Canada Accelerator and Incubator Program
CTF	Contribution to Firms
CTO	Contribution to Organizations
DTAPP	Digital Technology Adoption Pilot
FHI	Fairly high impact
HI	High impact
HIV	Human immunodeficiency virus
IA	Innovation Advisor
IRAP	Industrial Research Assistance Program
ITA	Industrial Technology Advisor
ICT	Information Communication Tools
MOU	Memorandum of understanding
NPV	Net present value
NRC	National Research Council Canada
PBCA	Partial benefit-cost analysis
R&D	Research and development
RCAO	Regional Contribution Agreement Officer
RIO	Regional Information Officer
SLT	Senior Leadership Team
SME	Small and medium sized enterprise
UK	United Kingdom
US	United States

Executive Summary

This report presents the results of the 2016-17 evaluation of the Industrial Research Assistance Program (IRAP). It was conducted based on the five-year evaluation cycle mandated by the Financial Administration Act and in accordance with NRC's Evaluation Plan. It examined the relevance and performance of IRAP between fiscal years 2012-13 and 2016-17.

IRAP is an NRC program aimed at accelerating the growth of SMEs and stimulating wealth creation for Canada. To achieve this, IRAP provides advisory services and funding to SMEs and well as funding to organizations that help SMEs. Overall, the findings of the evaluation confirm that IRAP addressed the needs of Canadian SMEs for support to be innovative and that it played an important role in strengthening the innovation capacity and accelerated growth of its clients. While there are areas for improvement, IRAP worked to deliver the program efficiently and effectively. Implementation of recommendations will facilitate improvements to its efficiency and effectiveness. The Concierge service was also explored during this evaluation. Recommendations about the Concierge service aim to formalize its delivery approach and maximize efficiencies.

Findings, assessment, recommendations as well as the management responses are summarized below under each of the key findings from the evaluation.

Table 1: Key Findings, Assessment, Recommendations and Management Response

Key Findings: Rationale for IRAP	Assessment
Need for IRAP
Evidence from the evaluation confirms that Canadian SME's required access to funding, business advice, and technical advice because they often lack these resources. They required this assistance in order to have the capacity to prioritize innovation. The federal government was strongly supportive of increased business innovation and this has been a cornerstone of the policies of both the previous and current federal governments. The federal government's role in supporting SME innovation was sound. Federal support is required because of the importance and magnitude of the challenge in stimulating innovation, because of the economic rationale for government support, because of the arms-length position of the government and for the continued competitiveness of Canada.	IRAP addressed the need for SME innovation support in Canada in a way that is consistent with government priorities and the role of government.
Role of IRAP
The complementarity of offering advisory services and funding together is the essence of IRAP; this is unique and is critical in supporting successful SME innovation. While there are other programs that provide funding to SMEs and other programs that provide advisory assistance, the combination of providing both is unique. IRAP programs addressed needs across the innovation spectrum by offering a wide range of technical and business advice as well as funding to address specific needs. IRAP and other federal and provincial programs leveraged one another to meet the needs of SMEs. This is done through delivery of various programs on behalf of other government organizations as well as coordinating support efforts informally, and formally through the Accelerated Growth Service. There was some collaboration between IRAP and NRC research portfolios; however, misalignment with their individual mandates makes client SME use of NRC facilities and expertise challenging. IRAP is taking steps to facilitate more collaboration.	IRAP played an important and unique role in supporting SME innovation in Canada.
Reach of IRAP
IRAP worked with SMEs and organizations across all regions and in a variety of sectors. However this represents only a small proportion of innovative SMEs in Canada. Many IRAP clients came from previous relationships or referrals; therefore IRAP conducted very little outreach activities to identify new clients. Evidence found that there is potential to find new clients, who face barriers to accessing IRAP. Fully understanding the current reach of IRAP remained a challenge. This is due to advisory services provided to funded and unfunded firms and firms that are assisted through funded organizations not being tracked consistently.	IRAP depended heavily on its existing network of clients and has an opportunity to develop a targeted strategy to reach new clients as well as to better understand its reach.
Increased innovation capacity of SMEs
Advisory services and funding strengthened the innovation capacity of firms. Advisory services helped to identify the needs of firms and provide the best support available to meet needs and reach objectives. Funding provided firms with the means to undertake innovative projects that would not have gone ahead or would have been significantly affected in the absence of IRAP funding. Funded organizations added a niche the mainstream IRAP does not fully address: start-ups and very small SMEs. Many services provided by funded organizations target these newer firms. SME clients of organizations found these services to be beneficial.	IRAP played an important role in strengthening the innovation capacity of many SMEs in Canada.
Growth of SMEs
IRAP facilitated SME growth, through a variety of internal mechanisms tailored to the individual SME's needs. When speaking with funded firms through the interviews and site visits, many found it difficult to quantify the success or growth that they could attribute to IRAP. However, these clients estimated that IRAP had a significant part in their success. Evidence of growth was also noted in the partial benefit cost analysis and through analysis of SME data. Further, many SMEs assisted through funded organizations increased their productivity, growth, or profitability as a result of the assistance they receive from these organizations.	IRAP facilitated the increased growth of many SMEs in Canada
Contribution to the Canadian economy
IRAP resulted in positive economic benefits. This is evident from the partial benefit cost analysis that was conducted for the evaluation which demonstrates that the benefits far outweigh the cost of the program. In fact, taking the average value between the conservative and optimistic assumptions used in the analysis and between the three discount rates used: IRAP's economic benefits are at least $10 billion over and above the cost of the program from 2005-06 through 2015-16; and The ratio of IRAP's economic benefits to the cost of the program during this period is at least 4.9/1.	IRAP-funded firms contributed positively to the Canadian economy
Delivery
IRAP had structures and process in place to manage the IRAP program such as guidelines, service standards, a web-based management system and client portal. Further improvements to these could increase efficiencies. The Dialogue exercise undertaken at NRC identified similar priority areas for improvement. Recommendations from this evaluation align with these priorities.	Delivery of IRAP could be improved through increased efficiencies
Resources
Human resources did not grow at the same rate as the various programs and services being delivered by IRAP, which increased the burden on its staff. IRAP recognized this issue and that it is a risk to the continued delivery of the program. They are reviewing their delivery model to ensure consistent and appropriate delivery of all program elements. There was a lack of appropriate ICT tools and support, which decreases efficiency and effectiveness. The majority of the internal interviewees indicated that ICT tools did not facilitate efficient and effective program delivery, particularly for ITAs who often work remotely. The lack of IT support is also a significant issue for IRAP, impacting their capacity to work effectively.	Resource constraints continued to pose barriers to efficient and effective delivery
Meeting Needs
Concierge supported SMEs' ability to find innovation programs that were most appropriate to their needs. The Concierge program was designed following recommendations of the Jenkins report. The report outlined the need for a service to help SMEs navigate the innovation funding environment. There were many challenges in implementing and delivering Concierge, largely due to a mismatch between resources allocated and expectations for the program. Concierge was intended to be a national program; however it was allocated insufficient resources to have an impact of national scope. Concierge's delivery model evolved to best serve later stage, high potential clients. Despite challenges, many, including external experts, recognize the value of the expertise in Concierge. It evolved to maximise the impact of the program while using available resources. While the program remains national and for all SMEs, the IA time is increasingly limited to serving high potential growth firms.	Since it was first implemented, Concierge evolved in order to best meet needs.
Impacts
The reach of concierge was limited by insufficient human resources available to deliver the program. However those that accessed the program were satisfied with the services and found them useful. This is largely due to the Innovation Advisors who worked hard to create awareness and meet the needs of their clients.	Despite limited resources, Concierge had an impact on increasing SMEs' awareness and access to innovation support.

1. Introduction

In 2016-2017, an evaluation of the National Research Council's (NRC) Industrial Research Assistance Program (IRAP) was conducted. The evaluation examined the relevance and performance of IRAP between fiscal years 2012-13 and 2016-17. As mandated by the Financial Administration Act, an evaluation of IRAP is carried out once every five years.

The evaluation focuses on three key areas: rationale for IRAP, impact of the program and delivery of the program. The implementation and early impacts of the Concierge program were also assessed as part of this evaluation. Temporary programs, such as BIAP, DTAPP and CAIP were assessed separately and are not included in this evaluation. However their impacts on delivery were explored as a result of discussions with IRAP staff on program efficiencies. The evaluation was jointly conducted by NRC's Office of Audit and Evaluation and KPMG LLP. This hybrid approach ensured the capacity and required expertise to conduct all aspects of the study. The evaluation methodology leveraged multiple lines of evidence and complementary research methods to enhance the reliability and validity of the information collected. The methodology includes the following lines:

• A document and literature review
• An analysis of financial, administrative and performance data
• Interviews with IRAP staff and external subject matter experts
• Interviews with IRAP client firms who have received contribution funding
• Interviews with clients of the Concierge Service
• Site visits with IRAP client firms who have received contribution funding
• Focus groups with ITAs
• Focus groups with IRAP client firms (most of which have not received contribution funding)
• An on-line consultation of IRAP (advisory services) client firms who have not received contribution funding
• Case studies of funded organizations
• A partial benefit-cost analysis (PBCA) of IRAP

A more detailed description of the evaluation methodology including a description of challenges and limitations is provided in Appendix A.

The next section (Section 2) of this report provides a short profile of IRAP. Sections 3 through 6 present the findings of the evaluation, organized by key finding. Section 7 presents a brief conclusion drawn from the evaluation, while Section 8 lays out the management response to recommendations from the evaluation.

2. Profile of IRAP

Established in 1962, IRAP is an NRC program aimed at providing innovation assistance to Canadian small and medium sized enterprises (SMEs). The mandate of IRAP is to "stimulate wealth creation for Canada through innovation," and its mission is to "accelerate the growth of SMEs by providing them with a comprehensive suite of innovation services and funding."

To achieve its objectives, IRAP provides the following assistance:

• Advisory services: business and technical advice and referrals to SMEs through every aspect of the innovation process, from conceptualization to commercialization.
• Funding contributions to firms: These are non-repayable contributions to SMEs to support R&D and technology adaptation and/or adoption up to the point of pre-commercialization. Contributions to firms include core IRAP funds and temporary programs. Core IRAP funds, which include the Youth program, receive continuous funding. During the evaluation period, IRAP also delivered temporary programs, such as the Business Innovation Access Program (BIAP), the Digital Technology Adoption Pilot Program (DTAPP), the Canada Accelerator and Incubator Program (CAIP) and the HIV technology and vaccine development programs. Often, these temporary programs are delivered on behalf of other government departments and last between two and five years.
• Funding contributions to organizations: IRAP provides financial assistance to not-for-profit organizations (e.g., industry organizations, regional innovation centres, university-based technical organizations) to enable them to provide innovation services to (primarily small) SMEs.
• Support to other government departments: IRAP contributes its expertise to help others with their innovation support programs. This includes providing guidance on program design, due diligence in terms of conducting technical assessments of funding proposals, and client outreach through the Concierge service. This service provides a single access point where Canadian SMEs seeking innovation-related assistance can find information on funding, expertise, and facilities and equipment. IRAP is also a key player in the Accelerate Growth Service (AGS), a cross-government initiative targeting high-potential firms to help them access the necessary support programs to grow and become internationally competitive.

IRAP is delivered nationwide by a network of over 230 Industrial Technology Advisors (ITAs) located across five regions: Atlantic & Nunavut, Quebec, Ontario, Prairies and Pacific. These ITAs generally have engineering or science backgrounds, and many of them have had successful careers as entrepreneurs or senior managers with SMEs prior to joining IRAP.

The role of the ITAs is to:

• Identify SMEs that have the potential to benefit from IRAP's assistance
• Establish relationships with these SMEs, which begins by working with them to identify their needs and explore opportunities to grow and successfully develop and commercialize new products and services
• Work with these companies to provide advisory assistance and funding contributions that are best suited to support their growth and profitability

2.1 Governance

ITAs are supported in the delivery of IRAP by staff in regional offices located across Canada as well as the Division Services office. The Vice-President of IRAP reports to NRC's President, and has overall managerial accountability for the Program. The Vice-President is supported by five regional Executive Directors and the Executive Director of Division Services. Together they form the Senior Leadership Team (SLT), which makes program-wide decisions. They are further supported by regional directors, operational managers, Regional Contribution Agreement Officers (RCAO) and other administrative staff. Currently, IRAP is operating with 322 total staff. Data is presented in Table 1.

Table 1: IRAP human resources between 2012-13 and 2016-17

Job Type	2012-13	2013-14	2014-15	2015-16	2016-17
Management	22	24	26	26	29
ITA	210	203	225	222	232
Other	65	66	70	71	61
Total	297	293	321	319	322

Source: IRAP human resources business partner

2.2 Financials

Between 2012-13 and 2015-16, IRAP has distributed over $869 million in contributions to SMEs and organizations through core and temporary funds. While temporary programs are excluded from this evaluation, they are included in Table 2 to illustrate the full scope of funds delivered by IRAP.

Table 2: Financials (actuals) between 2012-13 and 2015-16 (in millions $)

Financials		2012-13	2013-14	2014-15	2015-16	Total
IRAP Core (Youth, Firms, Organizations and Concierge)	Contributions	170.87	189.41	188.09	203.07	751.43
	Operating Costs	45.47	48.58	54.25	54.12	202.42
	Total IRAP Core	216.34	237.99	242.35	257.18	953.86
Temporary (BIAP, DTAPP, CAIP, CHTD CHVI)	Contributions	26.54	37.88	21.73	32.00	118.10
	Operating Costs	2.79	2.86	0.3	0.5	6.45
	Total Temporary	29.33	40.71	22.04	32.50	124.56
Total		245.67	278.70	264.38	289.66	1,078.41

Source: IRAP comptroller

3 Rationale for IRAP

3.1 IRAP addressed the need for SME innovation support in Canada in a way that is consistent with government priorities and the role of government.

Canadian SMEs required access to funding, business advice, and technical advice because they lacked these resources. There is extensive literature supporting the need for IRAP. For example, the Jenkins Report, a comprehensive review of federal government support for R&D, noted that government assistance for SMEs is needed to help overcome the disadvantage of small scale and to lower the barriers to getting started^{Footnote 1}. The need for access to funding is also highlighted through interviews with funded firms, who emphasized that it is very difficult for SMEs to divert funds from their operational needs to support R&D, especially riskier projects, or to support explorations of different products or different markets. The previous evaluation of IRAP also makes this point^{Footnote 2}, as do a number of sources, such as the 2013 World Economic Forum, which found that Canada's greatest weakness was SMEs' limited access to financing and insufficient capacity to innovate^{Footnote 3}. Lastly, as far back as 2005, the federal government's Expert Panel on Commercialization found that SMEs often do not possess the business or technical skills to progress to later development stages and to private sector support^{Footnote 4}.

The federal government was strongly supportive of increased business innovation. Federal government support for business innovation has been a cornerstone of the policies of both the previous and current federal governments. The previous Minister of Innovation, Science, and Economic Development (then Industry Canada) stated: "Our overarching goals…are to help Canadian businesses grow, innovate, and export, so that they can spur economic development and create good quality jobs and wealth [including] to help small businesses grow through trade and innovation...^{Footnote 5}"

In its first budget the current government announced the Innovation Agenda as a key element of a long-term growth strategy for Canada. Through the Innovation Agenda, the government would define a new vision for Canada's economy: to build Canada as a centre of global innovation. The 2017 Budget confirmed its commitment to innovation: "Canada must do more to encourage innovation^{Footnote 6}." Further, IRAP is consistent with the federal government's Science and Technology Strategy. The Strategy states that "IRAP plays a large role in supporting the diffusion of science and technology to small - and medium-sized firms..." It also identifies IRAP as one of the key mechanisms through which the government will achieve its entrepreneurship-related policy goals^{Footnote 7}.

The federal government's role in supporting SME innovation is sound. This rationale is supported by four main elements.

The importance and magnitude of the challenge: The Jenkins report presents compelling evidence that Canada's prosperity will depend, more than ever, on an innovative economy, but that the challenge is significant: "Studies have repeatedly documented that business innovation in Canada lags behind other highly developed countries."^{Footnote 8}There are approximately 1.17 million SMEs in Canada, and, as discussed above, they have multi-faceted needs for innovation-related support.

The economic rationale for government support: Individual firms, particularly SMEs, cannot capture all the benefits of their investment in R&D. Some of the knowledge generated is picked up "for free" by other firms and is used by them to increase their productivity. This leads young firms to under-invest in R&D and innovation-related activities, and government support is needed to overcome this "market failure."

The arms-length position of government: The government is an impartial, objective, and credible player that can provide support to SMEs without the conflict of direct financial gain.

The competitiveness of Canada: All of Canada's major competitors provide programs to support SME innovation such as the US Small Business Innovation Research program, Innovate UK, and the Australian Entrepreneurs Program. These programs are generally not structured the same way as IRAP (e.g., many countries provide separate advisory and grant programs). Nevertheless, since the need for SME innovation support described above are not unique to Canada and there are no trade restrictions to providing government support that stops short of direct commercialization subsidies, other countries that can afford to provide this support do so; and Canada may be disadvantaged relative to these countries if it did not.

3.2 IRAP played an important and unique role in supporting SME Innovation in Canada

The complementarity of offering advisory services and funding together is the essence of IRAP; this is unique and is critical in supporting successful SME innovation. While there are other programs that provide funding to SMEs and other programs that provide advisory assistance, the combination of providing both, as well as how this is structured in IRAP (i.e., through relationships), is unique. In a document called How IRAP Works, it is noted that "The ITA-client relationship is built on trust and is fundamental to the IRAP business model. ITAs typically establish long-term relationships, acting as their client's mentors and advisors as they navigate the innovation cycle at different stages of growth^{Footnote 9}." This uniqueness was emphasized in the five focus groups with ITAs, who are very familiar with SME support programs in Canada, and also by the interviews of internal staff and external experts.

IRAP's provision of both advisory and funding assistance is not just a "distinguishing feature" of IRAP, but also a key feature of its success. Typically, it is the initial advisory assistance that enables the ITAs to tailor subsequent IRAP assistance to the clients' needs. This was clearly illustrated in five of the six site visit case studies and all 21 of the case studies conducted for the PBCA.

IRAP programs addressed needs across the innovation spectrum. IRAP provides innovation support to SMEs across the innovation spectrum in the following ways:

• Offering a wide range of services such as technical and business advice, funding contributions, support for employing post-secondary graduates, and programs that address specialized needs, such as the now sunset BIAP and DTAPP
• Providing assistance from early stage technical and business advice up to commercialization of products, services, and processes
• Serving all sectors
• Serving all sizes of SMEs
• Serving all regions

IRAP funded organizations also help IRAP further meet the needs of newer firms. While IRAP serves new as well as established firms, many tend to be more established. Funded organizations ensure that newer firms, that do not yet meet requirements for direct IRAP funding, are well supported. For example, the previous evaluation found that the IRAP-funded organizations tend to focus their assistance services on start-ups and very early stage SMEs, thereby addressing a gap that might otherwise exist in the IRAP spectrum of services. This was confirmed in the case studies with funded organizations carried out in this evaluation.

IRAP and other federal and provincial programs leveraged one another to meet the needs of SMEs. IRAP's formal arrangements with other government organizations include:

• Coordination with Employment and Social Development Canada regarding the delivery of the Youth Employment Program
• Working with other government departments on international programs designed to help Canadian firms compete globally. This includes the management of Canada's national office for the EUREKA program and partnering with Global Affairs Canada on the Canada Export Program and the Canadian International Innovation Program
• Memoranda of Understandings (MOU) with 11 federal departments and agencies for the technical assessments of innovation program funding proposals

Evidence from the focus groups with ITAs and internal interviews regarding interactions with other programs was generally positive. A number of ITAs reported that they had good working relationships and extensive networks with other organizations such as regional economic development agencies and that they coordinate efforts to look for the most appropriate assistance available. Internal interviewees supported this finding, noting that while there is some overlap with innovation funding available, IRAP works with these programs to ensure complementarity.

Through a new initiative, the Accelerated Growth Service (AGS), the ITAs now have the ability to gather other government agencies together, including Global Affairs Canada and the Business Development Bank of Canada, to coordinate their support programs to assist SMEs. Several of the focus group participants and internal interviewees positively commented on their experience with this initiative.

There was limited collaboration between IRAP and NRC research portfolios. Collaboration between IRAP and NRC research portfolios has long been a priority of NRC senior management. However, the 2012 IRAP evaluation found these collaborations to be "modest." In the survey of funded firms conducted for the last evaluation, 13% of funded firms reported that their ITA linked or referred them to an NRC institute. Barriers to collaboration that were cited included the nature of NRC's research projects (generally longer-term), administrative delays, and costs. The 2012 evaluation recommended that IRAP should "determine if additional mechanisms are required to foster synergies and reduce barriers to collaboration between SMEs and [NRC's research institutes]." In response, IRAP renewed its sector teams, assigning specific ITAs to interface with portfolios in order to facilitate more collaboration. It should also be noted, that at the time, the newly reorganized NRC to a research and technology organization (RTO) structure alone was anticipated to encourage improved collaboration between IRAP and portfolios. Whether either resulted in an increase in SME use of NRC expertise and facilities is unclear.

Similarly, the ITA participants in the focus groups conducted for the current evaluation had made little use of NRC's research expertise or facilities. A minority reported having obtained support from the research portfolios, but the majority did not. Some noted that portfolios were often not running on "business time" (i.e., timing consistent with industry needs) and/or were too academic in nature to meet their needs. Several ITAs commented that the NRC is not used to working with SMEs and that the processes (such as contracting and invoicing) are too slow and services are too expensive for SMEs.

Nevertheless, some of the internal interviewees provided positive examples of SMEs that significantly benefitted from working with the research portfolios. For example, one internal interviewee mentioned that due to their collaboration with NRC researchers from the former Aluminum Technology Centre in Saguenay, a Quebec based company improved their product and eventually signed a contract with Tesla. However, the majority of internal interviewees stated that there are barriers to these collaborations, including the same barriers identified in the 2012 evaluation (e.g., research too academic, costs too high for SMEs).

It should be noted that, as part of NRC's current Dialogue^{Footnote 10} process, consideration is being given to actions that should reduce these barriers including:

• Establishing a voucher program to allow IRAP to support SME access to NRC expertise at a reduced cost
• Encouraging staff exchanges and assignments between IRAP and the research divisions
• Provide training to IRAP staff on the R&D capabilities within the NRC
• Establish a common framework that sets performance objectives to facilitate work between IRAP, R&D and BMS groups

3.3 IRAP depended heavily on its existing network for clients

IRAP worked with SMEs and organizations across all regions and in a variety of sectors. Over the evaluation period, IRAP provided funding to 7,631 unique firms and 322 organizations across the country.

While firms in many sectors have been supported, firms in the Information and Communication Technology (ICT) sector make up over a third of IRAP's clients over the last five years.

Figure 1: The ICT sector made up over a third of IRAP client firms

The proportion of ICT sector clients is particularly large in the Pacific and Ontario regions, making up 45% and 42% of clients respectively. Quebec has the largest concentration of Manufacturing and Materials clients (23% of their total clients) and Atlantic has the largest concentration of Agriculture and Food clients (18% of their total clients). There are fewer regional variances found with the other sectors.

The organizations that IRAP funds are also clustered in a few sector areas. While there is an opportunity for IRAP to create more precise breakdowns within this category, IRAP codes the largest proportion of its funded organizations as "Other" (34%). This includes chambers of commerce and regional centres and councils. The next largest type of organization funded are industry associations (29%) and incorporated organizations (15%). There are also a number of funded universities and government organizations^{Footnote 11}.

Figure 2: Industry associations and "others" were the organization type most often funded by IRAP

The Pacific and Atlantic regions fund 50% of the total funded industrial and professional associations.

While some sectors such as ICT tend to have more innovative firms than others, it is challenging to confirm the proportion of innovative SMEs in Canada. Whether or not an SME is innovative or plans to be innovative is not information currently collected by Statistics Canada. Nevertheless, with the 7,631 funded firms during the evaluation period, and even in addition to the firms served by funded organizations, it can be assumed that IRAP has reached a very small proportion considering the 1.17 million SMEs in Canada.

Many IRAP clients came from previous relationships or referrals. Because of its established reputation and extensive networks, IRAP conducts very little outreach activities. SMEs are often referred to the program by existing clients, industry associations and other organizations and programs including Concierge. For example, according to program data, in its first two years of operation, Concierge referred approximately 2,000 SMEs to IRAP. Many firms are also returning clients to IRAP, having received assistance in the past. This approach is confirmed in program documentation, and was supported through interviews with IRAP staff and clients. While most internal interviewees agreed with this approach, some did highlight areas where outreach could increase awareness of IRAP among potential clients. This included innovative firms in rural areas and those in the agriculture and food and natural resources sectors. Client data from the evaluation period confirm that firms in these sectors received less funding than others (see Figure 1). A few internal interviewees also mentioned that some firms do not realize they could obtain IRAP assistance. Considering the large population of (potentially) innovative firms, IRAP could widen their reach to those firms that are faced with more barriers (such as location and understanding of funding) in accessing the program. This would increase accessibility of the program and increase opportunities for all Canadian innovative SMEs.

Fully understanding the current reach of IRAP, particularly in terms of advisory services to unfunded firms, remained a challenge. Even if IRAP did widen its reach, the information required to understand reach is not in place.

The previous evaluation of IRAP devoted a considerable amount of effort to assessing the benefits of IRAP's advisory services. However, that evaluation was unable to reach any conclusions due to the lack of data on advisory services^{Footnote 12}. Since that evaluation, some progress has been made. According to IRAP guidelines, "ITAs are expected to enter advisory services which they deem to have been most impactful." It would be unreasonable to require ITAs to enter data on every phone call in which they addressed a simple request. However, the challenge is that "most impactful" is not well defined and ITAs likely have an inconsistent understanding of what needs to be tracked.

Internal interviews suggest that unfunded firms are not tracked consistently. For example, it was found that some ITAs only keep track of firms they feel have potential for eventual funding. It can then be safely assumed that a large portion of the work conducted by ITAs (and their reach) is not recorded.

Lastly, the reach that IRAP has through the organizations that they fund is also not being adequately recorded. Evidence from the internal interviews suggest that information on the firms being served by funded organizations has not been collected consistently. This, along with the other challenges mentioned above, result in a gap in understanding the reach of IRAP.

Improving the way in which all firms that access IRAP is tracked would improve understanding of the types of clients, the full scope of IRAP's reach and provide a more comprehensive understanding of the work being conducted by ITAs.

4 The impact of IRAP

4.1 IRAP played an important role in strengthening the innovation capacity of many SMEs in Canada

Advisory services strengthened the innovation capacity of firms. Advisory services have many benefits to client firms, such as increasing business knowledge and facilitating contacts, and they are necessary for successful project planning. In the focus groups, the ITAs were unanimous in considering IRAP's advisory function as critical to the success of many of their clients. Virtually all the ITAs reported that the initial advice provided to firms was important to developing strong candidates for moving forward with an innovation project.

The main ways in which ITAs are able to assist their clients, especially in the initial stages, were identified as follows:

• Identifying the "real" needs of the firms. The ITAs conduct needs assessments using unbiased "fresh eyes" to help identify the needs of the firms, which were noted as often differing from the firm's initial thinking
• Redefining and redirecting the firms' technical and commercial focus

Responses to the online consultation with advisory-only firms illustrate the impacts of advisory services in strengthening the firms' innovation capacity.

The two most commonly experienced immediate impacts related to a company's ability to better define or structure their project and strengthening their R&D team.

Figure 3: ITA advice was important to better define or structure projects

The respondents were asked a similar question regarding the usefulness of IRAP's advisory services in supporting certain longer-term impacts. More than half said the services were either moderately useful or very useful in each of eight specific areas, including increased internal R&D capability and increased ability to access external expertise, both of which contribute to strengthened innovation capacity.

Finally, respondents were provided the opportunity to elaborate on what they considered to be the greatest benefit of the advice provided by IRAP. Most frequently mentioned benefits were advice regarding project design (e.g., helping the client define and prioritize their potential improvement projects), information provision, and facilitation of contacts.

Firms that had received funding from IRAP also emphasized the importance of advisory assistance:

• In the interviews of funded firms, the large majority of interviewees highlighted the benefits of receiving advice from their ITA in defining their project structure. This included advice on proposal preparation and detailed project planning, including the clarification of their ideas. These firms also mentioned the benefits of the continued relationship with the ITA following funding.
• In the site visits with funded firms, most of the comments that related to IRAP benefits were linked back to advisory services. While funding was mentioned as facilitating R&D, the advisory assistance provided by the ITA was the most common benefit mentioned. Firms explained that their ITA provided valuable business advice, including assessing what projects would be most beneficial to the firm and facilitating access to expertise.
• In the post-project assessments completed by funded firms the benefit of advice regarding how to define or structure their project was identified by 90% of the respondents.

Funding strengthened the innovation capacity of firms. IRAP funding has a major impact on the ability of SMEs to undertake innovation projects. As discussed below, IRAP funding supports innovation projects that either would not have gone ahead or would have been significantly affected in the absence of IRAP funding (e.g., later, smaller, lower risk, or requiring more external funders).

In the post-project assessments completed by funded firms, respondents were asked to identify which of their capabilities had been increased by the IRAP project. "Technical capabilities" and "ability to conduct R&D," both of which are highly correlated with the capability to undertake innovation projects, were identified by about 90% of respondents.

Figure 4: Funded projects improved several facets of the firm

In the ITA focus groups the participants highlighted two impacts of IRAP funding on increased innovation capability; increases in key personnel and increased access to capital. In addition to funding direct project expenses, IRAP funding is used to hire key technical and management staff; and the fact that IRAP has "signed off" on the value of a firm's funded project increases the firm's ability to leverage other investment capital.

In the funded firm interviews, almost all identified IRAP funding as being critical to their success in undertaking their innovation project. Capacity and capability improvements as a result of IRAP funding were noted as increased staffing levels, whereby many of the personnel brought on board and trained through an IRAP project stayed with the firm through scale-up and commercialization. The Youth program was singled out by many interviewees, as 11 of the 25 firms interviewed reported that youth interns are still working with their companies. Increased capability to conduct R&D was also highlighted in the site visits with funded firms as a benefit of IRAP funding.

IRAP had an incremental impact on firms. About 90% of the post-project assessments indicated that IRAP funding helped widen the scope of the firm's project, advanced the timing of the project, and/or had a significant effect on whether the project would have been undertaken at all.

In the interviews of funded firms, some respondents noted that their funded projects would have been smaller, possibly delayed, and less risky in scope (with corresponding lower rewards).

This was confirmed in the case studies of funded organizations carried out in this study. Almost all of the 30 client firms interviewed were start-ups or small SMEs that needed assistance, but were not ready to receive IRAP funding. For example:

• A family owned an apple orchard wanting to find a way to use the apples on the ground
• An IT start-up in northwestern Ontario that needed technical assistance in testing scripts
• A craft food and beverage start-up needing assistance with prototype development
• An entrepreneur who needed help with finalizing and producing a 3D model of his design before he could file for a patent

SME clients of organizations found the services to be beneficial, and many of them were supported by organizations in undertaking innovation projects. A survey of clients of funded organizations conducted by IRAP in 2016 found that most of the services provided were business-related. Eighty-eight percent of the 462 respondents were satisfied with the services provided by the organization. As shown in Figure 5 below, these services were primarily focused on increasing the innovation capability of the client firms.

Figure 5: Services mostly improved the business and technical knowledge of firms

The case studies of funded organizations conducted in this evaluation included a review of the services provided by the organizations. This illustrated that these organizations are providing beneficial services, for example:

• Short-term R&D projects
• Access to the business and technical services of a university
• An assistance program to address business or operational issues
• Prototype development and product testing

The interviews of 30 client SMEs of these organizations confirmed that these services were beneficial. The vast majority (28 of the 30 clients interviewed) were satisfied with the services provided. It was also determined that half (16 of the 30 SMEs) were supported by the organization in undertaking innovation projects, which is a fairly high number considering that several of the organizations do not directly support innovation projects.

4.2 IRAP facilitated the increased growth of many SMEs in Canada

IRAP facilitated SME growth through a variety of internal mechanisms tailored to the individual SME's needs. As discussed previously, the structure of IRAP is key in supporting SME success. In particular, the relationships developed between ITAs and their client firms facilitate an understanding of needs and objectives that allows ITAs to tailor the assistance provided in order to best facilitate success and growth.

When speaking with funded firms during the interviews and site visits, many found it difficult to quantify the success or growth that they could attribute to IRAP. However, these clients estimated that IRAP had a significant role in their success. As noted previously, clients stated that IRAP assisted them with developing technologies faster and commercializing them earlier. IRAP-funded projects enabled clients to "get to the leading edge" and stay ahead of their competition.

Some interviewed firms did not have commercial products, but rather their IRAP projects were focused on improving internal processes to either increase yield or build efficiency.

Many SME clients of organizations increased their productivity, growth, or profitability as a result of the assistance they received from these organizations. SMEs that receive services from funded organizations also experience enhanced success as a result of IRAP. In the case studies of CTOs, 18 of the 30 firms (60%) that were interviewed experienced, or are likely to experience, increases in productivity, growth, or profitability. Impacts of the organization's assistance that were documented in the case studies included successful prototype and product development, production improvements (e.g., scale-ups, increased profit margins, increased efficiencies), increased product differentiation and improved distribution, and increased sales growth (in one case by $23M/year).

4.3 IRAP-funded firms contributed positively to the Canadian economy

IRAP resulted in positive economic benefits. This is evident from the PBCA analysis that was conducted for the evaluation.

It should be noted that a different approach to conducting the PBCA was used in the current evaluation than was used in the previous evaluation. In the current evaluation, NRC invested in a rigorous approach for estimating the economic benefits due to IRAP that is consistent with Treasury Board Secretariat's guidelines for conducting benefit-cost analyses, as well as with standard economics methodology^{Footnote 13}. The results of the PBCA conducted in this evaluation cannot be compared with the previous evaluation, because the methodologies used are very different.

The most significant difference is how the net economic benefits to client firms are valued. For example, if an IRAP client firm developed a new product (to which IRAP has made very important contributions), and it is further assumed that, in a given year:

1. The gross revenues the firm obtained from the sale of this product were $5M
2. In order to produce and sell the product, the firm had to hire 20 people at salaries of $100,000/year (total salaries = $2M)
3. Other costs incurred in producing and selling the product (production equipment, materials, marketing costs, etc.) were $1M

The net benefits to the firm in this year would be estimated as follows:

Current evaluation	Previous evaluation
Step 1: Estimate the gross revenues from the sale of the product = $5M	Step 1.:Estimate the gross revenues from the sale of the product = $5M
Step 2: Estimate the firm's costs associated with producing and selling the product = $2M (salaries) + $1M (other costs) = $3M total implementation costs
Step 3: Estimate the profit due to producing and selling this product = gross revenues from the sale of the product ($5M) – implementation costs ($3M) = $2M. This $2M is taken as the net economic benefit to the firm. The % profit on the product, 40% in this case (2/5), is often called the "gross margin" on the product. In this study, rather than obtaining estimates of all the product-related cost items, as illustrated above, information on gross margins for the company or typical gross margins for companies in the same field were used to estimate the net economic benefits to the client firms. Therefore in the example case, if gross margins for company X are 40%, the profit due to producing and selling the product would have been calculated as $5M x 0.4 = $2M.	Step 2: Estimate the profit due to producing and selling this product. This was done by applying the operating profit margins for the company as a whole ([total company revenue minus total expenses]/total company revenue) to the product-related gross revenue. In this study, company operating profit margins were based on industry average profit margins from Statistics Canada data. Company-wide operating profit margins are often lower than the gross margins on successful products, since they include a wide range of company-wide expenses such as insurance, depreciation on buildings and equipment, accounting, human resource operations, and other "overhead" items. Assume in this case that the operating profit margin for the firm is 20%. Then the profit due to producing and selling the product would have been calculated as $5M x 0.2 = $1M. Step 3: Add the profit due to producing and selling the product ($1M) to the salaries of the people who were hired to produce and sell the product ($2M): i.e., $1M + $2M = $3M. This $3M is taken as the net economic benefit to the firm.

As demonstrated, the main difference in the two methodologies for estimating the net benefits to client firms is the treatment of salaries. In the previous evaluation the salaries of the incremental people that were hired was treated as an economic benefit (and, in fact, these salaries plus related overheads comprised 72% of the overall IRAP program benefits). However, note also that, in the previous methodology, some salaries and overheads are implicitly subtracted from the overall IRAP benefits, because step 2 of this approach accounts for company-wide salaries and overheads as costs. In the current evaluation, the PBCA methodology treats the IRAP-related salaries the way a company would, as a cost, and is therefore is deducted from the bottom line.

The approach to collecting the information to conduct the analysis was also different. The previous evaluation relied on an online survey of funded firms, while the current approach, IRAP was consulted to identify a list of over 150 potential "big winners", that is, firms that have demonstrated considerable success. These candidates were explored and eliminated until a limited list of approximately twenty of the most successful firms remained. (See appendix A for more details on the PBCA methodology.) The objective of this PBCA is to assess whether the benefits of a small set of high impact cases can cover the total costs of the program (or ideally more than cover these costs).

Table 3 below provides a brief description of four IRAP-client high impact cases to provide an understanding of what these cases have typically involved.

Table 3: PBCA case study examples

Name of Firm	Description of firm activities and IRAP contribution and impacts
Calgary Scientific, Calgary, AB Year founded: 2005	The original business of Calgary Scientific was developing products to support the clinical use of medical images, such as radiology workstations. Following consultations with their ITA and feedback from industry contacts, and after two years of R&D, they changed their focus to developing technologies for transferring medical images via the internet, and they developed a web-based architecture for medical image transfer and web collaboration. IRAP has provided continuous advisory assistance along with a series of eight funded projects since 2006. The company has worked with the same ITA from day one, and according to a company representative, "IRAP has enabled us to push ahead at each stage." The company is now the world technology leader in providing web and mobile access to medical imaging, and their basic software is installed in more than 30 countries.
Sandvine, Waterloo, ON Year founded: 2001	Sandvine provides network policy control equipment to network service providers. (Network service providers essentially provide the infrastructure that makes up the internet.) The company's products are designed to implement a broad range of network policies, including service creation, network traffic, response times, billing, congestion management, and security. They have had a continuous relationship with their ITA, who is an adjunct professor at Waterloo, since 2007, and they have received IRAP funding for seven projects. Sandvine has grown to a global company with more than 300 employees, and over 700 employees in affiliated firms. Sandvine has attributed significant revenue growth to IRAP. The company has repeatedly been identified as the global leader in network policy control.
Smarter Alloys, Waterloo, ON Year founded: 2009	Smarter Alloys manufactures shape memory alloys, which are metal alloys that have the capability of taking on up to 50 memories that can change the shape of the metal based on external inputs, such as temperature. The technology to program metal with multiple memories was developed by the company founder, based originally on his Ph.D. thesis research. The first commercial application is to orthodontic dental wire, which adjusts the forces applied to the teeth as the teeth move and, as a result, reduces the time required for teeth straightening from as long as two years to as little as six months. Several more products are in development for the sporting goods, automotive, and medical sectors. The ITA recruited the firm to IRAP after hearing the founder speak at a seminar just after the company was formed. He has provided continuous advisory support, networking, and, working with another IRAP advisor, he helped the firm secure a $1.1M grant from a private funding program, as well as four IRAP-funded projects.
Medicago, Québec, QC Year founded: 2006	Medicago develops vaccines and therapeutic antibodies from plants to respond to emerging diseases. IRAP support has been continuous since the beginning of the company. In the initial years, IRAP supported Medicago's laboratory operations and internal development through advice and networking, and a close relationship was developed between the ITA and the company. A company representative is quoted in a 2009 article as follows: "Our ITA had complete training in agronomy and understood very well the technology of processing plant material. It's a great advantage…" They have worked with four ITAs since the company's founding, combined with a sequence of IRAP funding contributions for technology development. The company president has recently publicly acknowledged the benefits of IRAP's "continued guidance and contributions." Over the years, Medicago has obtained a number of large investments from outside sources, which have enabled them to develop research projects and production plants in Canada and the US. Today the company is the global leader in the plant-made vaccine field.

In interpreting the results of the PBCA, it should be noted that lower and upper bounds were used in the analysis to reflect more conservative vs. more optimistic assumptions regarding future benefit streams. Also, the net benefit and program cost streams were both discounted using three different values of the discount rate, representing alternate uses of the funds, 2%, 5%, and 7%.

The PBCA results demonstrate that IRAP has been beneficial. The net present values are positive under both upper bound assumptions and, even more noteworthy, lower bound assumptions (which are far more conservative), and for all discount rates modeled, including a 7% discount rate (which is the highest value usually employed in PBCA studies of innovation support programs).

The PBCA found that the combination of high impact (HI) and fairly high impact (FHI) firms that were studied in detail will return a profit on IRAP-supported products and services of between $4.4B and $18.5B over their assumed lifetimes, over and above the total costs of the IRAP program from 2005-2006 through 2015-2016.

The ratio of net benefits to total IRAP program costs from 2005-2006 through 2015-2016 ranges between 2.5/1 (lowest lower bound) to 9.3/1 (highest upper bound). In other words, even under the most conservative assumptions and the highest discount rate, the net benefits of the IRAP program to the Canadian economy have been two and a half times greater than the program's cost, and under "reasonable" assumptions the net benefits have been much higher than that.

There is no way to determine the "best" value for the benefit cost ratio.However, the average value between upper and lower bounds, at a 5% discount rate, is about 4.9/1. Similarly, the average value of the economic benefits of the program are at least $10 billion higher than the program's costs.

The significant variation from lower to upper bound figures reflects: (1) that the set of lower bound and upper bound assumptions are significantly different for some cases, and (2) the use of three discount rates.This is typical of PBCA studies and does not reflect anything unusual about IRAP.

Table 4: Summary of PBCA Findings

Benefits	Lower Bound ($M, 2016$)			NA	Upper Bound ($M, 2016$)
Discount rate	2%	5%	7%	NA	2%	5%	7%
Net benefits in “HI” cases	7,352.9	6,564.6	6,158.8		16,226.1	13,838.6	12,570.9
Net benefits in “FHI” cases	1,507.8	1,320.5	1,216.3		4,545.1	3,862.5	3,486.3
Total Net Benefits	8,860.7	7,885.1	7,375.2		20,771.3	17,701.1	16,057.2
Program costs (2005-2016)	2,242.2	2,639.8	2,949.9		2,242.2	2,639.8	2,949.9
Net Present Value	6,618.4	5,245.3	4,425.3	NA	18,529.0	15,061.3	13,017.4
Benefit Cost Ratio	3.95	2.99	2.50	NA	9.26	6.71	5.44
Average NPV and B/C Ratio @ 5% discount	NA	NA	NA	10,153 4.85	NA	NA	NA

The results of the PBCA suggest that IRAP has very positive impacts on the economy. If fact, in comparison with nine other similar government programs (industrial innovation support programs that were assessed using the same PBCA methodology), IRAP has by far the highest NPV and the highest B/C ratio. This positive result undoubtedly reflects two factors. First, the ITAs have a strong vetting process to primarily support firms and innovations which have a significant chance of commercial success. Second, there are significant advantages to firms due to the close relationship between the ITAs and their clients, which helps tailor the innovations, the funded R&D projects, and the firms' internal R&D plans, HR hiring, and business strategies to best fit market opportunities. At the same time, it should be noted that IRAP is more towards the commercialization end of the innovation chain than several of the other programs included in the comparison.

5 Delivery of IRAP

5.1 Delivery of IRAP could be improved through increased efficiencies

IRAP had structures and processes in place to manage the IRAP program. Further improvements to these could increase efficiencies. Ample guidelines and associated documentation exists in order to enable IRAP to manage the program. These include:

• Field manuals that detail processes and national policies for a consistent approach to managing IRAP activities and contributions
• The web-based client management (SONAR) system that facilitates the management of clients and their financial contributions
• The portfolio management model, a methodology for ITAs to manage their funnel of potential clients, as well as future investments in firms
• Client feedback through post project reports for information on the benefits, quality and usefulness of IRAP assistance
• Standardized risk assessment methodology for the financial monitoring of all projects
• Service standards outlining the level of service that clients can reasonably expect from IRAP

Internal interviewees confirmed that these systems are sufficient to support delivery of the program. However, they also identified some areas for improvement. Potential improvements identified in the study fall into three categories:

Application process: In the interviews of funded firms, it was noted that, because of the limited capacity and experience of many SMEs, they find the application process challenging. This is perceived as cumbersome for small companies because they do not have the staff to devote much time to applications. Some ITAs from the focus groups also highlighted that low dollar projects require more paperwork than is necessary.

Approval process: The majority of the internal interviewees said that there are opportunities to streamline some of the approval processes. Many noted challenges with the number and level of management involved in various approval processes. For example, improvements could be made to the approval process for:

• Changes in SONAR (e.g., removing need for Director's approval for an address change)
• Large projects (e.g., the levels of management approval could be reduced)

Further, it was stated in both the interviews of funded firms and the ITA focus groups that there are often multiple requests for revisions during the approval process for applications from small firms that could be reduced by further streamlining the process.

Client management systems (e.g., SONAR and the Client Portal): Many internal interviewees noted improvements, such as increased reliability of SONAR over the past few years. A few noted that further efficiencies could be gained if more capacities were built into the system. For example, one mentioned benefits to the ability of managing clients through a "LinkedIn" approach, where a client profile is updated as needed. Another example provided was having information in SONAR on typical performance for their client firm types in order to assess if they are performing well. Many interviewees also mentioned opportunities for improvements of the online portal, mentioning challenges with IT issues for online claims, template issues for online proposals, and inflexibility for some claims adjustments.

Improving the administration of IRAP has also been a topic discussed as part of NRC's Dialogue exercise. Many of the aspects discussed align with findings from the evaluation, and actions are being taken such as: streamlining IRAP's due diligence requirements and authority approval levels to increase efficiencies and simplifying SONAR. IRAP's 2017-18 plan also states a priority of identifying business processes that can be altered to improve the client experience. These IRAP priorities align with recommendations of this evaluation regarding program delivery.

Temporary funding programs can be a challenge to deliver. Temporary funding programs are mostly seen as positive by IRAP staff, but there were some concerns expressed in the focus groups of ITAs. On the positive side, some of these programs were seen to offer useful services to their clients that were more difficult to offer through IRAP. On the other hand, ITAs in the focus groups noted that these programs often sunset too soon. It typically took one or two years for the ITAs and the clients to really understand the details of a new temporary program. Just when this had been accomplished, the program would sunset. They recommended that temporary programs last longer (e.g., at least five years, before sun setting).

Lack of appropriate human resources can also be a challenge in delivering temporary programs. The internal interviewees stated that the delivery of temporary programs contributed to additional burden on staff and inefficiencies in IRAP program delivery. The majority of internal interviewees noted that there are inefficiencies associated with delivering short-lived programs (under five years), particularly because these programs often did not come with additional operating funds. For those programs that did have operational funds, the hiring of staff lagged behind program growth.

5.2 Resource constraints continued to pose barriers to efficient and effective delivery

Human resources did not grow at the same rate as the various programs and services being delivered by IRAP, which increased the burden on its staff. During the evaluation period, IRAP's core contribution budget has increased over $100 million. This does not include contributions through temporary programs such as DTAPP, BIAP and CAIP. Human resources of the program have not increased at the same rate. While there are plans to hire ITAs as well as other support staff, and a number of hiring actions are implemented, only about half of open positions have been filled.

Table 5: Total hiring actions initiated and filled by fiscal year

Description	2012-13	2013-14	2014-15	2015-16	Total
Number of ITA hiring actions	50	80	64	63	257
Number of successful ITA hires	13	34	34	35	116
Percentage of hires	26%	43%	53%	56%	45%
Number of hiring actions for all non-ITA positions	29	33	26	24	112
Number of successful hires for all non-ITA	10	25	29	13	77
Percentage of hires	34%	76%	112%	54%	69%
Total Hiring Actions	79	113	90	87	369
Total Successful Hiring Actions	23	59	63	48	193
Percentage of hires	29%	52%	70%	55%	52%

Source: SONAR data on unique firms with funded project between 2012 and summer of 2016

According to internal interviewees, factors contributing to this result included the limited resources allocated to meet the hiring needs of IRAP as well as difficulties in finding appropriate candidates. Criteria for ITA selection were noted by internal staff as particularly problematic. IRAPs 2017-18 operational plan also highlight this challenge, noting that the delivery model of hiring industry veterans with significant experience has resulted in a hiring cycle that takes almost half a year to complete. The plan also notes the risk of the program's rapid growth interfering with its ability to deliver the program. While there is no formal indication that criteria for ITA selection will change, the current operational plan notes that IRAP will be reviewing their delivery model to ensure consistent and appropriate delivery of all program elements.

There was a lack of appropriate ICT tools and support, which decreased efficiency and effectiveness. The majority of the internal interviewees indicated that ICT tools did not facilitate efficient and effective program delivery. For instance, ITAs conduct much of their work away from the office and do not have the appropriate tools to efficiently work remotely. Examples provided by ITAs during the focus groups were poor mobile devicesand poor accessibility to systems such as SONAR. Results of the NRC Dialogue exercise suggest that mobile access is an aspect that IRAP is hoping to improve.

The lack of IT support is also a significant issue for IRAP. Most staff interviewed noted the inadequacy of IT support. This includes extremely long wait times for crucial equipment such as telephones, mobile phones and computers to be purchased and provided. A few internal interviewees noted not having the necessary tools in time for the arrival of a new employee. Further, many IRAP offices do not have onsite IT support and must rely on a central service. This causes long wait times, but also shows a lack of understanding from IT support of the context and needs of the regional offices. One respondent noted that some IT technicians do not understand how IRAP is structured and are not aware of where IRAP staff are located, or who they are. Poorly planned IT support does not take into account the needs and context of the program and impacts on effective delivery of the program.

6 Implementation and early impacts of the concierge program

6.1 Since it was first implemented, Concierge evolved in order to best meet SME needs

Concierge supported SMEs' ability to find innovation programs that are most appropriate to their needs. The Concierge program was designed following recommendations of the Jenkins report.

The report outlined the need for a service to help SMEs navigate the innovation funding environment because it was found that:

• Half of firms were not aware of R&D support programs and were bewildered by the array of available resources
• Canada's landscape of programs that support business innovation is densely populated by initiatives spanning many departments and agencies, both at the federal and provincial level
• SMEs seeking to become more productive, more innovative, and more export-oriented were often discouraged as they struggled to identify, understand, and access government programs

Interacting with Concierge was intended to provide SMEs with a single point of access to assist them in identifying government programs and services that best meet their needs. The program has achieved this, and internal interviewees also noted that Concierge not only brings greater awareness of the existing programs, but also a body of knowledge of how these programs work, how to access them, and whether or not they fit the needs of a given firm's planned project.

In the absence of Concierge, the majority of the internal interviewees noted that SMEs search online or ask regional partners about innovation support programs available to them. This, however, is time consuming for SMEs.

There were many challenges in implementing and delivering Concierge, largely due to a mismatch between resources allocated and expectations for the program. Concierge is intended to be a national program, however its initial design and the resources allocated to the program led to many delivery challenges. The value of the Concierge program is the expertise of the Innovation Advisors (IA) who consult with firms and provide them with targeted referrals and linkages to innovation assistance. They have extensive knowledge of the support available to SMEs and take time to understand the needs of the firm in order to provide them with the best information. Since its launch in 2013, there have not been more than 15 IAs working to address the needs of innovative Canadian SMEs.

The call centre, initially used to field calls for the Concierge program was run by another government agency. This was noted by some internal interviews as problematic and not the most efficient approach. Recognizing potential to increase efficiencies, calls to the Concierge program are now fielded through the NRC Client Service Centre.

While Concierge was implemented with objectives, outcomes and targets in mind, many interviewed Concierge staff noted feeling a lack of initial direction and support for this program. This may be in part due to the turnover in leadership (the program had four Directors in three years). It may have also been that it has taken time for senior management to understand the most effective use and approach for the program. In order to solidify direction for Concierge, senior management held a two day all staff strategic planning session in January 2016. During this session, a business model outlining a new approach for the service was developed. This included the inclusion of an additional human resource layer, the Regional Information Officer, to serve clients needing less engagement, and allowing IAs to focus on those needing more engagement.

Concierge's delivery model evolved to best serve later stage, high potential clients. Despite challenges, many interviewees, including external experts, recognize the value of the expertise in Concierge. It has been evolving to maximise the impact of the program using available resources. The program remains national in scope and available to all SMEs, however as noted previously, IA time is increasingly limited to serving firms requiring longer term engagement, which are often high potential growth firms. Other firms are provided information through the website, from information officers in the call centre and through meetings with Regional Information Officers (RIO).

With the launch of the Accelerated Growth Program in 2016, IAs are also put to good use as they have the expertise and national coverage to play a key role in this program, assisting high potential growth firms in accessing innovation assistance. The 2017-18 IRAP Operational Plan notes the intention to focus IA resources on the AGS.

6.2 Despite limited resources, Concierge had an impact on increasing SMEs' awareness and access to innovation support

The reach of Concierge was limited by insufficient human resources available to deliver the program. Between 2013-14 and 2015-16, IA's met with a total of 8,827 firms, an average of 2,942 per year. This represents a small proportion of the innovative SME population in Canada.

As noted previously, there were limited resources allocated to deliver Concierge in a manner that would allow significant reach. Apart from the human resources delivering the service, outreach and communications were also limited. An advertising campaign run from February 23 2015 to March 31 2015 did result in a significant increase in website views. Even if further broad-reaching communication activities were conducted, the program did not have the bandwidth to support a significant influx in clients. That said, IAs have worked hard to promote their services. Interviewed IAs noted conducting significant outreach at trade shows and other events. Data from April 1 2014 to March 31 2016 show that 61% of the clients that met with IAs were a result of their outreach activities.

Concierge clients were satisfied with the services and found them useful. Concierge has had a positive impact on firms that have accessed the program. For example, the survey of Concierge clients found that, as a result of their meeting with an IA, 60% of the clients reported that they pursued the referred program or service for more information.

Figure 6: Most clients reported contacting Concierge referred programs

When asked how likely it is that they would recommend Concierge to their friends, family, or business associates, 71% reported that it was very likely or likely.

The interviews of Concierge clients conducted in the evaluation were consistent with these results. They found that:

• Concierge clients are satisfied with the services and find them useful
• Opportunities to expand their networks and obtaining guidance on "best fit programming" are regarded as key benefits
• Direct in-person attention is valued (almost all the interviewees said there had been an in-person meeting to discuss their business needs)

The focus groups of ITAs highlighted perceptions that the program had improved recently. ITAs reported a much higher proportion of useful client referrals from Concierge than previously. When receiving referrals from Concierge in the past, many ITAs said they were generally poor and that the majority of these firms were not suitable for IRAP support. This may be the result of a new program gaining experience and traction.

7 Conclusion

IRAP's mission is to "accelerate the growth of SMEs by providing them with a comprehensive suite of innovation services and funding." The evaluation found that IRAP addresses the needs of Canadian SMEs for assistance in developing innovative products and processes in a unique and very effective way, through the provision of both advisory and funding assistance, and often the two in combination. IRAP's services, including the services delivered by IRAP-funded organizations, are provided to all sizes and types of SMEs across Canada and across industry sectors. The IRAP program model is the main factor underlying its success, in particular the development of relationships between the ITAs and their client SMEs, which enable the ITAs to tailor IRAP assistance to the needs of the client.

IRAP has played an important role in strengthening the innovation capability of many SMEs in Canada. Both advisory assistance and funding were found to contribute significantly to this impact. IRAP has also had a major impact on increasing the growth and profitability of Canadian SMEs. As a result of this, IRAP has had very positive impacts on the Canadian economy. The evaluation found that IRAP's economic benefits are at least $10 billion, over and above the costs of the program, and the ratio of IRAP's economic benefits to the cost of the program is at least 4.9 to 1. In comparison with nine other somewhat similar government industrial innovation support programs that were assessed using the same methodology, IRAP has by far the highest net economic benefits and benefit-cost ratio.

IRAP has structures and processes in place to efficiently and effectively manage the program. However, it was found that further improvements to these could increase efficiencies and that resource constraints and sufficient IT support have posed barriers to program effectiveness.

Concierge was designed and implemented during this evaluation period. It faced initial challenges in aligning its considerable national mandate with allocated resources and worked to find the best approach to meet needs within these parameters. With the Accelerated Growth Service and the focus on high potential growth firms, Concierge may now have found this balance.

8 Management response

Recommendation	Response and Planned Action(s)	Proposed Person(s) Responsible	Timeline	Measure of Achievement
1. Develop a better understanding of the population of potential IRAP clients (SMEs that have the potential and willingness to grow through innovation).	Recommendation accepted. IRAP has an extended network of Industrial Technology Advisors (ITAs) located across the country. These ITAs build strong engagement with SMEs, and interact with the broader business community in general, with a view to understand and address pressing innovation challenges. Through its service offerings of advisory services and funding, IRAP reaches over 10,000 SMEs annually. IRAP understands that its reach doesn’t encompass all potentially eligible clients. The program will prepare an analysis on the characteristics and profiles of SMEs that can qualify for IRAP assistance, in order for it to fine-tune its outreach approach.	IRAP VP	December 2017	Completion of an analysis to support future outreach efforts.
2. Develop a targeted outreach strategy to identify new, high potential clients within the population.	Recommendation partially accepted. IRAP’s client base is directly related to the size of its funding envelope. IRAP’s Portfolio Management Approach focusses on SMEs with a high potential for growth. Should the existing intake of new clients not have sufficient high potential growth firms, IRAP will build upon the findings of the report on SME characteristics and profiles, and devise a targeted outreach strategy. On a need basis.	IRAP VP		Implementation of a targeted outreach strategy, if required.
3. Develop more comprehensive guidelines and consistent practices regarding the advisory services that need to be tracked (including unfunded firms) and a process for consistently tracking firms served by funded organizations.	Recommendation accepted. IRAP has in place a client management system (Sonar) with functionality to capture advisory services. In order to strengthen the value added of its information, IRAP will develop and implement expanded guidelines with regard to the provisioning and tracking of advisory services to funded and unfunded firms. IRAP currently administers an annual Post-Service Assessment (PSA). The PSA is administered to SMEs who received services from CTOs in order to gather performance information. IRAP will revisit its approach to PSA to ensure consistent and timely access to performance information.	IRAP VP	March 2018	Revised guidelines implemented. Report on PSA results.
4. Increase support to SMEs for the preparation of proposals	Recommendation accepted. The IRAP Innovation Portal (IIP) is a web-based tool that allows, among other functions, clients to prepare and submit their project proposal on-line. IRAP will seek feedback from staff on specific areas wherein clients are experiencing challenges and, where appropriate, prepare additional training or guidance material to assist them in leveraging IRAP support.	IRAP VP	March 2018	New or revised guidance material and support documents developed and made available to clients.
5. Review the approval process of projects to facilitate greater efficiencies	Recommendation accepted. IRAP program design is predicated on identifying and applying continuous improvements. It typically has a team or teams addressing challenges or opportunities specific to improvement initiatives. Related to project approval, IRAP will explore streamlining the processes with an emphasis on simplifying the procedure while keeping appropriate controls in place. IRAP will leverage findings from an initiative to ‘re-imagine’ its client portal, the IRAP Innovation Portal (IIP). These improvements are expected to lead to reduced administrative burden on SMEs.	IRAP VP	March 2018	Review of approval process conducted. Improvements to processes and tools implemented.
6. Consult with the regions to identify and implement changes to the client management system that best meet the needs of delivery staff.	Recommendation accepted. IRAP will leverage the Program Delivery Advisory Committee (PDAC), whose role is to review and support the implementation of proposed program improvements while balancing stakeholders’ perspectives and organizational change management capacity, under the auspices of IRAP’s Senior Leadership Team.	IRAP VP	September 2017	PDAC established and operating under the direction of the IRAP Senior Leadership Team.
7. Implement the best service delivery model for Concierge based on evolving Government requirements.	Recommendation accepted. Concierge is a complement to IRAP’s suite of services for innovative SMEs. Since its inception, Concierge has progressively updated its service offerings to address gaps experienced by innovative SMEs. This Concierge service delivery model will continue to evolve, further confirming and updating its objectives in order to support the Government’s Innovation Agenda as it moves into full implementation.	IRAP VP	October 2017	IRAP updated strategic objectives for Concierge.
8. Develop and implement an operational plan appropriate for the evolving needs for Concierge.	Recommendation accepted. Building on the Government’s Innovation Agenda and in support of Concierge strategic objectives, a supporting Concierge operational plan will be developed and implemented.	IRAP VP	December 2017	Operational plan developed and in effect.

Appendix A: Methodology

This section presents a detailed description of the evaluation methodology, including the evaluation design, the data collection methods as well as challenges and limitations of the evaluation.

Evaluation Design

The evaluation was carried out in accordance with the NRC's approved evaluation plan and TBS policies. The last evaluation of NRC-IRAP took place in 2011-12. The timing of the evaluation was also determined by the legal requirements of the Financial Administration Act.

The questions explored through the evaluation are listed in Table 1. These questions were developed following a review of key program documents, planning interviews with key stakeholders and in consultation with IRAP senior management.

Table 1: Evaluation Questions

Relevance

1. Is there a need to support SME innovation in Canada through financial or advisory support?
2. Has NCR-IRAP met the needs of Canadian SMEs for innovation support?
3. Is NRC-IRAP consistent with government priorities?
4. Is NRC-IRAP consistent with federal roles and responsibilities?

Performance

5. Has NRC-IRAP supported SME innovation?
6. Has NRC-IRAP supported increased SME performance?
7. Has NRC-IRAP resulted in positive impacts on the Canadian economy?
8. What have been other benefits of NCR-IRAP to SMEs?
9. Have current structures and processes facilitated efficient and effective delivery of NRC-IRAP?
10. Have available resources and support facilitated efficient and effective delivery of NRC-IRAP?

The evaluation of IRAP was conducted using a hybrid approach, with a portion of the work completed by an independent evaluation team from the NRC Office of Audit and Evaluation and a portion by an independent consultant, KPMG LLP. The evaluation was also supported by an Evaluation Advisory Committee to provide advice and input on key deliverables.

In order to maximize the generation of useful, valid and relevant evaluation findings, the evaluation used a mixed methods approach, allowing for triangulation (i.e., convergence of results across different lines of evidence) and complementarity (i.e., using complementary research methods to provide different perspectives when examining various facets of complex issues).

Data Collection Methods

The evaluation employed both qualitative and quantitative research methods including the following:

• Document and literature review
• Analyses of financial, administrative, and performance data
• Interviews with IRAP staff and external subject matter experts
• Focus groups with ITAs
• Interviews with IRAP clients who have received contribution funding
• Site visits with IRAP client firms that have received contribution funding
• Discussion groups with IRAP client firms who have not received contribution funding
• On-line consultation of IRAP client firms who have not received contribution funding
• Case studies of IRAP contributions to organizations
• Interviews with clients of the Concierge service
• Partial benefit-cost analysis

These are described below in more detail.

Document and literature review

The review of documents and literature primarily provided evidence to support questions about the need and relevance of the program. Approximately 40 documents in the following categories were obtained and reviewed:

• Program area documents (e.g., those related to Concierge, the provision by IRAP of funding to organizations, and IRAP's provision of advisory assistance and funding to SMEs)
• Strategic business documents (e.g., strategic and operational plans, management reports)
• Previous evaluation studies (e.g., evaluations of IRAP, BIAP, DTAAP and YES)
• Published and grey literature (e.g., international reports, academic research)
• Government of Canada publications (e.g., the Jenkins report, federal budget documents)

Analyses of financial, administrative and performance data

Analyses of available data were conducted to inform the reach and performance of the program. The following types of data were analysed:

• Administrative data on client: This included data on all firms and organizations with a funded project during the evaluation period (2012-2017)
• Feedback data from clients: This included self-reported information about revenue, income, R&D expenditures and employees. This data is self-reported by clients on an annual basis for five years following the end of a project. This data also includes feedback from clients through the post-project report (formerly post-project assessment), designed to obtain information on IRAP outcomes and program delivery. Firms and interviews funded through the Youth program also provided feedback through post-internship surveys that track outcomes of the program.
• Administrative data on advisory services: This included ITA reported advisory services, including the type of advisory service provided (e.g., business, technical etc.)
• Administrative data on the Concierge program: This included data on numbers of client website visits, client contact centre calls, and client referrals, as well as client satisfaction. The survey of clients carried out by the Concierge program in 2016 was also analyzed.
• Other data sources: Data from a variety of other sources was also analyzed (including IRAP HR, IRAP Finance, post-service assessments of CTO clients)

Interviews with IRAP staff and external subject matter experts

Interviews with IRAP staff and external subject matter experts were designed to collect information on all aspects of the evaluation. Due to the decentralized delivery of IRAP, it was important to include internal interview participants from each of the five regions, Atlantic, Quebec, Ontario, Prairies and Pacific. The interview approach also considered the various levels, from senior management to operations and identified the appropriate informant in each case. Most internal interviews were conducted in-person, during visits to each region. Others were completed over the telephone.

External experts were identified through an online search as well as by asking IRAP for referrals to their partner organizations. All external expert interviews were conducted over the telephone and participants were provided with the interview questions in advance. The total number of interviews completed is outlined below:

Interview group	Interviews completed
Senior Leadership Team	7
Regional Directors	6
Operation and Finance Managers	7
HR/Finance	2
Concierge staff	7
External subject matter experts	6
Total	35

Focus groups with ITAs

Focus groups with ITAs from each of the five IRAP regions were held. Participants were selected from volunteers and those nominated by the Regional Executive Directors. From this initial sample, participants were determined on the basis on availability during scheduled focus group times. The numbers of ITAs who participated in each focus group were as follows.

Region	Focus group date	Number of participants
Prairies	Dec. 16, 2016	7
Atlantic	Dec. 21, 2016	6
Pacific	Jan. 6, 2017	10
Ontario	Jan 17, 2017	8
Quebec	Jan. 19, 2017	5
Total		36

A list of discussion topics, as well as a note from the relevant IRAP Regional Executive Directors explaining the purpose of the exercise, was provided in advance. The topics were mainly related to the needs of SMEs for innovation support, the impacts of IRAP advisory and funding assistance for SMEs, and ways in which the IRAP program delivery and administration could be improved. Each focus group lasted approximately two hours.

Interviews with IRAP clients who received contribution funding

Interviews with funded firms were designed to collect information from companies who received IRAP (core) and Youth funding. Respondents were asked about the need for IRAP funding, the impacts they experienced as a result of IRAP advisory services and funding, impacts on their commercialization and capacity building efforts, the incrementality of IRAP funding (the extent to which their project activities were due to IRAP funding), and their general satisfaction with IRAP services.

The sample selection criteria included:

• Firms had to have received both IRAP and Youth funding
• They had to have projects that were completed earlier in the evaluation period
• They had to have an IRAP funded project larger than $100K
• The sample had to be representative of the IRAP regions

The sample firm lists were provided to the respective ITAs who then contacted the firm to provide an introduction to the evaluation and advance notice of the request for an interview. Follow-up phone calls and email messages were used to schedule the interviews. Interviews were conducted by phone. The numbers of interviews that were planned and completed were as follows:

Region	Number of target Interviews	Interviews completed
Atlantic	4	4
Ontario	10	10
Pacific	5	4
Prairies	5	3
Quebec	6	4
Total	30	25

Difficulties in securing participation from sampled firms resulted in fewer interviews being completed than anticipated. However, there remains sufficient representation of firms from all regions.

Site visits with IRAP client firms that have received contribution funding.

Six site visits were conducted with IRAP funded firms in Calgary, Ottawa, Vancouver, Boucherville, Halifax, and Toronto. For practicality, firms near regional offices were selected, but also firms in which IRAP funding had facilitated success. IRAP regional offices suggested firms, with the final selection done by the evaluation team. The site visits included an interview with the firm and a visit of their facilities. Interviews with the firm included discussions about their journey with IRAP, how IRAP assisted them, impacts of IRAP and benefits of working with the program.

Discussion groups with IRAP client firms who have not received contribution funding

Group discussions with unfunded advisory clients were included in the evaluation to provide more evidence about the importance of advisory services. One discussion group, primarily focussing client's journey with IRAP and the benefits of advisory services was conducted in each IRAP region.

Initial recruitment was conducted by the regional office, with confirmation calls completed by the evaluation team. Challenges in recruiting firms for the discussion groups meant that some firms that participated had received funding, and in two of the groups this deferred from the original intent of discussing advisory services to unfunded clients. In order to avoid this in the other groups, a smaller number of unfunded clients were targeted to facilitate discussions about advisory services. The breakdown of the groups was as follows.

Region	Discussion group date	Number of participants
Halifax	December 6, 2016	2
Toronto	January 12, 2017	7
Boucherville	January 19, 2017	6
Vancouver	February 28, 2017	4
Calgary	March 1, 2017	3
Total		22

On-line consultation of IRAP client firms who have not received contribution funding

The on-line consultation (survey) was designed to obtain information on the impacts of advisory services and IRAP's delivery process from firms that had not received funding assistance from IRAP. The sampling universe was the set of all IRAP client firms in SONAR that were indicated as not having received funding. After eliminating undeliverable introductory e-mails and contacts that were not appropriate (e.g., no longer worked at the firm), the final population count was 633. 157 responses were received (25%) of which 93 were suitable for analysis (almost all of the others had received funding from IRAP).

Fifty-one percent of the respondent firms had received business advice, and 39% had received technical advice (with some firms having received both – i.e., they're included in both the 51% and the 39%). The provincial distribution of the 93 responses that were analyzed was as follows:

Province	Number of respondents	Percentage of respondents
Alberta	27	30%
Quebec	20	22%
Ontario	19	20%
New Brunswick	7	8%
Manitoba	6	6%
British Columbia	5	5%
Saskatchewan	4	4%
Prince Edward Island	3	3%
Nova Scotia	1	1%
Northwest Territories	1	1%

Case studies of IRAP contributions to organizations

This line of evidence involved the preparation of 10 case studies of IRAP contributions to organizations (CTOs). First, 15 CTOs (10 primary and five alternates) were selected randomly from the set of CTOs whose clients were surveyed by IRAP in 2016 (so that some of the case study results could be compared with these survey results). The contribution agreements for these CTOs were then reviewed to ensure that they involved the provision of business and technical assistance directly to SMEs, as opposed to "diffuse" services such as organizing workshops or conferences. One CTO was eliminated on this basis, and one was eliminated because of non-responsiveness of the ITA, leaving a sample of 10 CTOs and three alternates. These 10 CTOs were the subject of the case studies.

For each case study 3 SMEs clients of the organization were selected to be interviewed by first asking the organization to supply the names of five clients that had received "significant" services provided under the CTO, and then randomly selecting three of these to interview. It was important to select clients that had received significant services in order to obtain good interview data from the small number of interviews, but it was recognized that there could have been bias by the organization in suggesting SMEs to be interviewed. Several measures were implemented to mitigate against this, the main one of which was to review the results of the interviews of client firms with the ITA for the CTO to confirm their validity. The main issues explored in the case studies were about the benefits of the services to client SMEs, if SMEs were supported in undertaking innovation projects and what had been the impact of the assistance provided by the organisation.

Interviews with clients of the Concierge service

Interviews with Concierge clients were designed to collect information from companies who accessed the Concierge service. Respondents were asked about the advice received from the Concierge Service, if that advice was helpful, and if they were satisfied with the services received from the Concierge Innovation Advisors.

An initial listing of clients was provided by the Concierge program that identified those clients deemed to have had the most engagement with the service. This listing was then sorted by province and Innovation Advisor. A sample of 16 client firms was developed. The sector and provincial distribution of interviewees were as follows:

Sector	Number of respondents
Construction	1
Energy and Natural Resources	2
Entertainment	1
Health and Life Sciences	2
Information, Communications and Technology	3
Retail	1
Total	10

Province	Number of respondents
Alberta	2
British Columbia	5
Ontario	1
Quebec	2
Total	10

Partial benefit-cost analysis

Partial benefit-cost analysis (PBCA) is a method for estimating the economic benefits of programs which are comprised of discrete "projects" and in which the economic benefits of the individual projects are unevenly distributed. (i.e., there are there are certain projects, called "high impact projects," whose economic benefits are much greater than the benefits of the vast majority of projects).

In these cases, the economic benefits of these high impact projects make up a high percentage of the economic benefits of the total program. Because of this, an estimate of the total program benefits can be based on the sum of the benefits of the individual high impact projects. More precisely, the sum of the benefits of the individual high impact projects is a lower bound of the total program benefits, assuming that all the other projects as a whole at least "break even." If the high impact projects are carefully selected, it is a high lower bound. This is the approach taken in PBCA.

In the case of IRAP, the "projects" that were studied were specific ITA-client relationships. The steps involved were as follows:

Step	Description
Identify high impact IRAP-client relationships	This was done by first carrying out a web survey of the ITAs to obtain suggestions regarding potential high impact relationships. This survey was followed by several supplementary data collection activities to identify additional candidate high impact relationships (including the review of a document on IRAP success stories prepared by the Prairie region and consultations with the IRAP Executive Directors in several regions).
Develop a short list of candidate relationships	This involved the review and analysis of the long list of candidates developed in step 1. The result was a short list of 29 candidate high impact relationships. The criteria for selecting these cases were: The information on the relationship had to indicate that the total dollar value of annual benefits of the IRAP assistance was at least $30 million. This $30 million could include benefits to the assisted firm, benefits to the firm’s Canadian customers, and benefits to the entire Canadian industry. However, in order to meet the $30 million criterion, there had to be a credible explanation for benefits in the latter two categories. At the same time 33 “fairly high impact” cases were identified according to the criterion that the total dollar value of annual benefits of IRAP assistance for the firm had to be at least $10 million.
Carry out in-depth data collection	Carry out in-depth data collection for the 29 candidate high impact relationships through the analysis of available data (mainly IRAP data and company websites) and interviews with the ITAs and the companies. These interviews dealt with the description of the IRAP assistance (often provided over many years), the apparent benefits for the company, expected future company revenue growth and profitability related to IRAP assistance, and, where appropriate, benefits to customers and the Canadian industry as a whole. This was done in an iterative fashion (i.e., several interviews with the same person in many cases) and resulted in a list of 21 relationships for detailed study, which was then reduced for methodological reasons to 16 relationships for quantitative analysis.
Calculate the net benefits of each high impact case (on an annual basis)	This was calculated in most cases to be the gross profits (e.g., sales minus associated costs of production and distribution) of the client firms in which IRAP assistance played an important role. (The net benefits of the fairly high impact cases were also quantified.)
Calculate total IRAP program costs	Calculate total IRAP program costs during the period over which IRAP assistance played an important role in the realization of the benefits calculated in step 4 (on an annual basis).
Inflate or deflate the net benefit and program cost	Inflate or deflate the net benefit and program cost streams to year 2016 dollars using the consumer price index.
Discount the following to present value terms	(a) the stream of net benefits for each high impact and fairly high impact case; and (b) the stream of IRAP total program costs.
Add the present values of the net benefits of the high and fairly high impact cases	Add the present values of the net benefits of the high and fairly high impact cases, and compare this with the present value of the total program costs during the calculation period. Two comparisons are made: The sum of the net benefits of the relationships studied minus the IRAP total program costs. This is called the “net present value” of IRAP’s economic benefits. The sum of the net benefits of the relationships studied divided by the ITAP total program costs. This is called the “benefit-cost ratio” for the IRAP program.

The main methodological considerations associated with the steps listed above, especially step 4 were as follows:

• Attribution to IRAP. It was recognized that many factors contributed to the innovation projects of the case study firms – not the least of which were the capabilities of the firms themselves, but which also included support from other government programs, support from financial institutions, advice and assistance from universities, outside investment in the firms, and other factors. Because of the complexity of SME innovation, it is impossible in almost all cases to say that an innovation has been "due to" factor X, and this was not attempted in this study. However, in this study a high bar was set for including the increased profits (gross margins) of these innovative firms in the calculation of benefits of the IRAP program. As a minimum, IRAP had to have played a very important role in supporting the key innovation activities – to the point where these activities likely would have been carried out at a lesser scale, or at a later time, or not at all in the absence of IRAP support.
• Methods used to estimate gross margins. PBCA is based on increases in gross margins (not on operating profit margins) of the assisted firms, because these are mainly what innovation support programs are affecting. The gross margins used in this study are specific to the individual firms or to firms in the same field generally. When the ITAs and the companies were interviewed, they were asked about "typical profit margins in this field." When these interviews did not provide the required information, it was obtained from annual reports of public companies in the field.
• Upper and lower bounds. There are many uncertainties involved in PBCA, not least because the future revenue streams modeled for each firm are uncertain. Therefore, both lower and upper bounds were modeled, which reflect more conservative vs, more optimistic future benefit streams respectively.

The lower bounds reflected the following (varying on a firm-by-firm basis):

• Lower estimates of gross profit margin
• Lower estimates of projected future gross revenues
• Lower estimates of how long the new product/process would remain competitive in the marketplace

The upper bounds reflected:

• Higher estimates of gross profit margin;
• Higher estimates of projected future gross revenues; and
• Higher estimates of how long the new product/process would remain competitive in the marketplace.

Upper and lower bound estimates were also used for calculating the net benefits of the fairly high impact cases.

• Discounting. As indicated in step 7 above, both the net benefit and program cost streams are discounted to present value terms. As a sensitivity test, three different values of the discount rate were used, representing alternate use of the funds: 2%, 5%, and 7%.
• Treatment of investments. This is a complex area in PBCA studies, and there is no generally accepted way of dealing with investments. In this study a middle ground was struck: A conservative point of view was used for the lower bound, in which investments in the firm are treated a financial costs that have to be repaid, and they are included in the company's cost stream. Since, during the R&D stage, investments are indeed benefits to the company, they are treated as such in the upper bound, and they are included in the company's benefit stream.
• Timeframe for the analysis. The 2016 survey of ITAs conducted in this study (step 1 above) obtained data on IRAP benefits back to the year 2000. However, there were few high impact projects prior to 2005, and it appeared that in the ITA survey it was highly likely that the survey results did not include a significant number of high impact projects during 2000-2005 (probably due to the non-availability of a number of ITA respondents who had been active during that period). Therefore, the benefit-cost analysis was based on individual company benefits and IRAP program costs beginning in 2005.

Limitations and mitigation strategies

The evaluation approach did include some limitations and challenges, which are discussed below, along with the strategies undertaking to mitigate them.

Limitation/challenge	Mitigation Strategy
Small sample sizes for some data collection methods, in particular for the interviews of funded firms (n=25) and the interviews of Concierge clients (n=10). This limitation is not applicable to the staff/expert interviews, which are used to obtain a range of informed opinions, not a consensus. It is also not applicable to the site visit interviews and discussion groups, which are only used as supplementary data sources.	Regarding the funded firm interviews, there are no quantitative analyses of these results, except for descriptive analyses. Also, for each evaluation question for which this is a primary data source, there are at least three other primary data sources addressing the same question. The use of the Concierge client interviews was minimal, so no mitigation measures were necessary.
Possible response bias in the survey of unfunded advisory clients (possibility that clients who perceived greater benefits from IRAP assistance were more likely to respond).	This was a primary data source for the sub-question on the impact of advisory assistance on strengthening innovation capacity. The findings were supplemented by the opinions of the ITAs in the focus groups, the site visit discussions with advisory-only firms, and the opinions of funded firms regarding the impact of advisory assistance.
There are a number of uncertainties underlying the PBCA results.	These uncertainties have been taken into account in the sensitivity tests (upper and lower bounds; three discount rates). Also, where there are methodological uncertainties (e.g., with regard to attribution to IRAP), the analysis has been based on conservative assumptions.

Appendix B: IRAP logic model

Appendix C: Evaluation framework

Evaluation Questions	Sub-Questions	Indicators	Data Collection Approach
1. Is there a need to support SME innovation in Canada through financial or advisory support?	1.1 What are the current needs for SME innovation support in Canada?	Description of importance of innovation to Canada	Document review Interviews with IRAP SLT, subject matter experts
		Description of SME need for financial assistance and advisory services	Document review Interviews with funded firms, subject matter experts, financial partners
	1.2 What are the synergies between the various components of NRC-IRAP?	Description of the various NRC-IRAP programs and services, where they fit in the innovation continuum and whether any gaps exist	Document review Interviews with Innovation Advisors, IRAP regional directors, IRAP SLT, subject matter experts
		Perceptions that new or temporary programs delivered by NRC-IRAP are strategically aligned to their objectives and address gaps in the innovation continuum	Interviews with Innovation Advisors, IRAP regional directors, IRAP SLT, subject matter experts
	1.3 What are the synergies between NRC-IRAP and other available innovation support in Canada?	Perceptions that other support programs compliment or overlap NRC-IRAP funding and advisory services	Document review Interviews with Innovation Advisors, IRAP regional directors, subject matter experts Focus groups with ITAs
		Perceptions that NRC-IRAP funding aids or hinders SMEs in accessing other support	Interviews with Innovation Advisors, IRAP regional directors, subject matter experts, financial partners, funded firms Focus groups with ITAs
2. Has NCR-IRAP met the needs of Canadian SMEs for innovation support?	2.1 Has NRC-IRAP reached its intended clients?	Program funds have been fully disbursed (no lapsed Gs and Cs funding)	Review of administrative/performance data
		# of SMEs receiving funding by sector and region	Review of administrative/performance data
		# of organizations receiving funding by sector and region	Review of administrative/performance data
		# of SMEs receiving services from organizations	Review of administrative/performance data
		Gender and age of YES funded interns	Review of administrative/performance data
		% of new SMEs receiving funding each year (trend over the last 10 years)	Review of administrative/performance data
		# of new SMEs introduced to IRAP through new or temporary funding programs	Review of administrative/performance data
		# of SMEs receiving advisory services by sector and region	Review of administrative/performance data
		New SMEs receiving advisory services each year	Focus groups with ITAs Interviews with IRAP regional directors Online consultation with unfunded firms that received advisory services
	2.2 Has Concierge contributed to greater awareness and access to innovation support in general and of NRC-IRAP in particular?	# of client website visits by year	Review of administrative/performance data
		# of client contact centre calls by year	Review of administrative/performance data
		# of client referrals to partners by year	Review of administrative/performance data
		# of client transfers from partners by year	Review of administrative/performance data
		% of clients satisfied with relevancy of identified innovation resources	Review of administrative/performance data
		Overall satisfaction with service provided through Concierge	Interviews with Concierge clients (SMEs)
		Perceptions of the benefits of Concierge	Interviews with Concierge clients (SMEs), Innovation Advisors, IRAP regional directors, IRAP SLT, subject matter experts Focus groups with ITAs
3. Is NRC-IRAP consistent with government priorities?	3.1 What are the priorities of the government in terms of innovation support to SMEs?	3.2 Description of priorities in government strategic documents and budgets	Document review
	How does NRC-IRAP align with these priorities?	Alignment of IRAP objectives to priorities	Document review
4.0 Is NRC-IRAP consistent with federal roles and responsibilities?	4.1 Is it the role of the federal government to deliver a program in support of SME innovation?	Description of role of government in supporting SME innovation (including advisory services)	Document review
		Description of role of government in international R&D support programs (including advisory services)	Document review
5. Has NRC-IRAP supported SME innovation?	5.1 Were the advisory services provided by NRC-IRAP beneficial to SMEs?	Type of advisory services provided	Review of administrative/performance data Focus groups with ITAs Online consultation with unfunded firms that received advisory services Interviews with IRAP regional directors
		Changes implemented by SMEs as a result of advisory services and their benefits	Review of administrative/performance data Interviews with funded firms Online consultation with unfunded firms that received advisory services
		Other outcomes of advisory services as reported by clients	Interviews with funded firms Online consultation with unfunded firms that received advisory services
		Satisfaction with advisory services provided	Review of administrative/performance data Interviews with funded firms Online consultation with unfunded firms that received advisory services
		Complaints/feedback about ITAs	Review of administrative/performance data
	5.2 Were the services provided by funded organizations to SMEs beneficial?	# of SMEs receiving services from organizations	Review of administrative/performance data
		Type of services provided by organizations	Review of CTO CAs Case studies with CTOs
		SME outcomes as a result of services provided by organizations	Review of CTO CAs Case studies with CTOs
		Satisfaction with services provided by organizations	Review of administrative/performance data Case studies with CTOs
	5.3 Has NRC-IRAP funding supported SMEs in successfully undertaking innovation projects?	# of projects funded	Review of administrative/performance data
		# of internships funded	Review of administrative/performance data
		% of projects achieving all or most anticipated technical objectives (over the last ten years)	Review of administrative/performance data
		Number of projects commercialized	Focus group with ITAs Interviews with funded firms Case studies with CTOs
		% of fully completed internships through YES (over the last ten years)	Review of administrative/performance data
		% of interns who stayed with the SME following the contribution through YES (over the last ten years)	Review of administrative/performance data
		Client perceptions on project/internship success and outcomes	Interviews with funded firms (those funded through Youth) Case studies with CTOs
		Promotion of and use of NRC services (expertise and facilities)	Review of administrative/performance data (available from portfolios) Interviews with funded firms, IRAP SLT
6. Has NRC-IRAP supported increased SME performance?	6.1 If yes, what has been the impact of NRC-IRAP in increasing SME's innovation capacity, on SMEs growth, increased productivity and profitability?	Increase in R&D personnel in clients (over the last ten years)	Review of administrative/performance data
		Increase in R&D investments by clients (over the last ten years)	Review of administrative/performance data
		Increase in client revenue (over the last ten years)	Review of administrative/performance data
		Increase in client income (over the last ten years)	Review of administrative/performance data
		Increase in client employment (over the last ten years)	Review of administrative/performance data
		Increase in client productivity (revenue/employees) (over the last ten years)	Review of administrative/performance data
		% of revenue generated by sales of products and services developed with IRAP support	Interviews with funded firms Case studies with CTOs
		Extent of investments from third parties	Interviews with funded firms Case studies with CTOs
		Perceptions on increased innovation capacity, growth, productivity and profitability and extent attributed to NRC-IRAP	Interviews with funded firms, financial partners Case studies with CTOs
	6.2 If yes, how does NRC-IRAP decide whether to invest a particular client or project?	Perceptions of when NRC-IRAP funding should no longer be provided to an SME	Interviews with funded firms, IRAP regional directors, IRAP SLT, IRAP DS Focus group with ITAs Review of CTF CAs
		Impact of no longer receiving NRC-IRAP funding	Interviews with funded firms (no longer funded), IRAP regional directors, IRAP SLT, IRAP DS Focus group with ITAs Interviews with firms unsuccessful in getting IRAP funding
7. Has NRC-IRAP resulted in positive impacts on the Canadian economy?	7.1 If yes, has NRC-IRAP contributed to wealth creation in Canada?	# of jobs supported	Review of administrative/performance data
		Increase in employees by year	Review of administrative/performance data
		% of Growth of IRAP clients	Review of administrative/performance data
		Perceptions of the extent of NRC-IRAP's economic impact	Interviews with IRAP SLT, subject matter experts
		Cost-benefit of NRC-IRAP (average return in $ to the Canadian economy per $ of program costs)	Review of administrative/performance data
8. What have been other benefits of NCR-IRAP to SMEs?	8.1 What would have been the impact on the project in the absence of NRC-IRAP?	Perceptions of SMEs on impacts of IRAP and what would have occurred differently in its absence (ex. Project not undertaken, taken longer or smaller scope, leverage other funds etc.)	Interviews with funded firms Case studies with CTOs
		Discussion with SMEs engaged in innovation that are not funded by IRAP on challenges, benefits, accomplishments before and after ITA guidance.	Online consultation with unfunded firms that received advisory services Interviews with firms unsuccessful in getting IRAP funding
		SMEs that received IRAP funding following advisory services	Focus group with ITAs Interviews with funded firms
	8.2 Have there been any unintended outcomes as a result of NRC-IRAP?	Description of unintended program outcomes	Interviews with funded firms, IRAP SLT, subject matter experts, Concierge clients Focus groups with ITAs Case studies with CTOs
9.0 Have current structures and processes facilitated efficient and effective delivery of NRC-IRAP?	9.1 Are the necessary administrative processes and mechanisms in place to effectively deliver the program?	Description of current administrative processes and mechanisms including but not limited to: -standard operating procedures, -accelerated project review, -delivery processes for the various funding programs, -onboarding processes for new and temporary programs, -use of field manual for consistent program delivery by ITAs, -client complaint management mechanism.	Document review Interviews with IRAP regional directors, IRAP DS Focus group with ITAs
		Perceptions of benefits and challenges as well as gaps in existing administrative processes and mechanisms	Document review Interviews with IRAP regional directors, IRAP DS Focus group with ITAs
		Client satisfaction rating on service delivery metrics (timelessness, accuracy, access and professionalism)	Review of administrative/performance data Interviews with funded firms Case studies with CTOs Online consultation with unfunded firms that received advisory services
	9.2 What has been the impact of temporary funding programs on NRC-IRAP’s program delivery capacity?	Perceived impact of temporary funding programs on NRC-IRAP	Interviews with IRAP regional directors, IRAP SLT, IRAP DS, subject matter experts Focus group with ITAs
10. Have available resources and support facilitated efficient and effective delivery of NRC-IRAP?	10.1 Are the roles and responsibilities of program delivery and support staff clear and widely understood?	Job descriptions for delivery and support staff	Document review Interviews with IRAP SLT, IRAP DS
		Extent of gaps or overlap in tasks	Document review Interviews with IRAP regional directors, IRAP SLT, IRAP DS
		Perception of roles and responsibilities and whether they are widely understood	Interviews with IRAP regional directors, IRAP DS
	10.2 Have resources been appropriately allocated to facilitate effective and efficient delivery of NRC-IRAP?	Financial resource allocation (ten year trend)	Review of administrative/performance data
		Human resource (FTE) allocation (ten year trend)	Review of administrative/performance data
		Perceptions that human resources are sufficient to sustain program delivery including delivery of new and temporary programs	Interviews with IRAP regional directors, IRAP SLT, IRAP DS
		Challenges related to recruitment of ITAs and impact on program delivery	Interviews with IRAP regional directors, IRAP SLT, IRAP DS Focus group with ITAs
		Extent that common services meet needs considering decentralized nature of program delivery	Interviews with IRAP regional directors, IRAP DS Focus group with ITAs
		Extent that IT tools available meet needs	Interviews with IRAP regional directors, IRAP DS Focus group with ITAs
		Extent that administrative support meets needs	Interviews with IRAP regional directors, IRAP DS Focus group with ITAs
		NRC Key Performance Indicators related to resources	Review of administrative/performance data
		Time spent on ITA tasks (admin, client site visits etc.)	Interviews with IRAP regional directors, IRAP SLT, IRAP DS Focus group with ITAs

Footnotes

Footnote 1

Innovation Canada, A Call to Action: Review of Federal Support to Research and Development – Expert Panel Report (commonly called "the Jenkins Report"), Industry Canada, 2011.

Return to footnote 1 referrer

Footnote 2

Report on the 2012 Evaluation of the Industrial Research Assistance Program, National Research Council, 2012.

Return to footnote 2 referrer

Footnote 3

The Global Competitiveness Report, World Economic Forum, 2013.

Return to footnote 3 referrer

Footnote 4

People and Excellence: The Heart of Successful Commercialization, Final Report of the Expert Panel on Commercialization, Industry Canada, 2005.

Return to footnote 4 referrer

Footnote 5

Message of the Minister of Industry Canada in the 2014-15 Departmental Performance Report of the National Research Council.

Return to footnote 5 referrer

Footnote 6

Budget Plan 2016, Finance Canada, 2016.

Return to footnote 6 referrer

Footnote 7

Budget Plan 2017, Finance Canada, 2017.

Return to footnote 7 referrer

Footnote 8

Innovation Canada, A Call to Action: Review of Federal Support to Research and Development – Expert Panel Report (commonly called "the Jenkins Report"), Industry Canada, 2011.

Return to footnote 8 referrer

Footnote 9

How IRAP Works, National Research Council Industrial Research Assistance Program, 2016.

Return to footnote 9 referrer

Footnote 10

NRC's President was mandated to conduct an internal assessment of NRC to determine how best to support the Government's Innovation Agenda. NRC Dialogue consisted of organization-wide consultations to take stock of the current state of NRC and how to move forward to ensure its continued relevance to Canada.

Return to footnote 10 referrer

Footnote 11

Over the evaluation period, IRAP also funded projects by other organizations such as business centres and municipal agencies. However these made up 1% of funded organizations or less nationwide therefore were not included in the figure.

Return to footnote 11 referrer

Footnote 12

Some data were obtained in case studies of advisory services, but even then the conclusions were based on the combined impacts of advisory services and IRAP funding.

Return to footnote 12 referrer

Footnote 13

See, e.g., Cost-Benefit Analysis, 5th Edition, E.J. Mishan and Euston Quah, Routledge, New York, New York, 2007.

Return to footnote 13 referrer