Evaluation of the Canada Accelerator and Incubator Program (CAIP)

List of acronyms

A/I

accelerator or incubator

CA

contribution agreement

CAIP

Canada Accelerator and Incubator Program

FY

fiscal year (of the federal government)

HIF

high impact firm

ITA

Industrial Technology Advisor

IRAP

the Industrial Research Assistance Program (of the NRC)

NRC

the National Research Council of Canada

NSERC

the Natural Sciences and Engineering Research Council of Canada

PBCA

partial benefit-cost analysis

SME

small or medium-sized enterprise

Executive summary

The Canada Accelerator and Incubator Program (CAIP) is a contribution program initiated in 2014 with a budget of $100 million over five years, FY2014‑15 to FY2018‑19. It funds accelerators and incubators^{Footnote 1} (A/Is) through contribution agreements to undertake new activities or offer increased levels of service to early‑stage firms. The program objective is:

Two evaluations of CAIP were required. In fall 2016, a mid-term evaluation of CAIP focused on the relevance and implementation of the program. The current evaluation assesses the extent that objectives have been achieved and explores the continued relevance of the program as well as the efficiency of its delivery.

The evaluation used several lines of evidence including a document and data review, an online survey of clients of the CAIP-funded A/Is, interviews with A/Is, program delivery staff and external stakeholders as well as a benefit-cost analysis.

Some limitations should be kept in mind when interpreting the results of this evaluation. Most relate to the validity and quality of the data available for the evaluation such as the incompleteness of the performance data that A/Is were required to provide to IRAP, having to collect much of the outcomes information about client-firms from the A/Is (as opposed to from the clients themselves), and some reluctance by A/Is to fully participate in the data collection, in part due to a perceived administrative burden. Some of the risks resulting from these limitations were mitigated by conducting extensive cleaning of data and validating some claims about client outcomes with a sample of client-firms.

The overall findings of the evaluation are as follows:

• CAIP continues to address a need and to align with government priorities. CAIP is well aligned with the government's priorities regarding support for SMEs. This includes recent priorities such as providing a broader range of support to SMEs and strengthening Canada's network of accelerators and incubators.
• Based on data available for the evaluation, CAIP has enabled funded A/Is to expand their reach to a large number of firms. CAIP has also helped A/Is to expand and increase their capabilities, allowing them to work with more mature firms, rather than only small, early-stage firms.
• CAIP allowed funded A/Is to provide a wide range of new or expanded services to their clients that would not have been possible without the program. Overall, surveyed client-firms have found the services useful and have incurred numerous benefits as a result.
• Based on data provided by A/Is, client-firms as a group have grown significantly in revenue and equity investment. Firms that have experienced especially high revenue or equity growth believe the A/I assistance was an important factor in enabling them to achieve this growth. While the study suggests modest benefits overall, CAIP has contributed to a positive net economic benefit for the Canadian economy.
• The cost to deliver CAIP is more than double the amount planned at the onset of the program. This is in part due to the complexity of the program, which led to unforeseen challenges. Overcoming these challenges has required a considerable level of effort. As a result, there are many learnings that should be considered for any future similar program.

Because CAIP is ending in March 2019, recommendations from the evaluation aim to provide guidance to delivery agents of future, similar programs. The objectives of the recommendations are to ensure a more efficient program delivery, as well as to ensure that information on performance is available:

Recommendation 1: Sufficient time for careful planning and development of administrative requirements and processes should be allocated prior to the launch of new programs similar to CAIP.

Recommendation 2: Sufficient resources should be allocated for program delivery of new programs similar to CAIP once the level of effort required is understood.

Recommendation 3: In the case of programs similar to CAIP, reporting requirements should be clearly specified (in accordance with a well-defined performance framework) and understood prior to signing a contribution agreement.

Recommendation 4: For programs such as CAIP, where beneficiaries engage through an intermediary, programs should consider working with other innovation and capacity support programs to develop a concerted approach to collecting performance data. This would ensure information is available to assess the value of government investments.

1. About CAIP and the current evaluation

The Canada Accelerator and Incubator Program (CAIP) provides non-repayable contribution funding to 16 Canadian accelerators and incubators (A/Is). The funding is meant to support the A/Is in undertaking new activities or in offering increased levels of service. The objective of the program is to establish a critical mass of outstanding A/Is that can develop innovative, high-growth Canadian small- and medium-sized enterprises (SMEs).

CAIP is an initiative of the Government of Canada's Venture Capital Action Plan and was conceptualized and designed by Finance Canada. NRC-IRAP was selected to deliver CAIP due to its experience in the delivery of contribution funding to organizations including A/Is. CAIP was announced in 2013 and the program was launched in 2014. This sunsetting program is due to end in March 2019.

CAIP had a budget of $100 million to be distributed over five years. The funding allocated to the A/Is during the first four fiscal year of the program (FY 14-15 through 17-18) plus commitments for the current year totals $92,990,612.

The distribution of funding amounts across A/Is varies from under $1 million to over $11 million over the five-year program period.

Exhibit 1
Funding allocation across the CAIP-funded A/Is

Long description of Exhibit 1: Funding allocation across the CAIP-funded A/Is

This is a horizontal bar chart that shows the value range of the CAIP contribution agreements and the number of CAIP-funded A/Is that received funding in each range. The chart shows that the distribution of funding amounts across A/Is varies from under $1 million to over $11 million.

Number of CAIP-funded A/Is	Contribution agreement value
2	$0 - $2M
4	$2 - $4M
4	$4 - $6M
1	$6 - $8M
3	$8 - $10M
2	$10 - $12M

Program requirements called for two evaluations of CAIP. In fall 2016, a mid-term evaluation of CAIP was completed. The evaluation took place early in the program's lifecycle, and as a result focused on the relevance and implementation of the program. This current evaluation considers the full lifespan of the program, assessing the extent to which program objectives have been achieved. The evaluation also explores the continued relevance of the program, as it nears its end, as well as the efficiency of its delivery. Findings from this evaluation can provide important insight and guidance to organizations tasked with the delivery of any future iterations of the program, or similar programs.

Evaluation methodologies included:

• Document and data review, including a detailed analyses of A/I performance data
• On-line survey of client-firms that the A/Is reported as having been assisted by CAIP funding (n=549)
• Interviews with a sample of high-growth client-firms (n=46)
• Interviews with the A/Is (n=15)
• Interviews with delivery staff (n=5) and external stakeholders (n=5)
• Benefit-cost analysis

Some limitations should be kept in mind when interpreting the results of this evaluation. Most are related to the validity and quality of the data available for the evaluation:

• Data about performance has been self-reported by A/Is over the course of the program. Overall, data provided was incomplete and required extensive cleaning.
• It was necessary to collect much of the client-firm information required for the benefit-cost analysis from the A/Is, potentially limiting the accuracy and depth of the data. This was mitigated to a certain extent by validating information through interviews with a sample of the client-firms^{Footnote 2}.
• There was reluctance by some A/Is to support some of the data collection activities, mainly because they perceived it as a burden to their clients. This required negotiations and resulted in tighter timelines and concessions to the study (e.g., relying on A/Is for information about client-firms).

A full description of the evaluation methodology and limitations is provided in Appendix A.

2. CAIP continues to align with evolving government priorities related to support for innovative SMEs

CAIP continues to address a need and to align with government priorities. CAIP is well-aligned with the government's priorities regarding support for SMEs. This includes recent priorities such as providing a broader range of support to SMEs and strengthening Canada's network of accelerators and incubators.

Supporting evidence

The history of federal government support for SMEs' innovation activities (such as advisory assistance and financial contributions to support company growth) dates back to the introduction of the Industrial Research Assistance Program in 1962. More recently, the continued need for direct support to SMEs is well documented in numerous government policy papers, budget documents, and expert panel reports, such as the 2005 Expert Panel on Commercialization. Federal government support for SME innovation activities continued to be a focus of government policies until after the Review of Federal Support to Research and Development – Expert Panel Report (the Jenkins report) in 2011.

The Jenkins report recommended a range of additional measures to support business innovation by SMEs. This was based on their conclusion that there was a need for broader assistance to help overcome the disadvantages of being at a smaller scale and a need to lower barriers to getting started. The government acted on this recommendation, and in Budget 2012, $400M was committed to support business innovation. In January 2013, the Venture Capital Action Plan was announced to deploy this funding. Budget 2013 announced several new initiatives, including CAIP, additional funding for the Business Development Bank of Canada, and the Entrepreneurship Awards. Additional innovation support measures designed to help SMEs become more investment-ready followed in subsequent budgets, especially Budgets 2016, 2017, and 2018, as part of the government's Innovation Agenda.

CAIP provides support for SME growth, and is an example of the government funding a range of services needed by SMEs, such as mentoring and coaching services; assistance with the preparation of business plans and marketing plans; facilitation of market and partner introductions; training in sales, business finance, and business management; and assistance with commercialization activities.

While CAIP is meant to support the growth of innovative SMEs, it was also established to support the development of Canadian accelerators and incubators. During interviews conducted for the evaluation, senior government officials involved in the formulation of SME innovation support policies noted that an evolving government priority at that time was to strengthen Canada's network of A/Is. This was based on the realization that Canada lagged far behind other countries (especially the US):

"At that time the A/I ecosystem in Canada was very nascent compared to other countries. Many A/Is were at the early stage of development."

"There was a need to grow the A/Is, to enable them to offer high quality services and advance toward self-sufficiency."

This priority was embedded into the design of CAIP, and many features of CAIP reflect this priority, such as the broad range of A/I activities eligible for funding and the requirement for the A/Is to use CAIP for incremental activities (activities they would not have been able to carry out without CAIP funding).

3. Available data suggests that CAIP-funded A/Is have increased their numbers of client firms, which in many cases have evolved from small, early-stage firms to more mature firms

Based on data available for the evaluation, CAIP has enabled funded A/Is to expand their reach to a large number of firms. CAIP has also helped A/Is to expand and increase their capabilities, allowing them to work with more mature firms, rather than only small, early-stage firms.

Supporting evidence

Program data collected by IRAP shows that between 2014 and 2017, CAIP-funded A/Is provided services to 8,520 client firms. Some of these firms were linked to multiple CAIP-funded A/Is; program data shows that one third (2,029 firms) received service from more than one A/I.

Many client-firms have long-term relationships with their A/Is. Based on results of the client-firm survey, almost two thirds (61%) had been receiving assistance for a year or more, with the greatest number of respondents having received assistance for two years or more as shown in Exhibit 2.

Exhibit 2
Most survey respondents received services for more than one year

Long description of Exhibit 2: Most survey respondents received services for more than one year

This is a vertical bar chart that shows the length of time survey respondents received services from CAIP-funded A/Is. The chart shows that most survey respondents received services for more than one year.

Time Frame	Number of respondents
3 months or less	46
3 to 6 months	64
6 months to 1 year	104
1 to 2 year	151
2 years or more	184

Further, CAIP funding has resulted in a considerable increase in the number of client-firms being served. Data provided by the A/Is to IRAP suggests that the number of new active client firms being served has grown steadily over the last four years, with the greatest increase occurring between year 1 and year 2. (Exhibit 3)

Exhibit 3
Available data suggests that the number of new active firms has increased considerably^Footnote 3

Long description of Exhibit 3: Available data suggests that the number of new active firms has increased considerably

This is a line chart that shows the increase in the number of new active firms being served by the CAIP-funded A/Is. It shows that the greatest increase happened between 2014 and 2015 (117.2% growth).

Year	Number of new active clients	Growth in new active clients between years
2014	703	-
2015	1527	117.2%
2016	2448	60.3%
2017	2989	22.1%

While the respondents of the survey represent a small proportion of client-firms, and results cannot be generalized due to the sampling approach used, they do provide some insight on the type of client-firms served by CAIP-funded A/Is.

Most survey respondents are considered micro-enterprises^{Footnote 4} as the greatest number of firms indicated they employ between two and four employees (23%), and overall 65% of client firms had less than 10 employees. Further, most noted being six years old or less (69%). The largest proportion of survey respondents were from firms in the information, communications and technology sector (32%). The overall distribution of survey respondents among sectors is presented in Exhibit 4.

Exhibit 4
The largest proportion of survey respondents are from the information and communications technology sector

Long description of Exhibit 4: The largest proportion of survey respondents are from the information and communications technology sector

This is a donut chart that shows the sector that survey respondents are from. The chart shows that the largest proportion of survey respondents are from the information and communications technology sector (32%).

Sector	Percentage of survey respondents
Information and communications technology	32%
Agriculture and agri-food Manufacturing Energy Retail and wholesale Transportation Natural resources Construction	22%
Science and technology	19%
Health and health care	16%

Along with survey results, interviews with A/Is also provided insight into client-firms served. Of particular interest is the change in the type of clients served over the study period. Some A/Is noted that their clients now tend to be more mature (e.g., more serial founders, looking more for growth and scale) and their pre-revenue clients are decreasing. While A/I clients are still SMEs and early stage firms, they now seem to be at a later stage of growth.

Many A/Is noted that their ability to serve more mature firms is a result of increased capacity and capabilities, facilitated by CAIP. Along with assisting SMEs, CAIP was intended to support A/Is that had the potential to grow and increase their capabilities, thereby strengthening the network of A/Is in Canada. According to the CAIP-funded A/Is themselves, there has been success in this respect.

The selection of the A/Is was done through a Request For Proposal (RFP) process. According to the RFP, recipients were to be market-driven and contributing to a sustainable venture capital system. Most of the A/Is that were successful and received funding (13 out of the 16) were mature organizations, having provided accelerator/incubator services for at least five years prior to CAIP.

According to A/Is, over the 4 year period of CAIP funding, they have been able to:

• Strengthen their profile, reputation and credibility
• Extend their reach to provide regional capacity, and in some cases national capacity
• Increase the size of deals being made
• Increase their ability to provide more specialized services and domain-specific expertise (as a result of increased internal strength or the ability to contract)
• Extend their services to additional sectors
• Increase their collaborator, service provider, and partner networks
• Assist companies at different stages of development, from start-up to growth to scale
• Obtain infrastructure that did not exist before (e.g., additional hardware development lab facilities or tenant and prototyping space)

4. CAIP-funded A/Is have delivered new or expanded services which surveyed client-firms have valued

CAIP allowed funded A/Is to provide a wide range of new or expanded services to their clients that would not have been possible without the program. Overall, surveyed client-firms have found the services useful and they have incurred numerous benefits as a result.

Supporting evidence

As previously indicated, CAIP provided funding to the selected A/Is so that they could provide new services, or expand their existing services. On average, CAIP supported 30% of the services provided by A/Is over the past four years. Individually, the portion of A/I service offerings supported by CAIP funding varied substantially across the A/Is, from funding 5% to 70% of their programming. (Exhibit 5)

Exhibit 5
The percentage of A/I services supported by CAIP varied across A/Is

Long description of Exhibit 5: The percentage of A/I services supported byCAIP varied across A/Is

This is a scatter chart that shows the percentage services supported byCAIP. The chart shows that the portion of A/I service offerings supported byCAIP funding ranged from 70% to 5%.

CAIP-funded A/I	Percentage of their services that is funded byCAIP
BC Technology Industry Association	70%
Centre d'entreprises Montreal	51%
PEI BioAlliance	50%
Ryerson University	43%
Bioenterprise Corp	40%
Biomedical Commercialization Canada (Manitoba Tech)	40%
Propel ICT	32%
Invest Ottawa	30%
Centre for Drug Research and Development	20%
Innocentre Quebec	12%
University of Alberta	11%
Communitech	10%
MaRS	5%

The range of services supported byCAIP also varied. Analysis of the program data provided to IRAP by the A/Is revealed that they delivered a broad range of services that fall into the following general categories:

• Physical space
• International "soft-landing" (Export orientation, new markets)
• Mentoring/Coaching (Leadership development)
• Training/Education
• Promotion/Communication
• Market Research
• Attracting equity

Based on results of the client-firm survey, A/Is had processes in place to select the firms they would work with and deliver their services to (Exhibit 6). Most survey respondents (78%) identified at least one mechanism and in some cases, more than one mechanism that was used to help A/Is identify the best candidates for their services.

Exhibit 6
Examples of selection processes used by A/Is to select their client-firms

The survey of client-firms also provides insight into how A/Is identified the services that would benefit their clients. Almost two thirds of respondents (61%) said they worked with their A/Is to collaboratively determine the services they would receive. Smaller percentages of respondents said they had services recommended to them by their A/I (24%) or that they had requested specific services from their A/I (13%).

Respondents of the client-firm survey noted they had received both business support and technical support, and, in most instances, client firms received more than one service. The service type received by the largest proportion of client firms was business planning assistance (90%). The least noted type of service assistance was the provision of laboratories and/or equipment for use (including testing services), with only 30% of client firm survey respondents having received this service. Exhibit 7 identifies the proportion of firms receiving, or not, each identified service area.

Exhibit 7
Survey respondents received a wide range of services fromCAIP-funded A/Is

Long description of Exhibit 7: Survey respondents received a wide range of services fromCAIP-funded A/Is

This is a horizontal bar chart showing the type of services that survey respondents indicated they received fromCAIP-funded A/Is. The chart shows that the service received by the largest proportion of survey respondents was "business planning" (90%)

Service Area	Have received service	Have NOT received service
Business planning	90%	10%
Introductions	80%	20%
Market analysis	80%	20%
Commercialization	79%	21%
Events (ex. Conferences, trade shows)	78%	22%
Financial analysis/company management	72%	28%
Securing investors	67%	33%
Office space/support services	60%	40%
Prototyping/scale-ups	51%	49%
Use of laboratories/equipment	30%	70%

When prompted to note any other service received beyond the existing categories, many survey respondents mentioned access to networks. This included connections to academia, experts, more senior talent, interns, a community of other entrepreneurs and start-ups, the local community, customers, industry, and government.

A/Is noted that theCAIP-supported services they provided to their client-firms were new or complimentary to what was in existence prior, providing a next step along the continuum of services, or increasing abilities to expand networks of consultants and internal expert resources for mentoring.

The new and expanded services provided by the A/Is would not have been delivered in the absence ofCAIP. According to A/Is, prior toCAIP, they did not have the means to expand. Further, in order for A/Is to maintain their relevance, they needed to increase their expertise, knowledge, and international/global supports, and develop new programming. In particular, the means to help start-up companies grow and scale-up was a gap that needed to be filled.CAIP allowed A/Is to address this gap.

Beyond A/Is not being able to expand their client base, as described above, other examples noted by A/Is of how services would have been affected in the absence ofCAIP are:

• Limited provision of in-depth advice from experts.
• Not being able to meet client needs for growth-related services, over and above the "start-up" programming that was previously available.
• Programming would not be at the same scale or developed and delivered as quickly, or at all.

According to program data, A/Is generally met or exceeded targets for the delivery of new and expanded services as identified in their contribution agreements. Deliverables related to direct service support to clients (e.g., mentoring) were most often exceeded, while deliverables related to promotion and communication type activities were slightly below targets. An example of an unmet deliverable target was the number of peer group sessions delivered in one year; where the target was four sessions, and only three sessions were provided. Another example was the commitment to deliver two showcase events and only delivering one. (Exhibit 8)

Exhibit 8
Available data suggests that each year, most A/I deliverables exceeded targets

Long description of Exhibit 8: Available data suggests that each year, most A/I deliverables exceeded targets

This is a vertical bar chart that shows the number of deliverables that exceeded targets during three years ofCAIP. The chart shows that each year, most deliverables exceeded targets.

Year	Number of Deliverables
	Below	Met	Exceed	Grand Total
Year 1	15	21	63	99
Year 2	17	17	84	118
Year 3	29	18	98	145

Overall, services provided byCAIP-funded A/Is were useful, however some more than others. Respondents of the client-firm survey were asked to rate the usefulness of the services they received. According to survey results, the most useful service was the provision of office space or support services, including ICT services. A number of other services followed closely in their perceived usefulness (Exhibit 9). The lowest rated service was assistance with securing investors, where the responses were split almost evenly between useful and not useful.

Exhibit 9
Survey respondents rate services provided byCAIP-funded A/Is between moderately useful and useful

Long description of Exhibit 9: Survey respondents rate services provided byCAIP-funded A/Is between moderately useful and useful

This is a horizontal bar chart that shows how survey respondents rated the usefulness of the services they received fromCAIP-funded A/Is. The chart shows that, according to survey respondents, the most useful service was the provision of office space and support services (with a mean of 4.3 based on a five-point scale).

Service	Mean
Office space/support services (n=330)	4.3
Business planning (n=494)	3.9
Events (ex. Conferences, trade shows) (n=430)	3.8
Market analysis (n=440)	3.7
Use of laboratories/equipment (n=164)	3.7
Commercialization (n=434)	3.7
Financial analysis/company management (n=398)	3.6
Introductions (n=440)	3.6
Prototyping/scale-ups (n=282)	3.4
Securing investors (n=369)	3.3

It can be assumed that the extent to which clients perceive a service to be useful is, in part, related to the outcomes of that service. For instance, survey respondents were asked to identify the various ways in which the services they received have assisted them. Approximately half of respondents noted that the services received resulted in:

• An increased number of contacts with potential partners, suppliers, or customers
• The preparation of a sound business plan or marketing plan
• The initiation of commercialization activities

It can be assumed that the extent to which clients perceive a service to be useful is, in part, related to the outcomes of that service. For instance, survey respondents were asked to identify the various ways in which the services they received have assisted them. Approximately half of respondents noted that the services received resulted in:

• An increased number of contacts with potential partners, suppliers, or customers
• The preparation of a sound business plan or marketing plan
• The initiation of commercialization activities

Exhibit 10
Survey respondents were assisted through A/Is services in numerous ways

Long description of Exhibit 10: Survey respondents were assisted through A/Is services in numerous ways

This is a bar chart that shows the results of the services provided byCAIP-funded A/Is that survey respondents indicated were achieved. The chart shows that about half of respondents noted that the services they received resulted in increased contacts, commercialization activities and the preparation of a business or marketing plan.

Result of Service Received	Percentage of survey respondents
Increased contacts (clients, partners)	50%
Initiated commercialization activities	46%
Prepared a business plan or marketing plan	46%
Improved management of the company	44%
Undertook proof of concept, testing, validation, prototyping	40%
Increased contacts (investors)	37%
Completed development of product(s)	30%
Increased internal investment in R&D	23%
"Graduated" from A/I program	21%
Developed an export strategy	14%
Other	9%

According to survey respondents as well as A/Is, other ways in which services affected client-firms' evolutions include:

• Learning and allowing for significant pivots in business focus (e.g., changing target customers or changing sector of application)
• Exploring and expanding into new markets, including global markets that in some cases the client-firm did not consider
• Increasing knowledge and business acumen across all dimensions, beyond the technology and into sales, customer attraction, network creation, and self-accountability
• Market validation and due diligence support to make quick decisions or validation of product or market fit, leading to go/no go decisions to create/develop a business
• Better commercial understanding of what is needed to get technologies to market
• Accessing networks, particularly peer networks, important for learning and growth of early-stage firms. Ad hoc or intentional "collisions" were noted as leading to outcomes for companies. All A/Is noted their belief in the "network effect"
• De-risking investment for other public/private investors through participation/ association with A/Is

5. Available data suggests thatCAIP funding and the resulting assistance provided to A/I client-firms have contributed to wealth creation in Canada

Based on data provided by A/Is, client firms, as a group, have grown significantly in revenue and equity investment. Firms that have experienced especially high revenue or equity growth believe the A/I assistance was important in enabling them to achieve this growth. While the study suggests modest benefits overall, available data suggests thatCAIP contributed to positive net economic benefits for the Canadian economy.

Supporting evidence

Performance data provided to IRAP by funded A/Is shows that, in total, the set of clients that were providedCAIP-supported services has grown in revenue, equity, and number of staff. Exhibit 11 identifies the growth in revenue and equity over the 2014-2017 time period.^{Footnote 5}

Exhibit 11
In total, available data suggests the revenue and equity investment of the total A/I portfolio has grown^{Footnote 6}

Long description of Exhibit 11: In total, available data suggests the revenue and equity investment of the total A/I portfolio has grown.

This is an infographic that shows the growth in revenue and equity investment (over the 2014-2017 time period) for all clients who were provided withCAIP-supported services. The exhibit presents all clients; it also suggests that there was significant growth in the revenue and equity of many individual client firms.

	2014	2017	Growth
Revenue	$159M	$2B	2000%
Equity Investment	$77M	$1.2B	3600%

While the chart presents the A/I's total portfolio, this finding does suggest that there was significant growth in the revenue and equity of many individual client firms. The analysis below used the program data collected by IRAP from the A/Is to examine the individual growth of client-firms in more detail. This analysis demonstrates the number of growth periods in revenue or equity experienced by client-firms. A "growth period" is defined to be increased revenue or equity reported between two, not necessarily consecutive, years. Therefore, the total sample of firms considered for this analysis was limited to firms for which revenue (n=526) or equity (n=394) was provided for at least two years. Overall, the majority of firms on which data was available grew over at least one year, with many others growing over more than one year:

• 76% of firms for which 2 or more years of revenue was provided achieved at least one growth period. (n=562)
• 61% of firms for which 3 or more years of revenue was provided achieved at least two growth periods (n=171)
• 42% of firms for which 4 years of revenue was provided achived three growth periods (n=83)
• 64% of firms for which 2 or more years of equity was provided achieved at least one growth period (n=394)
• 36% of firms for which 3 or more years of equity was provided achieved at least two growth periods (n=101)
• 10% of firms for which four years of equity was provided achieved three growth periods (n=39)

Making strong statements about attribution for programs likeCAIP is challenging, however, insight was obtained through the survey of client-firms. Survey respondents were asked about the importance of A/I assistance with respect to their anticipated growth. Just over half (52%) of the firms said that the A/I had a moderate or major influence on their estimated future growth.

Insight about the attribution ofCAIP on growth was also acquired from client-firms of the A/Is. For the partial benefit-cost analysis, A/Is were asked to nominate only high-growth clients for which growth was highly attributable to A/I assistance and were significantly assisted byCAIP-funded programming. Some of these claims about attribution by A/Is were validated with a sample of clients: during interviews with 46 firms that experienced high revenue or investment growth, they were asked about the importance of A/I services in helping them generate growth. On average, these "high impact firms" rated A/I assistance as very important.

The benefit ofCAIP was also assessed through a benefit-cost analysis of the program. This study was based on the revenue and investment increases of "high impact firms (HIFs)", firms that receivedCAIP-supported services from the A/Is and have experienced especially high increases in revenue and/or investment.

Essentially this analysis involves (1) adjusting these revenue and investment increases downward to reflect the degree to which they are attributable toCAIP; (2) estimating the net profits resulting from the revenue increases; (3) discounting each firm's stream of increased profits and investments to present value terms (since these benefits occur in different years); (4) adding up each firm's benefits across all HIFs; and (6) comparing this sum to the cost ofCAIP. This methodology is described in detail in Appendix B.

The results are shown below for two different scenarios, plus lower and upper bound assumptions (related to optimistic and pessimistic estimated profit margins), and with three different discount rates.

Exhibit 12
Available data suggests thatCAIP has resulted in positive net benefits to the Canadian economy

There are two important points of interpretation:

• The positive values for the net benefits ofCAIP and the benefit/cost values > 1 mean that overallCAIP has "made money" – i.e., its benefits are greater than its costs.
• The tables show various ways to interpret the economic performance ofCAIP. The values that are best aligned with most other PBCA studies are those for Option 2, lower bound. These values assume investments are costs that must be expended to bring these innovations to fruition. From a system perspective, these costs must be recouped before any given innovation makes a profit, and thus the value of investments is subtracted from any profits to date. (In Option 2, upper bound, investments are treated as benefits, because they are considered important benefits in the short-term for start-ups and SMEs, without which many of them would fail.)

6. The delivery ofCAIP was challenging and resulted in some valuable lessons learned

The cost to deliverCAIP is more than double the amount planned at the onset of the program. This is in part due to the complexity of the program which led to unforeseen challenges. Overcoming these challenges has required a considerable level of effort. As a result, there are many learnings that should be considered for any future similar program.

Supporting evidence

The delivery ofCAIP was challenging. This is in part because at the onset of the program, its complexity was not well understood. Challenges may also be the result of insufficient time allotted for planning and implementation. IRAP was selected to deliver the program based on their experience with contribution agreements with organizations including A/Is. However, IRAP's regular operations are much more straightforward and IRAP would have benefited from more time to understand the nuances ofCAIP as well as the processes and level of effort required for delivery. As was found in the first evaluation ofCAIP^{Footnote 7}, it took some time for IRAP to adapt their program infrastructure to the new parameters required byCAIP. The contribution agreements (CAs) typically used for IRAP contributions are relatively straightforward and not particularly difficult to negotiate. In the case ofCAIP, the materiality of the contribution amounts, along with the requirement to stipulate all the possible expense items that could be reimbursed, contributed to a much longer process of due-diligence and negotiation. This led to significant amounts of time and effort spent to establish the contribution agreements (an average of 82 days).

Further, program delivery processes and guidelines were developed late in the planning process. Individuals involved in the planning and delivery ofCAIP mentioned that guidelines on program management and claim processing were developed concurrently with the CA negotiations. Had processes and mechanisms been established sooner, but after fully understanding the requirements and complexity of the program, the required level of effort to actually deliver the program may have been known.

IRAP also had to learn to manage other program complexities such as managing multi-party contracts, as some of the A/Is partnered with other organizations. Reviewing claims was also complex and time-consuming, as they were accompanied by status reports specifying activities and deliverables eligible for reimbursement for each period (either monthly or quarterly). These needed to be reviewed for consistency with the CA.

These challenges have now largely been resolved and the program is functioning more smoothly. However, there are some challenges that continue to affect delivery including:

• The restriction ofCAIP support to incremental activities, since it takes time to verify in the claims process that activities were incremental
• The vague guidelines regarding in-kind contributions, which leads to definitional problems and administrative difficulties

The complexity and challenges of CAIP led to a program that was not adequately resourced. Program delivery costs have been higher than anticipated at the outset of the program (Exhibit 13).

Exhibit 13
CAIP's actual delivery cost are over double the planned cost

Considering the amount of allocated funds and the committed funds for the last fiscal year of the program, the delivery costs represent 5% (when corporate overhead is included in delivery costs; 3.3% otherwise).

With or without the challenges that led to increased delivery costs, it is difficult to assess the level of effort appropriate for a program such as CAIP. There are few comparable programs. While there is nothing similar in both size and structure, investigation into a few other federal financial assistance programs show similar delivery costs. Examples are certain programs of the Tri Council Granting Agencies. These programs involve the provision of funding to recipients (generally the network managers), who then provide funding and services to other program beneficiaries. If only in this way, they are similar to CAIP. Administrative costs for these programs range from 4% to 5%, not including the volunteer time of peer reviewers and committee members. Administrative costs for mainstream NSERC programs, which are not as comparable to CAIP, are higher, in the range of 4.5% to 6%, and these do not include all corporate overhead costs, such as common administrative services.

Another important challenge that has affected CAIP is the reporting requirements for performance information from funded A/Is. This was reported as problematic in the first evaluation, and continues to be problematic. Since the first evaluation, IRAP succeeded in improving and simplifying the information required from funded A/Is, however as indicated in the study limitations, the quality of the data was not always sufficient. Based on interviews conducted, such information is important for a number of reasons:

• CAIP is a contribution program and not a granting program, and therefore subject to the performance conditions specified in the contribution agreement.
• Information provides accountability and transparency.
• Information allows for assessments of the value and benefit of the government's investment.

As such, it is important that funding recipients abide by reporting requirements, providing the best quality information possible, and that the requirements are clear, detailed, and well understood prior to signing the contribution agreement.

Some concerns expressed by A/Is about providing data relate to the administrative burden, not only on themselves, but on their client-firms. This challenge is increased in the case where several support programs assist an A/I and also seek data at different times. To reduce this burden, data collection could be a concerted effort by federal and provincial capacity support programs. Based on discussions with external stakeholders, a performance data framework could be developed so that the same type of information is being collected about beneficiaries of all similar programs, and more streamlined approaches to collecting the data could be developed.

7. Conclusion

CAIP was a complex program to implement and deliver. Important lessons such as taking sufficient time to plan and develop administrative processes and better ensuring that resources available align with the level of effort required for delivery should be considered for future programs. Consistent reporting and collection of performance information about the program is not a challenge unique to CAIP, however this program provides an example of the challenges in demonstrating the value and benefit of government investments when adequate information is not available. With this caveat in mind, findings derived from information that was available, as well as information collected through the evaluation, do suggest some positive outcomes for CAIP. These can be summarized as the following:

• CAIP enabled A/Is to grow and become stronger
• CAIP enabled A/Is to reach a much larger client base and provide more services than otherwise would have been possible
• CAIP services were beneficial to client firms and facilitated revenue and investment increases for many of them
• Based on these revenue and investment increases, the net benefits of CAIP for the Canadian economy were positive

8. Management response

Appendix A - Methodology and limitations

This section presents a detailed description of the evaluation methodology, including the evaluation design, data collection methods as well as challenges and limitations of the evaluation.

Evaluation Design

CAIP required two evaluations. In fall 2016, a mid-term evaluation was completed. The evaluation took place early in the program life and, as a result, the study focused on the relevance and implementation of the program. The current evaluation considers the full life of the program, assessing the extent that program outcomes have been achieved. The evaluation also explores questions about the need for the program and the efficiency and economy of its delivery.

The questions explored through the evaluation are listed in table 1. These questions were developed following a review of program documents and literature on best practices in evaluating accelerator and incubator programs as well as consultations with stakeholders and with IRAP senior management. The approach also considered the uniqueness of each CAIP recipient and the breadth of the services they provide.

Table 1 – Evaluation Questions

This evaluation was conducted by KPMG LLP, an external evaluation consultant, with oversight by NRC's Office of Audit and Evaluation.

Data Collection Methods

The evaluation used a mixed methods approach, incorporating both qualitative and quantitative data collected using several lines of evidence. This includes the following:

• Document and data review
• On-line survey of client-firms
• Interviews with delivery staff and external stakeholders
• Interviews with CAIP recipients
• Interviews with high growth client-firms
• Benefit-cost analysis

These are described below in more detail.

Document and data review

A review of documents such as contribution agreements, amendment and claims was conducted to provide an understanding of the activities funded, their expected outcomes, as well as what has been achieved and any variation (in activity or outcome) that occurred over the duration of the program. This activity was designed to be the main line of evidence to inform the performance of each CAIP recipient as well as their contribution to CAIP's overall success. Of particular importance to assess performance was data from CAIP recipients on services rendered as well as client-firm revenues and investments. This data was collected in a standardized excel work book issued by IRAP, with a separate tab allocated to each activity area as specified in the Contribution Agreement. For the purpose of analysis, annual excel reports were combined into one master data file. Overall, data was inconsistent and required cleaning, which was done in the following ways:

• Duplicate firms were removed from the data set using pivot table analysis.
• Recoding of data elements was completed using lookup tables to increase consistency. This included:
  • Substituting a mean value for a range of values provided in the FTE, revenue, and equity fields (e.g., $100,000 - $500,000, became $250,000)
  • Standardizing the coding of firm status (e.g., replacing all French terms with English)
  • Where a firm was reported on by more than one A/I, averages were calculated for all quantitative fields and a new record was created

Lastly, administrative data (financial and human resources) was also reviewed in order to inform questions about efficiency and economy.

Online survey of client-firms

The objective of this line of evidence was to provide information on the impact that activities and services offered by CAIP recipients have had on their client-firms.

The survey was meant for all clients having received one or more services from CAIP-funded A/Is. The list of firms was developed using the 2016 and 2017 annual reports submitted to IRAP by the A/Is.

Due to preferences of some A/Is, the survey was distributed in two ways: For most of the A/Is (11 of the 15), the A/Is themselves distributed the survey to their clients; KPMG distributed the survey to the clients of the remaining four A/Is. A total of 549 responses were received representing an estimated response rate of 16%^{Footnote 8}.

Interviews

Interviews were conducted with key external stakeholders (n=5) and with staff from IRAP (n=5). These provided context as to the economic and political climate at the time CAIP was developed, speaking to the need and intent of the program. Speaking with IRAP staff also provided important insight into the delivery of the program and the challenges that emerged.

Interviews with the 15^{Footnote 9} CAIP recipients was important in order to validate information provided in their annual reports and to gain a better understanding of perceived outcomes of their CAIP-funded activities. In particular, respondents were asked about their programming, progress against objectives, CAIP assistance, client outcomes, and delivery experience.

Consultations with a sample of client-firms (n=46) who received services from CAIP recipients was also important to augment information provided by the A/Is and from the survey about the impacts of these services. This activity was also leveraged for the benefit-cost analysis, therefore "high impact firms" were interviewed (see appendix b). The sample of 46 A/I clients that have experienced especially high revenue and/or investment growth, were also asked to validate information about them provided by the AI, for use in the benefit-cost analysis.

Benefit-cost analysis

The extent to which the benefits of CAIP outweighs its cost is an important question for program management. This analysis was conducted using a modified method of benefit-cost analysis called "partial benefit-cost analysis" (PBCA). In PBCA the analysis focuses on calculating the economic benefits accruing to firms that experienced very large benefits (e.g., large increases in revenues over the period of A/I assistance). The PBCA methodology is described in more detail in Appendix B.

Study limitations

Some limitations should be kept in mind when interpreting the results of this evaluation.

1. The A/I performance report data is self-reported by A/Is and required extensive cleaning, as discussed above. The same was true, to some extent, of the data provided by the A/Is for the benefit-cost analysis. This required assumptions and approximations that may not have always been correct.
2. As noted above, the survey of the clients of eleven of the A/Is was administered by the A/Is themselves, which resulted in some delays and short response deadlines. This affected the survey response rate.
3. The client-firm survey was limited to those firms currently being supported by A/Is. It was not possible to survey firms no longer supported by A/Is, since the client information provided by the A/Is only included current client data at the time of report preparation. Some of the firms who "left" the A/Is may have had different opinions regarding A/I services.
4. Much of the client-firm information required for the benefit-cost analysis was collected from the A/Is potentially limiting the accuracy and depth of the data collected. This includes information on client-firm revenues and investments, as well as the attribution of revenue and investment. Collecting this information from the clients themselves would have been preferable, however the time and capacity required to interview the 191 nominated high growth clients were not within the scope of the evaluation. This limitation was mitigated somewhat through the validation of this information with 46 of the firms, but this is less validation than is normally the case in benefit-cost analyses.
5. There was reluctance by some A/Is to support some of the data collection activities, mainly because they perceived it as a burden to their clients. This required negotiations and resulted in tighter timelines. However, although there were some delays, all A/Is participated in all parts of the study.

Appendix B – Benefit-cost analysis methodology

The net economic benefits of CAIP were calculated using a modified method of benefit-cost analysis called "partial benefit-cost analysis" (PBCA). Ideally, in a program consisting of individual projects, the net benefits of the program are estimated by first calculating the net benefits of each project and then adding these up (after appropriate adjustments for inflation and discounting). In partial benefit-cost analysis, rather than first calculating the net benefits of every project, calculations are done for a subset of the projects; those which have obtained very large economic benefits as a result of program assistance (e.g., very large increases in revenue) and then the economic benefits are added to obtain an estimate of the benefits of the program. This is a lower bound estimate of the net benefits of the program, but it's a high lower bound (i.e., close to the actual net benefits of the program) if the "high-impact projects" used in the calculation are carefully selected.

The high-impact projects in this case were clients of CAIP-funded A/Is that had achieved great economic benefits (called "high-impact firms" (HIFs), and were identified by the A/Is in accordance with the following criteria:

• Each HIF had to have obtained either (a) increased gross sales revenues of at least $2 million per year and/or (b) increased investments of at least $2 million (cumulative) since the firm became involved with the A/I;
• These revenues and/or investments were strongly tied to A/I assistance (each A/I rated the importance of their assistance in contributing to the revenue or investment increases of each of their HIFs on a 1 to 5 scale); and
• The A/I assistance was strongly tied to CAIP funding.

A total of 191 HIFs were identified in this way. The A/Is were also asked to provide estimates, if known, of each HIF's average percentage growth in gross revenue for the next three years.

The net benefits of CAIP were then calculated as follows. For the sake of simplicity, this discussion deals with benefits based on revenue increases, as opposed to investments, although investments were also included in the analysis.

• (1)Fifty-one of the 191 HIFs submitted by the A/Is had an A/I importance rating of 5 (i.e., A/I assistance = very important). The analysis was based on these 51 HIFs. The net revenue increases of each of these HIFs were calculated for each year since they began receiving A/I assistance, where net revenue increases = gross sales revenue increases x profit margin, with profit margins set equal to those developed in previous PBCA studies.
• (2) These net revenue increases were added up for all the HIFs and appropriately discounted and deflated; call this sum S.
• (3) The net benefits from HIF projects were then calculated as S x 0.985 x 0.30.
  The figures 0.985 and 0.30 are attribution adjustments that were determined as follows:
  • The 0.985 is an "A/I attribution adjustment" – an adjustment related to the extent to which these benefits are attributable to services provided by the A/I. As noted in (1) we only included the HIFs in our calculations for which the A/Is rated the importance of their services as 5. In order to check the validity of the ratings provided by the A/Is, we asked a sample of client-firms how important they themselves would rate the services of the A/Is in contributing to their increased revenues. The ratings of the interviewed client-firms were equal to 98.5% on average of the ratings of the A/Is.
  • The 0.30 is a "CAIP attribution adjustment", an adjustment related to the extent the benefits are attributable to CAIP funding. As discussed above, the A/Is were asked to only select HIFs for which the impactful A/I services were "significantly funded by CAIP." However, because of the way funding is mixed up within the A/Is (i.e., programs and services are often based on a mix of funding sources), this proved difficult for many of the A/Is. Therefore it was decided to use 30% as a conservative estimate of the percentage of A/I funding for the impactful services that came from CAIP. 30% is the overall percentage of A/I program and service funding that has come from CAIP.
• (4) The net benefits of CAIP = the net benefits from HIF projects minus CAIP program costs. The benefit-cost ratio = the net benefits from HIF projects divided by CAIP program costs.