Evaluation of the Canada accelerator and incubator program (CAIP)

1.1 Evaluation Overview

This evaluation focussed on the first two years of operation of the program: 2014-2015 and 2015-2016. Because the evaluation took place early in the program life, the study focussed on the relevance of the type of intervention and the implementation of the program. Impacts were not the focus of this evaluation as an evaluation to explore early outcomes is planned for 2017-18.

The evaluation was carried out to fulfill the conditions of the Program's approval by Treasury Board, which included the conduct of a mid-term evaluation and of an impact evaluation of the Program. The evaluation was also expected to provide information to management at the Industrial Research Assistance Program (IRAP) of the National Research Council (NRC) on opportunities to strengthen program delivery. The findings from both evaluations (mid-term and impact) are expected to provide information that may contribute to the renewal of the Program's Terms and Conditions and will gather information to inform broader policy making and future program design.

1.2 Evaluation Design and Methodology

The evaluation questions were established by the Office of Audit and Evaluation of the NRC with the collaboration of CAIP management. The evaluation design meets management needs as well as the requirements of the 2009 Treasury Board Policy on Evaluation. Evaluation questions are reproduced in Appendix B along with a crosswalk of evaluation issues to sections of the report.

To enhance the reliability and validity of the information and data collected, the methodology for this evaluation includes multiple lines of evidence and complementary research methods. The specific methods used in the study include:

• A review of available program data
• A review of program documentation
• A review of literature about the performance of incubators and accelerators
• A review of program management processes including six interviews in other departments and agencies
• Forty-five interviews with program managers and staff, as well as program recipients and other stakeholders

More information on the methodology is presented in Appendix B.

1.3 Accelerator/Incubator Distinction

At the outset it is important to note that, while the program terms and conditions ( CAIP , 2013c) distinguish accelerators^{Footnote 1} and incubators^{Footnote 2}, CAIP recipients themselves and most NRC-IRAP staff have not classified recipients as one or the other. In effect, based on their activity profile, all but one CAIP recipient act as incubators rather than accelerators. In this report, the distinction is made only when germane.

Objectives

The objective of CAIP is to establish a critical mass of outstanding business incubators and accelerators that can develop innovative, high-growth SME s, which themselves represent superior early-stage investment opportunities through innovation assistance services in Canada ( CAIP , 2013a). The program logic is described in Appendix A.

Selection process

CAIP recipients were selected through a one-time request for proposals ( RFP ) that was launched on September 23, 2013 with a proposal submission deadline of October 30, 2013. Some 100 proposals were received. All proposals were first assessed by NRC-IRAP on eligibility and selection criteria ( CAIP , 2013a). The proposals that met the program criteria were then presented for evaluation and recommendations to the independent Canadian Venture Capital Expert Panel ( VCEP ), a five-member panel with experience in the venture capital asset class, business, and finance sectors who had been selected by Finance Canada. The final selection of recipients by the VCEP , was announced in late June 2014 (Thomas, 2014)^{Footnote 5}.

CAIP recipients

Eligible recipients were to fall under one of two categories: outstanding^{Footnote 6} or high potential^{Footnote 7} ( CAIP , 2013c). In the end, only "outstanding" recipients were selected. According to the RFP , recipients were to be market-driven, led by the private sector, and contributing to a sustainable venture capital system. Sixteen A/I s (individual entities or partnerships) were selected by the VCEP to receive funding under CAIP .

Contribution Agreements

The contribution agreements are contractual documents between NRC-IRAP and CAIP recipients that express the key terms, conditions and obligations under which NRC-IRAP will make contribution payments to the recipient. Examples of obligations include the requirement to report key performance data annually.

The process for developing the 16 contributions agreements began with NRC-IRAP completing a due diligence process for the selected organizations. Contribution agreements were then negotiated between CAIP recipients and NRC-IRAP regional offices based on the original proposal of each applicant, the framework of the Financial Administration Act ( FAA ) and the Directive on Transfer Payments. All CAIP recipients are required to demonstrate matching contributions on at least a 1:1 basis during the period of the contribution funding.

All but one of the contribution agreements have starting dates between June 1, 2014 and September 1, 2014. The remaining contribution agreement was finalized early in 2016. All agreements were signed after the effective starting date – between 14 and 218 days – but no payments were issued before the signature of the contribution agreement by both parties.

Eligible Costs

Eligible costs refer to the expenses incurred by recipients that are eligible for re-imbursement under each A/I 's contribution agreement. The negotiated definition of 'eligible costs' and the resulting claims processing protocol employed to validate expense claims arose as a central feature of this evaluation. Recipients bemoaned the complexity and administrative burden of the process and program representatives expressed that these processes were essential for appropriate oversight.

CAIP 's non repayable contributions were designed to support incremental activities such as higher output of firms from accelerators and incubators; increased range, availability and quality of services; or stronger entrepreneur networks.

Eligible costs in support of these activities are specified in individual contribution agreements. Eligible costs include ( CAIP , 2013a):

• salary costs
• overhead costs
• professional fees/rates
• contractor fees
• travel and living expenses
• operating and maintenance expenses

Non-eligible costs were capital expenditures including the purchase of land, leasehold interest in land, or the payment of property taxes.

According to CAIP s Terms and Conditions, contribution payments are made based on claims received for (a) achievement of pre-determined performance expectations or milestones – a description of the performance expectations or milestones is to be provided; or (b) reimbursement of eligible expenditures. Payments have been made by NRC-IRAP mainly on the basis of the latter, although where deemed necessary, advance payments have been made in accordance with the recipient's cash flow requirements.

Spending and Commitments as of April 2016

Appendix C contains the details of the amounts spent and committed to date by the program for each of the 16 contribution agreements.

While program documentation indicates that $1.5M was earmarked for program management over five years, according to key informant interviews, this amount was used to support regional and headquarter activities of the NRC-IRAP staff mobilized for CAIP administration in the early phases of CAIP . The evaluation team had insufficient details to probe this further.

3.1 Need for Start-up Support by Accelerators and Incubators

All interviewees agreed that government funding is important in supporting the development of small and medium enterprise through services that incubate entrepreneurial ideas to a level where they can crystallize into a potential business and that accelerate these emergent firms to a level of development that can put them on a track to commercial viability^{Footnote 8}. This is supported by various government statements; for example, the 2013 Budget (Finance Canada, 2013, 204) emphasized the need "to support high-potential young businesses, to ensure that they are able to grow into globally competitive firms that drive job-creation, innovation and economic growth."

Interviewees noted that consequently, there has been growing investment by different levels of government on A/I s as a tool for innovation and economic development. They also indicated that governments are more willing than the private sector to seek out and encourage large numbers of high risk, early-stage ventures, weed out unpromising prospects, and work with those with promise to get their business basics in place. Although key informants in this evaluation endorsed A/I s for support of innovation, they also acknowledged that only a portion of incubated and accelerated firms will go on to be successful.

The literature review found that much of what is written on A/I s focusses on the few most highly successful companies. Interviewees also cited instances where incubation of Canadian firms had led to successful outcomes, including access to next levels of capital.

To assess the effectiveness of A/I s in increasing likelihood of access to capital, the literature review also examined the few controlled studies that compared the outcomes of non-incubated businesses to comparable incubated counterparts and made careful efforts to track incubator participants over time. The literature review findings on the effects of A/I s on success of SME s suggest that (a) the highly successful cases are outliers in a very mixed pattern where failure is more common than success; and (b) effects are enhanced in A/I s using best practices.

3.1.1 Evidence on Incubated/Accelerated SME Success

Effects of A/I s on access to capital: Studies show that A/I s are not very effective in helping incubated firms get access to capital (Al-Mubaraki and Busler, 2010; M'Chirgui, 2012). The relationship between access to capital, financial performance and firm survival appears to be complex, especially since some studies include acquisitions as survivorships and some do not. In Canada, availability of follow-up funding differs significantly across sectors, regions and stages of growth (DEEP, 2015a) and risk capital for early stage ventures in Canada is the scarcest (DEEP, 2015a). The A/I structure influences outcomes accessible to SME s; venture-backed organizations almost exclusively focus on follow-on funding; others, notably publicly funded organizations, focus more broadly on the development of sustainable companies (survivorship and growth) (DEEP, 2015a).

Success indicators: First, the evaluation found that while the expectation is for CAIP to increase SME access to capital, published studies tend to focus on three other indicators of SME success: graduation rates, growth in size and revenue, and survival rates post-incubation. In the literature, "access to capital" does not appear to be considered the most useful immediate indicator of incubator success. For example, in a 2005 survey of business incubators, Statistics Canada used a range of indicators to reflect success, including: growth (jobs created; revenue generation; receipt of R&D tax credit); survival rates post-incubation (continuing, bought out or merged and closed), and revenue raised (Joseph et al., 2006, 15; Bergek and Norrman, 2008). Key informant interview data showed that CAIP stakeholders expect a wide variety of outcomes from participating SME s, but that there was consensus only on survival and growth in terms of jobs – not access to venture capital

Effects of A/I s on viability - Graduation rates: Only a small proportion of incubated firms graduate from the incubator within the expected time frame of three to five years (Amezcua, 2012). A/I s with more stringent selection criteria for participating SME s have higher graduation rates (Lewis et al., 2011; Amezcua, 2012). A/I s with more stringent graduation criteria produce SME s that are more likely to survive and grow (Phan et al, 2005).

Effects of A/I s on viability - Survival rates post incubation: Data available to measure incubatee survival are limited, use inconsistent measures, and are often biased by attrition (Dempwolf, 2014; Schwartz, 2009, 407). Reported post-graduation survival rates vary widely, from 15% to over 90% (Schwartz, 2009). Two controlled studies found that failure rates are higher in incubated businesses than comparable non-incubatees (Amezcua, 2012; Dalziel; 2012). There is a high risk period in the three years immediately after graduation (Schwartz, 2009). Most (90%) non-survivors have gone out of business (liquidation or bankruptcy), rather than being acquired (Schwartz, 2009; Brander et al., 2008)^{Footnote 9}. Likelihood of survival is higher among firms that grew in employees and sales during the incubation period (Schwartz, 2009). Some studies suggest that one of the effects of A/I s is to delay inevitable failure (Schwartz, 2009, 407) while others suggest that incubated entrepreneurs have learned to jettison probable failures more quickly and pivot towards more promising ventures (Amezcua, 2012; Culp, 1997; DEEP, 2015b; Herman and Williams, 2013).

Effects of Accelerators and Incubators on SME growth: Effects of A/I s appear to be stronger on creating employment and increasing sales within incubated firms rather than increasing their capacity to operate independently (Dettwiler, 2006; Amezcua, 2012). However, effects are small (Amezcua, 2012) or inconclusive (Dalziel, 2012) and may not offset the costs of running incubators (Amezcua, 2012). Canadian data also show that there are distinct patterns for different types of SME growth: revenue increases faster than profit and the number of employees, in that order ( BDC , 2015).

Regional effects of A/I s: Studies of effects of A/I s on regional economies have shown mixed findings. Some suggest they may have a negligible or net negative effect (Amezcua, 2012), while others find various positive impacts on the wider business environment that extend beyond the incubated/accelerated SME s (DEEP, 2015b; Dee et al., 2011; Fehder and Hochberg, 2014). The diversification of funding sources and the rise of accelerators may increase potential benefit to peripheral regional ecosystems (Quebec City Conference, 2014).

3.1.2 Accelerator and Incubator Best Practices that Enhance SME Success

A/I practices that have been empirically shown to be related to better outcomes (particularly survivorship and growth) for participating SME s are:

• Rigorous incubatee selection using a balanced set of indicators (Lewis et al., 2011; M'Chirgui, 2012; Dee et al., 2011)
• Provision of proactive services including proactive crisis intervention and structured activities requirements for incubatees as opposed to a "laissez faire" approach (Dettwiler et al., 2006; Dee et al., 2011)
• Focus on incubatees' business and project management knowledge skills, as opposed to technology or product related skills (Dee et al., 2011; Dempwolf et al., 2014; M'Chirgui, 2012; Lewis et al., 2011)
• Adaptation of services provided through the incubation process and period (Dee et al., 2011; Mian, 1997)
• Provision of access to networks, active mediation on behalf of, and showcasing incubatees with those networks, including networks of: co-located firms, skilled consultants and business services suppliers (e.g. legal), government officials, bankers and venture capitalists (Lewis et al., 2011; Dee et al., 2011; Hackett and Dilts, 2004b; M'Chirgui, 2012; Culp, 1997)
• Provision of, or support to develop, a management/governance team or advisory board experienced with entrepreneurial experience (Lewis et al., 2011; Al-Mubaraki and Busler, 2010)
• Provision of expertise and support to develop strong IP licenses and management (Lewis et al., 2011; Rothaermel and Thursby, 2005)
• Collection of client outcome data including long-term follow-up of incubatees, with indicators adapted to their stage of maturity, and use of outcome data to adjust practices (Lewis et al, 2011; Vanderstraeten et al., 2011)
• For accelerators: attention to all three customer markets: the accelerated firms, other firms that may be looking to grow by acquiring an accelerated firm, and venture capitalists and other investors (Dempwolf, 2014)

To the extent that CAIP recipients are implementing known best practices, it is plausible but not certain that they will increase likelihood of success of some of their participating SME s over and above what would have happened to those firms in the absence of CAIP support.

3.2 Need for Support to Accelerators and Incubators

3.2.1 Accelerators, Incubators, and other regional-level mechanisms

Across Canada as well as internationally, there are a wide range of supports available to support SME growth. No studies were found that specifically addressed the mechanisms of complementarity between different sets or suites of economic development programs, although some authors have noted that research is starting to examine the interplay between A/I s and the presence of other economic development efforts such as business development associations (Amezcua and Noble, 2012). A/I s that facilitate access to economic development agencies and agents have more successful client firms, in part because other components of the regional systems including local government officials can be influential in educating funding sources such as venture capitalists about the A/I s (Lewis et al., 2011). Embedding A/I s in strong, multi-support regional systems, including support from local authorities facilitates their client SME s success (M'Chirgui, 2012).

3.2.2 CAIP and Other Programs

There is no complete picture of government innovation or economic development programs, even at the federal level. It is therefore difficult to ascertain complementarity or duplication of efforts and the positioning of CAIP within this suite^{Footnote 10}. Moreover, there appear to be varying underlying notions amongst the key informants with stakes in CAIP results about the program's intention. Some described its role as supporting SME development by providing intensive services to a select few highly promising firms, while others spoke of attracting and providing earliest-stage services to as many emergent ideas as possible. Another view is that CAIP recipients should complement the regional or local array of economic development services. To some extent, views of CAIP 's degree of complementarity and overlap with existing programs depended on the interviewees' view of what they thought it should be doing.

Within the burgeoning innovation ecosystem, however, the role of CAIP is understood amongst key informants to be focused on early-stage firms, at the incubation end of development, although moving into acceleration phases over the period of support. CAIP funding was specifically to be used for incrementally expanding (i.e., in new sectors, regions, partnerships) existing acceleration and incubation services. CAIP recipients and partners interviewed indicated that this is the appropriate space for CAIP to occupy.

What sets CAIP apart from other programs supporting A/I s and is most valued amongst key informants is that CAIP funding is substantive (larger amounts relative to one-off projects/initiatives) and longer (over five years versus annually/one-offs).

Consistent with the finding in the literature, a theme emerged from key informant responses that the particular combinations of support depend very much on the particular sector and regional context of activity^{Footnote 11}, and that the particular combinations of supports were seen as needing to fit with the assets and strengths in the region.

A concern about complementarity was raised by recipients and other organizations involved in providing supports to SME s regarding the various federal programs that also support innovation and business development and most particularly federal regional economic development agencies^{Footnote 12}. Interviewees had observed little coordination between CAIP and their regional federal counterparts at the time of CAIP design and recipient selection, although this was not within NRC-IRAP 's CAIP mandate. This has added to the complexity of systems to be navigated by entrepreneurs and potentially missed opportunities for synergy. Another specific concern was also noted with potential for overlap with Global Affairs' CTA Initiative.

When asked about synergies they might have observed between CAIP and other forms of support to SME development, respondents tended to mention funding sources from other federal, provincial, and territorial regional innovation and economic development initiatives and describe how they were different from CAIP but were able to work together. CAIP recipients are skilled entrepreneurs themselves, highly effective in crafting their business endeavours through multiple sources of funding. In this context, CAIP has provided a larger and more stable building block, but is one among several.

In addition, various types of recipient-level synergies were reported. In one instance, a recipient reported that having received CAIP funding was allowing them to more easily leverage funding from other sources^{Footnote 13}. Another recipient noted that CAIP funding led to the creation of a relationship with an international partner and may result in an investment in a Canadian Science Park.

3.3 Alignment with Federal Roles and Responsibilities

According to the 2013 budget (Finance Canada, 2013, 188), the VCAP was supported by four measures:

• $60 million over five years to help outstanding and high-potential incubator and accelerator organizations expand their services to entrepreneurs, i.e., CAIP
• $100 million through the Business Development Bank of Canada to invest in firms graduating from business accelerators
• Promoting an entrepreneurial culture in Canada through new Entrepreneurship Awards
• $18 million over two years to the Canadian Youth Business Foundation to help young entrepreneurs grow their firms

According to other sources, other components of the VCAP were the establishment of four large-scale private sector-led funds in partnership with institutional and corporate investors as well as interested provinces and additional resources to continue developing a robust venture capital system and a strong entrepreneurial culture in Canada^{Footnote 14} .

However, beyond this information, it seems that the VCAP is not a structured program and has no accompanying documentation^{Footnote 15}. In particular, no documentation could be found to explain the expected logic of the action of the VCAP 's components and within this, CAIP 's contribution to it. How the relationship between CAIP and the VCAP is operationalized is thus not clear. NRC-IRAP staff interviewed did not refer to the VCAP as a source of orientation for CAIP . Interview data also suggested that attempts to develop a common outcome framework for CAIP and one other VCAP component, the Canadian Technology Accelerator ( CTA ) Initiative^{Footnote 16}, were not concluded.

The Budget 2013 description of the VCAP suggests a logic that CAIP was to support SME s to the level where they would become attractive to venture capital investors, while simultaneously increasing availability of venture capital funding. However, neither the contribution agreements nor recipient interviews identified a clear expectation that CAIP recipients are to produce SME s that are sufficiently attractive to the BDC or other sources of capital upon graduation.

4.1 Program Design

According to key informants, CAIP 's fundamental parameters were determined by Finance Canada. These original parameters include the following aspects of the program:

• CAIP is a contribution program and not a granting program
• CAIP recipients must match program funds 1 to 1 with funds from other sources
• matching of funds can include in-kind contributions
• The utilization of a request for proposal process where potential recipients define their project in isolation of NRC-IRAP whereas NRC-IRAP is used to co-constructing projects with applicants
• The selection of recipients by the VCEP , a group of experts chosen by Finance Canada; the VCEP was also responsible for establishing the initial amount of the contribution that CAIP would make to each recipient

NRC-IRAP has a long track record of delivering innovation support programs for a variety of federal and provincial partners. The program was sought out by Finance Canada as a delivery partner for the CAIP due to its experience in managing contribution agreements and its innovation support capacity which includes a network of over 200 Industrial Technology Advisors (ITAs) located across the country. Because the structural design of CAIP diverged from NRC-IRAP 's traditional program delivery structure and the extended amount of time required for negotiation of deliverables with the I/As, there were delays in program delivery, tools, processes and guidance.

In the program's design and early implementation phase NRC-IRAP experienced some difficulty in adapting their program infrastructure to the new parameters required by CAIP s pre-determined structure. NRC-IRAP traditionally supports smaller R&D focused projects for which the contribution agreements, which are the contractual bases for the non-repayable contributions, are less onerous to negotiate. In the case of the CAIP , the materiality of the contribution agreements along with the requirement to stipulate al the possible expense items could be recuperated contributed to a much longer process of due-diligence and negotiation. This lead to significant amount of time required to establish the contribution agreements^{Footnote 17}

The development of the guidance on program management and claim processing^{Footnote 18} took several months. The program developed the management guidance, guidelines, and tools in the months that followed the ministerial announcement of the recipients, concurrently with the negotiation of the agreements and the management of the first claims. The simultaneous development of guidance as the program was rolled out led to delays in the establishment of the agreements^{Footnote 19} and in the review of claims and payments to recipients^{Footnote 20}. According to key informants, the documentation required for in-kind contributions was particularly problematic.

Notwithstanding these initial challenges, CAIP is now operating more smoothly.

4.1.1 Industrial Technology Advisor Involvement

According to interviewees, NRC-IRAP was selected to deliver CAIP because of its knowledge of small and medium size businesses and of its track record at managing contribution agreements.

Program documentation establishes the ITA role as: "The Lead ITA has the leading role in managing the relationship with the recipient and in ensuring that the claim is adequately reviewed prior to the Lead ITA providing their recommendation for approval" ( CAIP , 2015b). The ITA may be assisted by a Regional Contribution Agreement Officer ( RCAO ) for quality assurance reviews of claims and formal recording of amounts ( CAIP , 2015b).

Key informants clearly indicated that the ITA s' role in CAIP has consisted largely of the development of the contribution agreements and the management of claims – sometimes limiting the ITA to purely administrative or clerical functions of analyzing claims and channeling information back and forth. ITA s themselves indicated that claim processing is a time-consuming component of their involvement in CAIP .

For recipients, the face of NRC-IRAP is their regional ITA and as a result, the benefits of regional delivery are associated with the quality of the relationships they have with their ITA s. These vary significantly. Overall, recipients indicated that support provided by their ITA was helpful in conveying both an understanding and the specificities of their context and operations to more far removed central offices. A few key informants specifically mentioned having accessed NRC-IRAP 's Concierge Program through their ITA .

While NRC-IRAP was able to leverage its connections and knowledge through ITA s in some instances, challenges were noted with integrating CAIP within the program. NRC-IRAP deals directly with SME s though smaller agreements with fewer components to activities and claims.

NRC-IRAP 's value added happens through the advice function of the ITA s rather than the processing of claims.

Setting aside the some early implementation challenges NRC-IRAP itself was highly regarded amongst key informants: it was seen as stable and with significant on-the-ground knowledge (through the ITA s) of industrial sectors across Canada. Overall, it is seen as a logical and supportive delivery mechanism.

4.2 Balancing Control with Flexibility

Two years into the management of the program, procedures have been clarified and most claims are processed in a timely manner. NRC-IRAP has also reduced its claim control requirements after the first claim, has started making advance payments where necessary, and has allowed putting aside a disputed part of a claim from the main claim to allow for the more rapid repayment of the acceptable expenses.

4.2.1 Contribution Agreements

CAIP 's funding recipients have expressed feeling constrained by the highly structured and detailed contribution agreements and that these contracts are limiting their agility with regard to their day to day activities. At the same time, program representatives argue that the detailed activities and outcomes in the CAs are essential for appropriate oversight and to ensure the achievement of the longer term project objectives.

CAIP 's overall premise is that high-performing incubators and accelerators provided with significant additional and longer-term funding would be able to use their knowledge and skills to ramp up success for client SME s. The contribution agreements that are to enable these results are all structured in the same manner: contribution objectives, project activities, project deliverables, expected project outcomes, and best practices requirements, and basis of payment. They detail this information very finely (for example, activities to take place in year 5 of the agreement). The details are specific to each recipient and are based on the recipient's response to the RFP and further negotiation with NRC-IRAP .

Considering the duration and size of the agreements, many key informants took the position that the level of detail found in the contribution agreements is counter-productive. They felt that, while facilitating financial control, the detailed description of activities and expected outcomes constrain the recipients in a five-year plan that does not allow them flexibility to adapt in a learning and changing environment. For example, issues were raised with five-year agreement targets that were reached in the very first year of the program. Processes are in place to allow for amendments to the contribution agreements. However, recipients report that these processes do not offer enough agility to meet their needs.

In the rapidly evolving innovation environment, experiential learning about which projects work and which do not, along with the need for course correction accumulates quickly. From the operational perspective of the recipient organizations, these factors are at odds with detailed five-year contracts.

The program perspective on contribution agreements diverges from that of recipients. Key informants from within the program report that contributions agreements with detailed activities and expected outputs are important tools for ensuring the necessary fiscal oversight that is expected of contribution programs. While they understand that recipients would prefer the flexibility that would come with a grant, their position is that the existing level of detail is appropriate to the fiduciary responsibilities of a contributions program, particularly considering the substantial size of the contribution agreements.

4.2.2 Claims Analysis

Evidence, gathered from key informant interviews with internal and external stakeholders shows that recipients consider the process of submitting expense claims to be extremely onerous. This is supported by an analysis of the claims process which indicates that NRC-IRAP chose to manage this contribution program using strict guidelines. For example, all claimed expenses have to be unequivocally associated with activities listed in the agreement and all amounts in the first claim were reviewed in detail^{Footnote 21}.

NRC-IRAP 's interpretation of CAIP 's risk profile has resulted in contribution agreements that are very stringent. While amendments to agreements are possible and some were indeed made, key informants indicated that amendments are complicated to set up and time consuming to negotiate. Stringent agreements have also translated into fixed and quantified performance expectations laid out over five years.

NRC-IRAP s claims management process is guided by a rigorous approach to risk management of public funds. However, as is typically the case, the emphasis on rigorous controls comes with an associated administrative burden on recipient organizations.

Overall, evaluation findings suggest that CAIP claims processing is not optimally balanced for the level of risk that each transaction represents. The current protocol for CAIP 's claims processing ties up recipient and program management energies in a burdensome administrative process which redirects effort from innovation support to administrative tasks. For example, all claims are treated the same way, unrelated to the amount in question or to the recipient track record. Meanwhile, one of the key principles put forward in the Blue Ribbon Panel report (2006) and in the ensuing Government of Canada Action Plan to Reform the Administration of G&C Programs was that the management of G&C programs should balance control over the use of public funds and burden on the recipients through a risk-based management.

Conversely, it is important to consider that the materiality of the individual contribution agreements indicate a level of financial oversight that is above that required for the traditional NRC-IRAP projects. The average project size for a regular NRC-IRAP project is approximately $125,000. Comparatively, the average CA size for CAIP is approximately $6,000,000.

It should also be noted that NRC-IRAP has made adjustments to its claiming process to decrease the administrative burden on recipients. For example, during the early implementation period NRC-IRAP chose to validate 100% of line items presented in each claim. The burden associated with this level of control was considered onerous by recipients and NRC-IRAP staff alike. Following a consultation process, NRC-IRAP streamlined their claims analysis so that validation of costs-incurred would only be required for certain cost categories. Less risky cost categories would be validated on a sample basis only. The process was also adapted so that if some line-items were holding up a re-imbursement, a partial payment could be made while the items at issue were investigated. This was considered a positive move that helped free up the cash flow of recipient organizations.

4.2.3 Options for Increasing Flexibility

An analysis of relevant Government of Canada studies and policy documents, complemented by interviews with government departments and agencies that manage contribution agreements, suggest that there are other potential models for managing contributions programs that may offer a different balance of control vs flexibility.

CAIP is modelled on NRC-IRAP 's Contributions to Organizations (CTO) program which is a cost-incurred Grants and Contributions ( G&C ) program. As a G&C program CAIP is subject to TBS requirements which are stipulated in its Terms and Conditions (T&Cs). CAIP 's T&Cs were examined to understand what, if any additional flexibility they afford NRC-IRAP . Evidence from interviews and the evaluators own analysis suggests that NRC-IRAP has chosen to interpret the legal and policy frameworks to minimize risk rather than to optimize flexibility.

The strict controls were explained to the evaluation team as driven by a management concern that allowing more freedom to recipients could translate into the need to claw back some funds if reimbursed expenses were later found to be ineligible. Other departments and agencies have managed this risk by emphasizing information, education, and capacity building among recipients, and by withholding a certain amount pending final audit results.

CAIP 's T&Cs do allow for NRC-IRAP to manage CAIP on a results basis rather than on an activity basis. While such a model may seem attractive there are a number of practical considerations that made it unattractive for NRC-IRAP . For example, predicating the claims process on milestones rather than activities requires the definition of reliable milestones or performance indicators that can be used to ascertain whether CAIP funds are being appropriately allocated. NRC-IRAP made the decision during the design phase that defining such milestones or PM indicators would be impractical and would represent a real risk for both program and recipient. This concern was to some extent proven correct as the majority of recipients failed to provide acceptable performance records at the end of year one. Using a results model, the program would be forced to claw back funds based on a lack of evidence that performance milestones had been reached. Given the amount of money in question, this would have been a difficult process and would be a significant business risk for recipient organizations.

However, the cost-incurred and paid model is not the only option available and other models have been used to good effect in other departments and agencies. Documentation and interviews suggest that these other models would allow CAIP to apply flexibility to the management of contribution agreements while abiding by the letter of the FAA and of the Policy on Transfer Payments. Examples of such flexibility include:

• Allowance-based reimbursements which balance the trade-off between expense control and flexibility in claiming with a pre-defined maximum amount
• Strategic use of audits to ensure the correct interpretation of the terms of the contribution agreements
• Long-term agreements with shorter term action plans and regular reassessments of the most effective response to current situations

4.2.4 Self-Organization

CAIP recipients are few and organizationally sophisticated – indeed they were selected as "outstanding" A/I s. They will also be involved in CAIP for five years. It is therefore not surprising that they have started creating a community, organizing meetings, discussing issues among themselves, and communicating with NRC-IRAP about their challenges. While a significant proportion of recipients' early discussion has been in defense of their interests and hence somewhat adversarial, NRC-IRAP could see this self-organizing as an opportunity to further engage with recipients. NRC-IRAP 's response to date has been polite but engagement has been limited. This may be a prudent attitude because of the responsibilities of NRC-IRAP in controlling program finances, but it may not be the most effective stance: according to some key informants, opening a frank dialogue and adopting a learning attitude could prove more constructive in the long run.

4.2.5 Sustainability

The CAIP documentation and contribution agreements do not express clear expectations regarding the sustainability of the CAIP -funded initiatives: is it expected that CAIP -supported activities will be maintained once the CAIP funds run out? Some recipients indicated that they would not be able to carry forward projects developed using CAIP funds unless more funding is found for 2019 and beyond. There are indications that, without continued funding, some recipients will start winding down their CAIP activities in 2018-2019, thereby reducing overall program results.

In order to address issues arising from the program's termination of funding following the 5-year timeframe NRC-IRAP has scheduled a lower level of funding in the final year. However, it remains unclear whether the expectation is that the recipients maintain the level of activity they were able to sustain under the CAIP in the post-project period.

5.1 On-going Performance Measurement

NRC-IRAP was late in collecting performance data from recipients for the first year of CAIP 's lifecycle. Under CAIP 's PM-S, recipients are expected to deliver on an annual basis a list of the start-up companies that they serviced along with the type of service rendered. Further, recipients are expected to track their client firms and provide to NRC-IRAP the following data:

• firm status (in operation, closed, sold)
• number of FTEs
• total revenue
• net income
• equity investment received

Tracking these indicators has been identified as a best practice in the literature (Lewis et al, 2011; Vanderstraeten et al., 2011). In addition, although the reporting requirements have been criticized by some recipients as onerous, others indicate that this information is collected by all accelerators and incubators as a standard business practice.

Some recipients have also pushed back on their reporting requirements stating objections such as the sensitivity of client data and competitive concerns. In response, NRC-IRAP has accepted that client information be anonymized as long as a unique identifier permits analysis of related information at the company level from year to year. Overall, there is little evidence that the performance data requested is overly onerous or inappropriate.

The quality of the data provided by funded accelerators and incubators is concerning. Few recipients delivered the required information and some offered information in an unusable format. For all intents and purposes, outcome-level information is not available for the first year of CAIP activity.

Despite developing a performance measurement strategy (PM-S) during the program's design phase, NRC-IRAP was delayed in providing a reporting template to recipients. This contributed to program's inability to enforce compliance with the reporting obligation detailed in the contribution agreements.

5.2 Impact Evaluation

NRC-IRAP management is confident that the outcome information will be sufficient to build a summative evaluation in two years – although there are dissenting voices indicating uncertainty. In the analysis by the evaluation team, it is unlikely that NRC-IRAP will be able to carry out an adequate summative evaluation in two years. This is because:

• Year 1 annual data are of poor quality
• The outcome data do not associate results with the nature of the intervention. Assuming the data are indeed gathered, it will not be possible to distinguish which interventions have had what impact on that type of start-up in which circumstances.
• No provision has been made for the analysis of CAIP 's contribution to results and to a comparison group (counterfactual)
• The existing logic model is of limited use and the program has not articulated a theory of change (i.e., how the various activities supported by the program are supposed to generate the intended effect)
• NRC-IRAP does not collect data that to inform the gender-based analysis (GBA+) required for the upcoming summative evaluation^{Footnote 22}. Other sources that can inform the GBA+ will need to be identified

The literature on accelerators and incubators offers several models of outcome measurement. The DEEP Centre has developed a methodology and used it to evaluate the impact of support to accelerators and incubators (DEEP Centre, 2015b).

The literature and key informant interviews suggest that an impact evaluation should deal with the following dimensions of CAIP impact:

• Micro level: information concerning the SME start-ups serviced by the CAIP recipients (e.g., profile, outputs over time, performance over time, macro-economic impact)
• Mezzo level: information concerning CAIP recipients (e.g., services offered, delivery of outputs and outcomes described in the agreements, quality of the service offered, quality of the response to end user needs, sustainability of CAIP activities)
• Macro level: the business acceleration ecosystem (e.g., production of incubation/acceleration tools or programs, lessons learned, best practices) and the regional/sectoral ecosystem (e.g., job creation, community's entrepreneurial climate, structural change to the regional or sectoral dynamic, spin-offs)

Existing data collection will not support such an analysis plan. Moreover, recipients are knowledgeable about their business and keen to be involved. This suggests that the program could team up with the evaluation group and the recipients to plan a useful, learning-based summative evaluation.