Continuous Audit of the Industrial Research Assistance Program (IRAP) – FY 2012-13

1.0 Executive Summary and Conclusion

Background

This audit report presents the findings of the National Research Council Canada's (NRC) 2012-13 Continuous Audit of the Industrial Research Assistance Program (IRAP).

Audit Objective

The objective of this continuous audit is to provide independent assurance on the adequacy of select key elements of IRAP's management control framework including governance and accountability, monitoring and oversight, information for decision making, contribution agreement management process, training capacity and competencies and values and ethics.

Raison d'être

IRAP enhances innovation and commercialization capacity by supporting:

  • Small and Medium Enterprises (SMEs) with their innovation projects; and
  • Organizations that deliver complimentary and relevant innovation enabling services through financial resources and technical expertise.

IRAP was identified as a high audit priority area in the NRC 2012-13 to 2014-15 Risk-Based Internal Audit Plan. Proper and effective oversight is integral to demonstrating transparency, accountability and benefit to Canadians in the distribution of transfer payments. IRAP is featured prominently in the Government of Canada's Economic Action Plan following a doubling of its base budget and additional funding from pilot projects and other temporary initiatives. In fiscal year (FY) 2012-13, the Program administered more than $170M in contributions to firms and organizations.

Audit Opinion and Conclusion

Overall, we found that the key elements of the IRAP control framework within the audit scope are adequate to support the management of transfer payments. Where applicable, audit recommendations consider opportunities for process improvements and increased organizational effectiveness using IRAP systems currently in place.

Table 1 below summarizes the audit conclusions by line of enquiry. Following this table is a list of related recommendations designed to improve current business processes.

Table 1: Summary of Audit Conclusions
Line of Enquiry Assessment Associated
Recommendation(s)
Governance and Accountability Strong n/a
Monitoring and Oversight Adequate 1
Information for Decision Making (Financial) Adequate 2
Contribution Agreement Management Process Adequate 3
Training Capacity and Competencies Adequate 1 and 4
Values and Ethics Adequate 4
Overall Audit Conclusion Adequate

Summary of Recommendations

  1. Recommandation 1: IRAP should consider establishing a comprehensive oversight framework that aligns all management monitoring activities and processes and ensures level of effort corresponds with management's risk tolerance. IRAP management should assume ownership of the framework which would be vetted by NRC's Chief Financial Officer and approved by the President of NRC. The framework should include quality assurance elements embedded in the business flow, an updated approach to recipient audits, and standardized checklists for processing recipient claims. The comprehensive oversight framework should consider training opportunities for operational staff in the context of the new control environment for web-based transfer payments system (SONAR). [Priority: HIGH]
  2. Recommandation 2: IRAP should finalize the systems improvement project allowing for better linkages between the SONAR system and SAP, as well as the development of dashboards for information management. [Priority: HIGH]
  3. Recommandation 3: IRAP should be more proactive in managing the possible risk of stacking and should better leverage SONAR's capabilities. This will also ensure consistency of the procedures and agreement templates with the approved 2013 Terms and Conditions for the IRAP Program. [Priority: LOW]
  4. Recommandation 4: In finalizing the implementation of its training strategy IRAP should consider including standardized components for values and ethics. [Priority: MODERATE]

Statement of Conformance

In my professional judgment as the Chief Audit Executive, sufficient and appropriate audit procedures have been conducted and evidence gathered to support the accuracy of the audit opinion and conclusion. The audit conforms with the Internal Auditing Standards for the Government of Canada, as supported by the results of the quality assurance and improvement program.


 

Alexandra Dagger, Chief Audit Executive

NRC Audit Team Members Footnote 1:
Irina Nikolova, FCCA, CIA, CISA
Scott Wark, CPA, CA, CIA
Andy Lang, CIA

 

2.0 Introduction

This audit report presents the findings of the 2012-13 Continuous Audit of the Industrial Research Assistance Program (IRAP). The decision to conduct this audit was approved by the President following the recommendation of the Senior Executive Committee and thereafter by the Departmental Audit Committee on June 26, 2012 as part of the NRC 2012-13 to 2014-15 Risk-Based Audit Plan (RBAP).

2.1. Background

The Industrial Research Assistance Program (IRAP) program is required to comply with Central Agency and NRC policy and legislative requirements including: the Financial Administration Act, the Federal Accountability Act, Treasury Board Secretariat's (TBS) Policy on Transfer Payments and its associated policies and directives Footnote 2; the terms and conditions of the IRAP program, the NRC Financial Management Manual and the NRC IRAP Field Manual.

The Policy on Transfer Payments requires that transfer payment programs are managed with integrity, transparency and accountability in a manner that is sensitive to risks; are citizen and recipient-focused; and are designed and delivered to address government priorities in achieving results for Canadians. The Policy on Transfer Payments assigns responsibility to Deputy Heads for ensuring that: cost-effective oversight, internal control, performance measurement and reporting systems are in place to support the management of transfer payments; and administrative requirements on applicants and recipients, including those related to monitoring, reporting and auditing, are proportionate to the level of risks specific to the program, the materiality of funding, and to the risk profile of applicants and recipients.

2.2. Context

NRC-IRAP's base total budget for 2012-13, detailed in exhibit 1(a) and 1(b) below, was $250M, including operating funds.

Exhibit 1(a): FY12/13 IRAP Funding, by Type of Funding
Funding / Expenditure Type FY12/13 Budget (000s) % of Total Budget 2012-13 Actual (000s)
Contributions to Firms and Organizations $172.0 69 % $165.8
CHTD Footnote 3 $2.4 1 % $2.4
Youth $5.2 2 % $5.1
DTAPP Footnote 4 $24.9 10% $24.2
Operations $10.4 4% $6.8
Salary $35.4 10% $34.7
Total $250.3 100% $239.0

(Source: NRC-IRAP FY12/13 Budget Summary prepared by IRAP)

Exhibit 1(b): IRAP Funding for FY12/13: Budget by Region
Funding / Expenditure Type Budget FY 12/13
($000s and % of total)
2012-13
Actuals
Pacific $40.6 16.2% $40.2
West $45.6 18.2% $44.2
Ontario $73.6 29.4% $72.3
Quebec $43.4 17.3% $42.3
Atlantic $34.9 13.9% $32.0
National Office and Program Support $12.2 4.9% $8.0
Total $250.3 100% $239.0

(Source: NRC-IRAP FY12/13 Budget Summary prepared by IRAP)

Budget 2012 introduced an additional $110 million per year in program base funding on an ongoing basis, this requires a ramp up in operational capacity. The increased funding was intended to allow NRC-IRAP to support additional Canadian SMEs that create high value jobs, and to expand the services offered through the Program's Industrial Technology Advisors (ITAs).

IRAP plays an integral part in the Government of Canada's strategy to support small businesses and innovation activities. Budgets 2013 and 2014 identified new and continuing funding initiatives to be managed by IRAP including but not limited to:

  • The Digital Technology Adoption Pilot Program (DTAPP) – FY14 will be the final year of the three year pilot program worth $80M, which aims to improve the productivity of SMEs through the adoption of digital technology;
  • The Canadian HIV Technology Development (CHTD) Program – The collaborative program between the Government of Canada and the Bill & Melinda Gates Foundation provides $2.4M in funding annually over five years to support the Canadian HIV Vaccine Initiative;
  • A new pilot program spanning three years funding SMEs to access research, technology and business development services at universities, colleges and other non-profit research institutions of their choice;
  • The Canadian Accelerator & Innovation Program (CAIP) – funding over five years to support Canadian business accelerators and incubators; and
  • Concierge Services – Launched in December 2013 to support SMEs in accessing the innovation and support resources available at the federal and provincial levels.

2.3. About the Audit

Objective

The objective of this continuous audit is to provide independent assurance on the adequacy of select key elements of IRAP's management control framework including governance and accountability, monitoring and oversight, information for decision making, contribution agreement management process, training capacity and competencies and values and ethics.

Scope

The audit assessed select key management controls of the transfer payment program in place during fiscal year 2012-13 within the operating regions of Ontario, West and Atlantic and Nunavut. The audit reviewed contribution agreements with firms and organizations and took into consideration other monitoring activities of IRAP, in order to not duplicate efforts and to develop effective recommendations which address the root cause instead of symptoms.

Values and ethics related to experiments involving human and animal subjects and accounts payable verification procedures were not included in the scope of the audit.

Audit testing and examination was conducted between April and October 2013.

Approach and Methodology

The audit was conducted in accordance with generally accepted professional auditing standards and the standards and requirements set out in the Treasury Board Policy on Internal Audit. The audit criteria, presented in Appendix A, were primarily derived from the TB Policy on Transfer Payments, TBS Audit Criteria Related to the Management Accountability Framework: A Tool for Internal Auditors (2011) and as applicable, the IRAP Field Manual and NRC Financial Management Manual. The criteria were discussed with program management in advance of the audit.

Continuous auditing is an ongoing and independent assessment of control frameworks and risk management systems. Continuous auditing techniques have been successfully used by the NRC Office of Audit and Evaluation in recent years for a number of assurance engagements identified in the annual risk-based audit plan. The audit scope and procedures are developed as part of the project risk assessment to target a localized set of audit criteria related to key control elements that must work well. Audit tests and analyses are adjusted as the examination proceeds to address new risks as they emerge. Continuous auditing methodologies have been applied to audits of NRC IRAP Program since Economic Action Plan 2009 due to the continuously evolving profile of the program.

The audit addressed the audit criteria as they existed at the time of examination. The resulting audit recommendations take into consideration any known factors that could impact inherent program risks, such as increase in program funding and resulting requirement for capacity building. The audit methodologies were selected to ensure that the root cause of findings was identified and to ensure recommendations add value for NRC and are detailed in Exhibit 2.3 below.

Exhibit 2.3: Overview of Audit Methodologies

  • Review and trend analysis of the results and findings of recently completed audits, recipient audits by external auditors and reviews, monitoring, and Quality Assurance (QA) exercises by Financial Monitoring Division (FMD) and Transfer Payment Advisory Services (TPAS) of Finance Branch (FB);
  • Interviews with IRAP executives and senior managers, ITAs and Regional Contribution Agreement Officers (RCAOs) in the National Capital Region (NCR) and three regional offices (Ontario, West and Atlantic), and with selected FB personnel involved in the execution of financial authorities and/or monitoring and oversight;
  • Review of relevant IRAP documentation and records including but not limited to: the most recent IRAP strategy and business plan; meeting minutes and governance documents, IRAP contribution agreement templates and sample agreements; submissions to TBS and to the Minister; the IRAP Field Manual and its associated tools, templates and checklists; the procedures and results of IRAP reconciliation processes including those related to SONAR/SAP reconciliation, and IRAP training strategies, plans, materials and schedules;
  • Data mining of financial (SAP) and IRAP contribution agreement details (SONAR); and
  • Substantive verification procedures applied to a judgmental sample of IRAP files identified through data analytics.
 

3.0 Audit Findings

The table below presents the detailed audit findings with accompanying assessments. Suggested management priorities for addressing risks are identified as high, moderate or low.

Audit Findings Assessment
1. Governance and Accountability Strong

Criterion 1.1: The program's management control framework is supported by an adequate governance structure and related governance mechanisms.

IRAP's governance model is comprised of an active central decision making body, the Senior Leadership Team (SLT), which is supported by several working groups and sub-committees. The SLT members include the Vice-President of IRAP and the Regional Executive Directors who provide strategic oversight to the IRAP program.

The SLT receives support and regional input through the Program Delivery Advisory Committee (PDAC) and the IRAP Management Forum (IMF).

Each region has its own regional management committee (RMC), comprised of the Executive Director, ITA Directors and Manager of Operations and Finance. The RMC has direct line of communication with the SLT through the Executive Director while the IMF includes several regional directors; both committees facilitate two-way communication between the regions and the SLT.

The key working groups for IRAP; (the Finance Management Committee, sector working groups such as aerospace, construction and biomedical, Animal Care Committee, Human Ethics Committee and the IRAP Youth Coordinators) provide support for the management of key operational issues.

Program management has demonstrated commitment to continuous improvement to governance mechanisms and processes.

The IRAP governance structure and mechanisms adequately support program operational requirements within the decentralized regional delivery model. The SLT is well supported by working groups and sub-committees and has considerable and appropriate regional, management and operational representation.

Criterion 1.2: Authority, responsibility and accountability is clearly defined and well understood for all phases of the program's life cycle for both Head Office and regional locations.

IRAP maintains a Field Manual for both firms and organizations that provides an accessible reference source and guidance on the stages of the IRAP process, from the recipient assessment to the monthly claims and status reports for each IRAP project.

Program delivery relies on different reporting structures and budgeting responsibilities defined on regional requirements. Interviews with Executive Directors, Directors and Managers of Finance and Operations provide positive assurance that the roles and responsibilities of ITAs and RCAOs are well understood and defined for IRAP and across regions.

The specific tasks and responsibilities of RCAOs and ITAs will be evolving due to changes emanating from the new Sonar web application. RCAOs will have greater responsibility for monthly claim verification and the Post Payment Validation (PPV) for IRAP projects. The ITA will retain the principal authority for performance certification (Section 34) under the Financial Administration Act (FAA).

Authority, responsibility and accountability is adequately defined and well understood for all phases of the program's life cycle for both Head Office and regional locations.

The roll-out of the web-based SONAR system and future planned system improvements represent an opportunity to clarify key aspects of the RCAOs and ITAs monitoring roles.

(included in Recommendation 1)

2. Monitoring and Oversight Adequate

Criterion 2.1 (Management Oversight): Oversight processes and mechanisms provide NRC management with assurance that risk of non-compliance with TBS policy and legislation is adequately managed within senior management's risk tolerance level.

Management oversight of the IRAP program is based on regular meetings between Directors and their ITAs to assess the status of on-going IRAP projects and determine which IRAP recipients require additional monitoring. Each region prepares its own project tracking sheet based on SAP financial data to verify the committed funds, amounts paid and cash-flow burn rate for projects on a per region and ITA basis (i.e. the Regional Report). RCAOs also download project status reports from SONAR for each ITA to verify any outstanding monthly claims for their IRAP projects. Regional management teams utilize the Regional Report to assess the overall progress of IRAP projects in their region.

NRC Financial Monitoring Division (FMD) of Finance Branch is responsible for helping NRC maintain an effective and efficient system of internal controls over financial reporting. In line with this responsibility, FMD evaluates the effectiveness of key financial controls of the IRAP program. Transfer Payments and Advisory Services (TPAS) of NRC Finance Branch is the centre of expertise for financial knowledge and advice on transfer payments at NRC. TPAS activities ensure that NRC transfer payments follow TB and NRC policies. TPAS retain responsibility for managing, on IRAP's behalf, external recipient audits of IRAP contribution agreements. Recipient audits provide assurance to NRC management that contributions have been expended by recipients according to the terms and conditions outlined by IRAP in the contribution agreement template.

Issues identified in the monitoring results of FMD and TPAS are incorporated by IRAP management into the IRAP program and procedures. The monitoring results provide assurance that agreements are administered in accordance with TBS and NRC policy requirements.

IRAP management has the opportunity to respond to monitoring results, develop action plans to address the recommendations and institute the necessary changes to guidance and procedures, as noted during interviews with TPAS and IRAP Directors and Executive Directors. Key results or potential issues noted during the recipient audit process for specific files are communicated to the affected region for proactive risk management activities. Proactive risk management activities include reductions in contribution funding and changes to monitoring activities. Recipient audit results are attached in SONAR for the specific IRAP projects for future reference. Where clients submit new projects for consideration, it is at the ITA's discretion to reference any relevant findings from previous recipient audits.

The current oversight processes in place for IRAP ensure the program adequately complies with Treasury Board policies and requirements related to transfer payments

There is an opportunity to achieve further organizational effectiveness and efficiency through harmonizing the monitoring activities, both internal and external to IRAP, in a comprehensive oversight framework that would address both financial and non-financial aspects of the program.

Criterion 2.2 (Risk-Based Monitoring): The level of monitoring of recipients and the reporting required from recipients is risk-based.

IRAP has established a risk-based monitoring approach related to ensuring compliance with the financial terms and conditions of the contribution agreements regarding the eligibility of claimed expenses.

Compliance with the financial terms and conditions are monitored primarily through the monthly claim review process and the conduct of Post Payment Validations (PPV). The frequency of the conduct of PPVs depends upon the lead ITA's assessment of risk related to the firm's ability to meet their financial and records management responsibilities. As part of the risk-based approach noted in the Field Manual, current IRAP guidance requires that all agreements be subject to the following basic monitoring requirements: preparatory meeting with client, claim related meeting with client and PPV of the first claim, unless waived by the ITA. IRAP has not developed a risk-based approach for monitoring compliance with non-financial terms and conditions of the IRAP agreements (i.e. terms and conditions other than the specific financial-related ones noted above). The only IRAP guidance related to monitoring, other than the PPV process, is brief with general guidance around monitoring project progress and a note that officers should monitor compliance with "special conditions".

The level of monitoring of recipients is risk-based for the financial aspects of the agreement.

The extent and nature of monitoring of recipients related to ensuring compliance with the performance and non-financial terms and conditions is not formally defined in IRAP guidance documents and is left to the discretion of each ITA and regional office.

Recommendation 1: IRAP should consider establishing a comprehensive oversight framework that aligns all management monitoring activities and processes and ensures level of effort corresponds with management's risk tolerance. IRAP management should assume ownership of the framework which would be vetted by the Chief Financial Officer and approved by the President of NRC. The framework should include quality assurance elements embedded in the business flow, an updated approach to the recipient audits, and standardized checklists for processing recipient claims. The comprehensive oversight framework should consider training opportunities for the IRAP operational staff in the context of the new control environment for web-based SONAR. [Priority: HIGH]

3. Information for Decision Making (Financial) Adequate

Criterion 3.1: Management has access to timely and reliable financial information to support monitoring, reporting and decision making.

Information for decision making is available from two main sources, SAP and SONAR, in a variety of formats including pre-set reports and specialized reports prepared by Head Office or the region to meet specific reporting needs. SAP is NRC's financial system and manages all financial aspects of IRAP. SONAR is IRAP's contribution agreement management system.

The IRAP Head Office, with Finance Branch, prepares weekly budgets and monthly and quarterly reports for the Senior Leadership Team (SLT). The weekly budget is downloaded directly from SAP and tracks IRAP funding levels, amounts paid, operational costs and both reserved and committed funds. Weekly budget reports are prepared from SAP financial data and the monthly and quarterly budget reviews are prepared with pre-generated reports directly from Business Intelligence (BI) and Business Explorer (BEx).

The weekly and monthly reports provide a high level summary for SLT to monitor funding commitments. The reports are used by the regions only as a comparison of their funding levels and commitments relative to overall program amounts. The regions prepare their own summary reports from SAP to verify total IRAP funding commitments, payments and project burn rates on a Director and ITA basis. The Regional Report is prepared by either the Manager of Operations and Finance for the West and Atlantic Regions or the Regional Program Analysis Manager in Ontario.

The regions also use the pre-generated SONAR - ITA Project Report to verify any overdue contribution claims. The Regional Report and the ITA Project Report are regularly reviewed by the ITA and the Director in each region to determine if any IRAP projects are behind schedule and require further site visits or other monitoring. The process for preparing the regional tracking report has not changed based on the introduction of the web-based SONAR application.

Original SONAR provided standard templates for cross-program monitoring and performance reporting on attributes such as the population of amendments, post payment validations performed, economic information such as recipient industry categories and completion rates for post-project assessments. The reporting system allowed users to generate both high level summaries as well as targeted reports for individual contribution agreements to monitor performance of projects by recipient, ITA and region. The system generated reports were not readily convertible to spreadsheet format and required further revisions before the data could be analyzed. New SONAR simplifies reporting functions by consolidating numerous data fields into fewer report templates; and allowing improved filtering capabilities to facilitate targeted data extraction. New SONAR currently has nine template reports that provide primarily summary level information. No reports exist detailing all contribution agreements with amendments, post payment validations or other specific criteria.

There were gaps in the data available from both original SONAR and new SONAR due to paper based file management of certain aspects of the contribution agreement management process and missing or incomplete data recorded in the system limiting the effectiveness of data-analytics and monitoring activities.

New SONAR introduces dashboards for ITAs and Directors to track the portfolio of projects for which they are responsible. ITA dashboards include details on their active client list and number of projects. The Director dashboard provides a breakdown of projects by project phase (projects under development, projects in progress, projects finalized and projects approved). New SONAR also incorporates a change management log documenting system changes made to address user concerns.

The SONAR/SAP monthly reconciliation report allows for a comparison of financial information and commitments between SAP and SONAR. The reconciliation is prepared by merging SAP and SONAR datasets and applying database queries to search for discrepancies and/or duplicate information. Any discrepancies are communicated to the regions and Finance Branch for resolution and are usually due to journal vouchers or other adjustments. The reconciliation report is used as the basis for other ad-hoc reports requested by Treasury Board, the Minister, Parliament or the SLT.

The introduction of web-based SONAR requires a revision of the reconciliation process. Reconciliation activities are expected to be streamlined and the quality of information improved once the process is redefined.

IRAP management has access to timely and reliable financial information to support monitoring, reporting and decision making.

The SLT and regional management teams have the necessary access and information to manage the program through SAP and SONAR. The regions utilize SAP financial information to create Regional Reports in an accessible format that meets their specific requirements and can be easily manipulated for different reporting needs.

The reporting functionality has improved with the introduction of web-based SONAR due to fewer gaps in available data. There is an opportunity to further leverage data available in the contribution management system through dashboard reporting and linkages to the financial data in SAP for effective and timely decision making.

Recommendation 2: IRAP should finalize the systems improvement project allowing for better linkages between the SONAR system and SAP, as well as the development of dashboards for information management. [Priority: HIGH]

4. Contribution Agreement Management Process Adequate

Criterion 4.1: Employees are provided with the requisite guidance, tools, resources and information to ensure compliance with NRC and TB requirements for contribution agreement management. Employees adhere to the requirements and guidance provided for the program. Note: Audit scope is limited to Stacking, Advances, Assignment and Novations, Amendments and Post Payment Validations (PPVs).

IRAP leverages the capabilities of NRC's SONAR contribution management system to manage, maintain and administer contribution agreements. The Field Manuals for both firms and organizations serve as the primary sources of reference for IRAP staff in the execution of their duties and outline the minimum requirements to maintain consistency in the IRAP program. Web SONAR is a new interface added in May 2013 to modernize the contribution management process. The system is currently geared towards management of contributions to firms.

IRAP management and employees are provided the necessary guidance, tools and resources, such as web-based SONAR, the IRAP Field Manual and firm and organization templates, to effectively manage the key elements of the contribution agreement management process and ensure compliance with TB and NRC policies and requirements.

4.1 (a) Stacking

Stacking is a key factor in the risk management of transfer payment programs and quantifies the total amount of funding contributed by the government compared to project costs incurred by a recipient. The TBS Directive on Transfer Payments requires departmental managers to establish a stacking limit in the terms and condition of a transfer payment program, i.e. setting a maximum permitted level of total Canadian government funding for a project (direct and indirect Footnote 5 ). The IRAP Terms and Conditions define the stacking limit as a percentage of total eligible expenditures in line with Treasury Board requirements (i.e. 75% for firms and 100% for organizations).

The IRAP Field Manual provides guidance and resources to ITAs and RCAOs on the requirements, definition and calculation of the stacking limit. IRAP ITAs discuss stacking during on-site visits with recipients and verify limits at the project assessment stage. Externally hired auditors verify, as part of recipient audits, that total government funding, including indirect funding related to tax incentives, does not exceed the maximum permitted under the terms and conditions of the program. IRAP manages the risk of stacking, i.e. exceeding the maximum permitted level of total government funding, by "capping" contribution amounts at 80% for salaries and 50% for contractor fees for firms. Overhead is not funded by IRAP for firms and may be funded on a case-by case basis for organizations.

Total government assistance as part of stacking monitoring

IRAP recipients self-report any additional direct government assistance received during the project proposal phase, with each status report submitted and when completing the Post Project Assessment (PPA). IRAP recipients are not required to provide documented support of the other government funding received such as annual financial or project statements or copies of SR&ED claims submitted to the CRA. IRAP contribution agreement templates clearly define the calculation of total government assistance. However, the new SONAR reporting functionality does not include total government assistance as a reporting tool. The FY2010-11 and FY2011-12 recipient audits identified 34 of 50 projects as having received other government funding. Our review found that of the 34 files identified, 19 had other government funding self-reported in the project proposal, status report or documented in SONAR; the rest either made no mention of stacking or stated that no other government funding was received.

Differing basis for calculation of stacking limits

Total Project Costs (TPC) are all costs that a recipient expects to incur within the timeframe of an IRAP project and may include salary costs, contractor fees, materials and supplies, travel costs and equipment. TPC is an estimate provided by the recipient at the project assessment stage and is recorded in SONAR as part of the initial stacking limit verification by the ITA. TPC is used to verify the amount of leverage generated by IRAP's contribution to a project. IRAP recipients are required to report on IRAP Project Costs (IPC) incurred as part of the monthly claim process. IPC is comprised of eligible salary and contractor costs and overhead which is limited to a maximum of 65% of salary costs. IPC is considered when determining a level of funding (i.e. actual contribution amount) for a project. Reporting requirements as part of monthly claims process do not include an update on TPC. We found that SONAR does not always include up-to-date data on TPC and that the external auditor tests of stacking limits are based on IPC as the baseline. The basis for calculating stacking limits is not clearly defined at the contribution agreement level as noted by the external audits of recipients.

Low risk of non-compliance

Initial data mining identified nine potential stacking errors for CAs under $100K (100% Stacking Limit) and 16 potential stacking errors for contributions over $100K (75% Stacking Limit). However, it should be noted that from the audit population subject to analysis 27% (578 of 2,140 files) did not have any information for the TPC and 7% (159 of 2,140 files) did not have information on IPC. The same rates of inconsistent data were applicable to the full population of IRAP firm and organization projects that received funding in FY 2012-13 (3,233). Substantive testing procedures were applied for 18 files to verify compliance with the stacking requirements based on the IPC and the IRAP funding received. Overall risk of non-compliance is deemed low as only one case where stacking limit could potentially be exceeded was identified. Our trend analysis for 2011-12 files noted three potential cases (out of 25 files tested) of stacking. However, a final conclusion as to non-compliance of all these cases could only be made upon availability of up-to-date TPC data. The recipient audits for contributions in FY10-11 and FY11-12 noted four potential stacking issues (out of 50 files tested). Two of the projects were still ongoing at the time of the assessment.

IRAP has the necessary tools, resources and guidance to manage stacking limits as per the Treasury Board Directive on Transfer Payments and the policies specified in the Field Manual.  Risk of non-compliance is adequately managed. However, varied methodologies in the calculation of stacking (use of IRAP Project Costs [IPC] or Total Project Costs [TPC]) are used by different assurance and monitoring functions, resulting in "false positive" conclusions as to non-compliance and impacting negatively the effectiveness and efficiency of the process.

Clarification with clients regarding stacking through the contribution agreements and leveraging SONAR to gather more consistent data (IPC, TPC, total government funding) represent opportunities to improve the efficiency and effectiveness of the process to mitigate risk of non-compliance.

4.1 (b) Advances

Treasury Board policy requires that advances are only allowed in limited circumstances.

  • TBS Directive on Transfer Payments (6.4.3): Departmental managers are responsible for ensuring that, where advance payments of a contribution are essential to the achievement of objectives, they are specifically provided for in the funding agreement and are based on the recipient's cash flow requirements.

IRAP program documentation very clearly states that IRAP contributions are to be made on a reimbursement basis (Field Manual, SOPs, contribution agreement template, and interpretation guidelines). Therefore there is no documentation providing guidance on criteria for the approval of advances, any special authorities for approving advances or definition of processes and requirements surrounding the reconciliation, adjustment and approval of subsequent claims.

During interviews, IRAP officers and managers emphasized that IRAP is a reimbursement-based program and that advances are exceptions. Interviews noted an advance may occur when a contractor deposit is required but this would need to be documented manually in the paper file or as an attachment in SONAR. Neither SONAR nor SAP has the capability to automatically identify to a system user the existence of an advance payment.

Data mining techniques were employed to identify possible payments made in advance of costs incurred. Only 421 of 3,233 first claims for the selected regions were of rounded amounts and only seven of them were made within 30 days of the contribution agreement signing date. Reasonable assurance was obtained that risk of non-compliance in relation to advances is low.

IRAP policies limit the potential for advance funding payments as the program is managed on a cost reimbursement basis.

4.1 (c) Assignment and Novations

An assignment agreement is a standard three-party document that is used when a firm and NRC agree to assign an IRAP project to a third party. A novation occurs when NRC chooses to assign an IRAP project to another party when the original firm is in default of the contribution agreement terms and conditions.

The requirements for assignments and novations are documented in the Field Manual along with standard templates. Interviews conducted with the Atlantic, West and Ontario regions indicated that ITAs and RCAOs have a clear understanding of their roles and responsibilities and noted that assignment and novations are not a frequent occurrence. ITAs and RCAOs are able to identify assignments and novations as part of their monitoring of IRAP recipients and can obtain further assistance from the IRAP operational group if required. Contribution agreements require recipients to obtain NRC's written consent prior to undertaking any action that would limit their ability to meet their contribution agreement terms and conditions; changes in ownership or a sale of intellectual property are events that would trigger an assignment or novation.

An IRAP working group is reviewing the process for recording assignments and novations in SONAR and SAP to ensure that files are accurately cross-referenced and that committed funding levels are properly documented in SAP. The working group is assessing existing procedures and will determine the appropriate steps for recording the assignment and/or novation in SONAR and processing the contribution agreements in NRC's financial system while retaining the necessary document controls.

Data-mining analysis provided reasonable assurance that assignment and novation are not a common occurrence thus resulting in low inherent risk. An analysis of the overall population of 3,233 contribution agreements indicated 25 agreements with nine potential assignment and novations based on common project names; of those, only two groups of projects, or four agreements in total, were assignments The audit fieldwork found the assignments were properly documented in SONAR.

 

4.1 (d) Amendments

The process for contribution agreement amendments for both firms and organizations are understood by ITAs and RCAOs as noted in the interviews with the regions and well documented in the respective Field Manuals.

There is an effective amendment module in SONAR for documenting the recipient request, ITA assessment and the contracting authority's rationale and approval of the amendment. Currently the amendment module is only utilized for firms. The amendment procedures for organizations are similar, but they are either documented directly in the paper file or through supporting document attachments in SONAR.

The normal amendment process requires sign off by the NRC-IRAP contracting authority and is utilized when there are changes to the funding level or substantive changes to the description of activities. The Field Manual allows for certain amendments to be processed without sign-off from the contracting authority. These amendments can be utilized for administrative changes to the contribution agreement such as revisions to the recipient's name or address, end date extension within the fiscal year and changes to the claim frequency or scheduled meetings. The reason for the informal amendment and the approval by email must be recorded in the IRAP file.

The audit sample population for firm contribution agreement amendments was selected from Atlantic, West and Ontario regions. The audit sample shows that 18% (388 of 2,140) of IRAP projects had amendments: 184 were for funding level changes (114 were increases and 70 were decreases). The remaining 204 were for non-financial changes. There were 220 projects that had only one amendment and 168 had two or more amendments.

The full audit population for amendments (excluding DTAPP and Youth) showed that 21% (689 of 3,233) of IRAP projects had amendments: 301 were for funding level changes (212 were increases and 89 were decreases). The remaining 388 were for non-financial changes. There were 379 projects that had only one amendment and 310 had two or more amendments.

A sample of 12 firms was selected for the three regions based on the amendments documented in SONAR. Within the sampled firms, we found a total of 40 amendments with 100% compliance (amendments were properly documented and supported in SONAR). A sample of 12 organizations for the three regions was selected randomly as organization amendment data is not tracked in SONAR. We found that 5 of 12 organizations did not have an amendment and the remaining 7 files had a total of 11 amendments which were properly documented and supported in SONAR. Organizations comprise 7% of the audit population (230 of 3,233) and 8.5% of the IRAP funding ($27M of $319M).

There is an effective SONAR amendment module for firms to provide accurate and timely monitoring information on the total number and type of amendments.

Amendments for organizations are not managed through the SONAR amendment module; hence there is not data available on the type and number of organization amendments. Amendments for organizations were found to be effectively managed through the paper file or by attachments in SONAR.

4.1 (e) Post Payment Validations

The Post Payment Validation (PPV) process is well documented in the Field Manual for both firms and organizations; describing the required documentation to support the process as well as the required frequency of the PPV based on the ITA's risk assessment of the project. IRAP has transferred the responsibility of the PPV from the ITA to the RCAO as per a recommendation in the IRAP Business Process Review. The RCAO is responsible for the preliminary review of both the monthly claim and claim supporting documents as per the RCAO checklist in the Field Manual. The ITA retains responsibility for Section 34 approval (see Governance and Accountability).

A PPV is required on the first claim for low risk projects unless waived by the ITA. Medium risk projects have a PPV completed on the first and final claims. High risk projects require a first and final PPV and subsequent PPVs at no more than six month intervals.

Data-mining of firms and organizations from all regions excluding DTAPP and Youth projects found the following breakdown for 3,233 contributions: 1,646 low risk projects; 1,130 medium risk projects; 17 high risk projects; 225 projects unrated (organizations); and 215 with waived PPVs. From the three selected regions, Ontario, West and Atlantic for firms and organizations (2,140) had 1,136 low risk projects, 674 medium, 16 high, 153 unrated (organizations) and 161 projects with waived PPV requirements.

Our sample of 12 files was selected from Ontario, West and Atlantic (three organizations and nine firms). The nine firms had 16 PPVs and the three organizations had PPVs waived. We found 100% compliance with the PPV processes. Validations were found to have been reviewed as per the RCAO checklist and approved by an appropriate delegated authority. Where no PPV had been performed, supporting documentation was on file to justify the waiving of the claim validation.

Finance Branch FMD tested 40 IRAP projects for FY2012-13 noting eight (20%) critical deviations for insufficient Section 34 payment certification. TPAS recipient audits for 2010-2011 and 2011-2012 did not test the PPV or Section 34 processes directly. External auditors noted that 10 of 50 files reviewed had over-claims relating to salaries, contractors fees and/or ineligible costs.

 

Recommendation 3: IRAP should be more proactive in managing the possible risk of stacking and should better leverage SONAR's capabilities. This will also ensure consistency of the procedures and agreement templates with the approved 2013 Terms and Conditions for the IRAP Program. (Priority: LOW)

5. Training Capacity and Competencies Adequate

Criterion 5.1: The program's training framework provides adequate support for employees (ITAs and RCAOs) to discharge their responsibilities.

Responsibility for IRAP training is shared between IRAP National Office, IRAP Regional Executive Directors and NRC's Finance Branch. National Office has focused on process-related training, Finance Branch has been responsible for financial and financial system-related training and the regions have focused on technical/scientific and other training. IRAP senior management has highlighted training as a priority and a number of initiatives have been implemented to develop a more formal and coordinated approach to training.

A Training Needs Assessment and Training Plan, as well as a National Training Framework, were developed in FY11/12 to prioritize and guide training activities during the three year period starting in FY12/13. As well, IRAP's Strategic Plan (2012-2015) identifies "Invest in our People" as one the program's three priorities.

The National Training Manager position was staffed in FY11/12 and is well supported by the Senior Leadership Team (SLT). The IRAP training program includes: (1) identification of the types of training required; (2) prioritization of training needs by SLT: (3) identifying common training needs across the country and developing consolidated offerings; (4) leveraging existing IRAP structures and committees to share information and support training (5) developing on-line courses, where appropriate, to maximize access to training; and (6) in support of continuous improvement, initial work on developing a systematic approach to assessing the effectiveness of IRAP training.

Although good progress is being made, interviewees noted several ongoing concerns:(1) delays in the development and availability of the national orientation program for new hires and the resultant need for regions to develop and deliver regional programs, (2) lack of access to training opportunities for regional personnel, (3) inability to participate in training due to workload, and (4) insufficient opportunities for refresher training.

The IRAP training approach adequately provides employees (ITAs and RCAOs) with the necessary training to support discharge of their responsibilities.

Several initiatives have been implemented in recent years to develop a more formal and coordinated approach to IRAP training. The most recent versions of IRAP's strategic and business plans highlights training as a priority for the program and therefore it is anticipated that there will be continued efforts to improve the effectiveness and efficiency of IRAP training.

IRAP should include the web-based SONAR program and the changes to ITA and RCAO responsibilities in the IRAP training plan

See Recommendations 1 and 4.

6. Value and Ethics Adequate

Criterion 6.1: The program's values and ethics, code of conduct or equivalent policies are understood and support the needs of the program.

IRAP is adequately supported by value and ethics and code of conduct policies related to conflict of interest (COI).

IRAP senior management relies on the work of NRC's Secretary General's Office (SGO) to develop and communicate NRC values and ethics. In addition to leveraging the tools and mechanisms provided by the SGO, IRAP senior management has taken considerable action to support the development, communication and application of IRAP values and ethics, including: (1) development of the IRAP Code of Conduct and an accompanying "Fact Sheet" for IRAP clients; (2) inclusion of clauses in all IRAP funding agreements stating that the recipient warrants and covenants against engaging in specific COI transactions and activities; (3) appointment of an IRAP position (the Strategic Project Officer) to serve as a national focal point for IRAP-related COI issues, a position that reports directly to the Vice President, NRC-IRAP and sits as a member of the SLT; and (4) ensuring that the Strategic Project Officer liaises and co-ordinates with NRC's SGO.

IRAP Executive Directors ensure that values and ethics are periodically discussed at national or regional management meetings. At the operational level, Directors are required to regularly communicate expectations around IRAP values and ethics and COI to their direct reports and are also tasked with providing operational level advice and guidance on all COI, or potential COI, matters. The audit found that all interviewees were familiar with IRAP's Code of Conduct. The senior IRAP officers interviewed were aware of their responsibilities related to ensuring compliance with IRAP values and ethics and the importance of strict compliance with federal government, NRC and IRAP values and ethics. The IRAP Field Manual has links to key values and ethics documents and there are specific clauses in the contribution agreement for conflict of interest and lobbying.

Director-level IRAP officers across the country hold one on one COI-related discussions and offer case-by-case advice to ITAs. Workload and the absence of operational level or interpretive guidelines creates a risk that the guidance and advice provided is not consistent nationally or aligned with senior management expectations.

IRAP management is actively involved in ensuring that IRAP's value and ethics and code of conduct expectations are understood and adequately support the needs of the program.

The inherent values and ethics risk for IRAP is high due to the complexity and significance of the program, the decentralized management structure, the expansion of program responsibilities and the on-boarding of new ITAs. There are opportunities for IRAP to enhance the training for values and ethics to ensure consistent communication and guidance to IRAP employees for values and ethics.

Recommendation 4: In finalizing the implementation of its training strategy IRAP should consider including standardized components for values and ethics. [Priority: MODERATE]

 

Appendix A: Audit Criteria

No. Lines of Enquiry
1. Governance and Accountability
1.1 The program's management control framework is supported by an adequate governance structure and related governance mechanisms.
1.2 Authority, responsibility and accountability is clearly defined and well understood for all phases of the program's life-cycle and for both Head Office and regional locations.
2. Monitoring and Oversight
2.1 Management Oversight: Oversight processes and mechanisms provide NRC management with assurance that risk of non-compliance with TBS policy and legislation is adequately managed within senior management's risk tolerance level.
2.2 Risk-Based Monitoring: The level of monitoring of recipients and the reporting required from recipients is risk-based.
3. Information for Decision Making (Financial)
3.1 Management has access to timely and reliable financial information to support monitoring, reporting and decision making.
4. Contribution Agreement Management Process
4.1 Employees are provided with the requisite guidance, tools, resources and information to ensure compliance with NRC and TB requirements for contribution agreement management. Employees adhere to the requirements and guidance provided for the program. Note: Audit scope is limited to Stacking, Advances, Assignments and Novations, Amendments and Post Payment Validations (PPVs)
5. Training Capacity and Competencies
5.1 The program provides employees (ITAs and RCAOs) with the necessary training to support discharge of their responsibilities.
6. Value and Ethics
6.1 The program's values and ethics, code of conduct or equivalent polices are understood and support the needs of the program.
 

Appendix B: Potential Overall Ratings

Management Attention Required

There are significant weaknesses in the design and/or effectiveness of the selected key management controls that require management's attention. Critical practices / processes are not in compliance with Government of Canada and NRC regulations, policies and directives. There are significant opportunities for development.

Needs Improvement

The design and/or effectiveness of the selected key management controls needs improvement. Some areas of practices / processes are in compliance with Government of Canada and NRC regulations, policies and directives. There are several opportunities for improvement.

Adequate

The design and/or effectiveness of the selected key management controls is adequate. Most areas of practices / processes are in compliance with Government of Canada and NRC regulations, policies and directives. There are a few opportunities improvement.

Strong

The design and/or effectiveness of the selected key management controls is strong. All areas of practices / processes are in compliance with Government of Canada and NRC regulations, policies and directives. No areas for improvement were identified.

 

Appendix C: Management Action Plan

Definition of Priority of Recommendations
High Implementation is recommended within six months to reduce the risk of potential high likelihood and/or high impact events that may adversely affect the integrity of NRC's governance, risk management and control processes.
Moderate Implementation is recommended within one year to reduce the risk of potential events that may adversely affect the integrity of NRC's governance, risk management and control processes.
Low Implementation is recommended within one year to adopt best practices and/or strengthen the integrity of NRC's governance, risk management and control processes.
Recommendation Corrective Management
Action Plan
Expected Implementation Date and Responsible NRC Contact

1. Line of Enquiry: Monitoring and Oversight

Recommendation 1: IRAP Should consider establishing a comprehensive oversight framework that aligns all management monitoring activities and processes and ensures level of effort corresponds with management's risk tolerance. IRAP management should assume ownership of the framework which would be vetted by NRC CFO and approved by NRC president. The framework should include quality assurance elements embedded in the business flow, an updated approach to the recipient audits, standardized checklists for processing recipient claims and training in the context of the new control environment for web-based SONAR. [Priority: HIGH]

  • IRAP will further develop its comprehensive oversight framework, the Management Control Framework (MCF).
  • The MCF will take a holistic approach to business flow processes, allowing for:
    • Risk management
    • Identification of any deficiencies
    • Adequate support for staff
  • IRAP will integrate the QA elements in its MCF;
  • IRAP will take ownership of the Framework to be vetted by NRC CFO and approved by the President;
  • IRAP will review its approach to recipients audit by revising its TORs to ensure they reflect IRAP's evolving programs;
  • IRAP will develop standardized instructions (e.g. check-list) for RCAOs for processing claims;
  • With web-based SONAR, IRAP will develop on-line instructions for staff; and
  • IRAP will provide ongoing training for SONAR (e.g. WebEx sessions).

Status: In progress

Date: August 1, 2014

Contact:

Executive Director,

IRAP National Office

2. Line of Enquiry: Information for Decision Making (Financial)

Recommendation 2: IRAP should finalize the systems improvement project allowing for better linkages between the SONAR system and SAP, as well as the development of dashboards for information management. [Priority: HIGH]

  • IRAP continues to improve the reporting functionality of web-based SONAR in support of effective and timely decision making;
  • IRAP's systems improvement project (SONAR SAP Integration) is in progress and will:
    • reduce the number of manual entries;
    • reduce the number of errors;
    • result in a streamlined process; and
    • allow for better information for management.
  • The Dashboard is being enhanced to better support decision making.

Status: In progress

Date: August 1, 2014

Contact:

Executive Director,

IRAP National Office

3. Line of Enquiry: Contribution agreement Management Process

Recommendation 3: IRAP should be more proactive in managing the possible risk of stacking and should better leverage SONAR's capabilities. This will also ensure consistency of the procedures and agreement templates with the approved 2013 Terms and Conditions for the IRAP Program.(Priority: LOW)

  • IRAP will bring additional clarity on Stacking in its Contribution Agreements;
  • IRAP applies the Total Project Costs (TPC) to calculate stacking; and
  • In addition to the Contribution Agreement clause dealing with stacking, IRAP will ensure that stacking is addressed verbally with clients at the beginning of any funded project and addressed again at the end of the funded project.

Status: In progress

Date: June 1, 2014

Contact:

Executive Director,

IRAP National Office

4. Line of Enquiry: Value and Ethics

Recommendation 4: In finalizing the implementation of its training strategy IRAP should consider including standardized components for values and ethics. [Priority: MODERATE]

  • Training is being provided to staff (WebEx, conference calls, post-hiring meeting with supervisor, etc.);
  • IRAP has been addressing proper support to the changing responsibilities of ITAs and RCAOs with regards to treatment of claims and Post-payment validations;
  • Training on Values and Ethics is being done at regional meetings by IRAP's COI resource person; and
  • Annual message from NRC-IRAP VP reminding employees' obligations and referring to the official NRC documents on ethics and COI.

Status: In progress

Date: July 1, 2014

Contact:

Vice-President, IRAP