Table of contents
Alternate format: Audit of Research and licensing agreement management (PDF, 923 KB)
Prepared by: Office of Audit and Evaluation
Approval: September 2019
Cat. No.: NR16-310/2020E-PDF
ISBN: 978-0-660-34230-6
Executive summary
As outlined in the National Research Council (NRC) Priorities for FY2018–2019, the NRC has a renewed focus on research excellence and the enablement of research and development collaboration with other government entities, post-secondary institutions, and business innovators. In working towards these priorities, it is important to ensure that the NRC has a robust framework that supports the establishment and ongoing management of these agreements.
The NRC Office of Audit and Evaluation undertook an audit of research agreement management upon the request of senior management. The objective of the audit was to provide assurance that the appropriate governance, risk management, and controls have been implemented for the framework that supports the NRC's research and licensing agreement activities, to ensure they are managed according to Government of Canada (GC) and NRC policy, while enabling the organization's innovation and research-based mandate. The audit focused on activities between April 1, 2017 and November 30, 2018.
The NRC moved from a decentralized Research Institute model before 2012 to a Portfolio model that included some centralization of corporate services. In 2017, Portfolios were rebranded to Research centres. There is shared or joint responsibility between corporate services and research centres for some of the processes within the audit scope, representing a hybrid approach for service delivery. When implemented correctly this model can be very successful, as this approach mitigates risk and achieves efficiencies while not impeding the effectiveness and innovation within individual business units (e.g. research centre). A hybrid approach requires significant ongoing attention to ensure it operates effectively in terms of governance and oversight, policy and training, enabling systems, as well as monitoring / reporting. This means it is even more important to ensure the most efficient use of resources, in terms of standardized, automated and streamlined processes enabled through common systems and tools, and the establishment of processes for the sharing of best practices and continuous improvement.
The audit found well documented and robust processes around agreement initiation (e.g., project approvals, project pricing) in some of the research centres. In addition, through the centralization of the Contract Advisory Team (CAT), there has been an increase on the use of standard contract agreement templates. Furthermore, a Project Management (PM) Harmonization Working Group has put significant effort into the development of guidance, tools, and templates related to agreement initiation and project management to share across the organization. However, the group is informal, with no authority and as such, the tools and process improvements developed by them are not leveraged by all research centres.
In addition to the strengths outlined above, the audit identified key areas for improvement as outlined below and further detailed in the audit report.
Figure 1: key takeaways
Long description of Figure 1: key takeaways
This is a flow chart that describes the business process control weakness identified through detailed audit file testing.
- A lack of clarity on accountability, roles and responsibilities (Finding 1) has an impact of accountability weakness, which leads to 3 findings:
- There is lack of clarity in regards to policy requirements, and processes have not been standardized or aligned to requirements across the organization (Finding 2).
- Processes are "one size fits all" and are not currently related to the level of assessed risk (Finding 3).
- IT systems are currently not integrated and have limited automated workflow within each system (Finding 4).
These 4 findings have an impact on business process control weakness (including inefficiencies) and non-compliance with NRC policy.
Finding 1 - governance and accountability
There is a lack of clarity on the accountability and the nature of the roles and responsibilities for agreement management between research centres and the Business Services Team/ Business Management Services (BST/BMS), specifically related to the nature of how BST/BMS should be enabling these processes.
The management of research and licensing agreements cut across the NRC's current organizational structure, implicating research centres, as well as supporting corporate functions under the direction of the VP, Business and Professional Services and the VP, Corporate Services and Chief Financial Officer (CFO). This has limited the ability for decisions to be taken, or to further clarify the accountability for agreement management. Performance reports related to agreements are only presented to the NRC Executive Committee (EXCO), currently not a decision-making forum.
Recommendation
- The Vice-President (VP), Business and Professional Services, through the Director General (DG), National Programs and Business Services, should define a governance model for the management of research agreements for discussion and approval by SEC members. This should include clearly defined accountabilities, authorities and responsibilities for BST/BMS and the research centres. [Priority: short-term]
Finding 2 – policy and processes
The NRC Client Agreement Policy (2014) does not provide a complete set of requirements, nor enough direction to ensure effective and consistent "operationalization" of the requirements that are outlined within the policy. BST/BMS has developed guidelines and templates related to agreements; however, taken as a whole, these do not represent a comprehensive set of tools. In a number of aspects, current practices differ from the process outlined in the policy where both BST and Research centres have made adjustments to reflect changes in the operating environment.
Research centres have also developed their own processes. This represents a barrier when different areas within the NRC are working on the same project.
A compliance program to ensure adherence to policy and business process requirements has not been established for agreements.
Research centres have implemented different models for project management and support under a commonly mandated use of the corporate project management system for coding of project information. There is no central project management support office to provide project management oversight or support (i.e., common tools, processes or a forum for sharing best practices and lessons learned). No recommendations for the improvement of project management practices are included in this report as this will be examined in a separate audit in fiscal year 2019–2020.
Recommendations
- Once a governance model is established (Recommendation #1), the VP, Business and Professional Services, through the DG, National Programs and Business Services, in collaboration and consultation with the Research DGs and VPs should update existing agreement policies and ensure related tools and training are available to staff and management. This presents an opportunity to align key business controls across the NRC. [Priority: medium-term]
- The VP, Business and Professional Services, through the DG, National Programs and Business Services, should develop a process to monitor and report on policy compliance for the agreement process. This includes developing a formal process to identify and approve risk-based exceptions to policy requirements. [Priority: Long-term]
Finding 3 – risk management
A consistent, enterprise-wide risk-based framework for agreements has not been implemented for the research and licensing agreement processes. The process for the approval of agreements within individual research centres is generally "one size fits all", and is not currently tailored to the level of assessed risk. This has resulted in process inefficiencies that could be streamlined for routine or lower risk agreements, as well as the potential of the acceptance of projects that exceed management's risk appetite.
Recommendation
- Dependant on the implementation of recommendation 1, the VP, Business and Professional Services, through the DG, National Programs and Business Services, in collaboration and consultation with the Research DGs and VPs, should develop and implement an enterprise-wide risk-based framework for a consistent and effective approach to risk management. [Priority: medium-term]
Finding 4 - IT systems and reporting
The IT systems used in the management of research agreements are currently not integrated and have limited automated workflow within each system, resulting in inefficient and more error prone manual processes and data entry. In some cases, research centres and/or individual users are maintaining separate or parallel systems, and using workarounds. This also has an impact on the quality of reporting information for decision making and performance management for research agreements.
Recommendation
- The VP, Business and Professional Services, through the DG, National Programs and Business Services, in consultation with the Research DGs and VPs should identify common agreement business requirements and implement the functionalities in the IT systems, automate workflows, provide integration between systems where required, and ensure data integrity. [Priority: long-term]
Overview of recommendations and priority
Long description of Overview of recommendations and priority
This chart shows which recommendations will be implemented in the short term, medium term, and longer term.
- 2 recommendations need to be implemented in the short-term:
- Define a governance model for research agreement for discussion and approval by the Senior Executive Committee.
- Clearly define accountabilities, authorities, and responsibilities, for BST/BMS and Research Centres.
These need to be implemented before completing the implementation of all other recommendations in order to ensure accountability is clear for decision making.
- 4 recommendations should be done in the medium-term:
- Update existing agreement policies.
- Key business process controls are aligned across NRC, supported with common tools and templates.
- Training material should be developed and communicated to staff.
- Implement an enterprise-wide risk-based framework for a consistent and effective approach to risk management.
Although all should be done in the medium-term, there are dependencies in that policies should be completed first to inform tools/templates, and then training developed on those.
- 2 recommendations could be implemented in the longer-term:
- Develop a process to monitor and report on policy compliance for the agreement process including a formal process to identify and approve risk-based exceptions to policy requirements.
- Implement the functionality in the IT systems required to support business requirements as well as automate workflows and provide integration between systems where required.
Note: Some "quick wins" related to automation could be introduced much sooner. Consistent requirements are required to be determined first in order to achieve success for this recommendation.
Audit opinion and conclusion
In my opinion as Chief Audit Executive, improvements to the NRC's governance, risk management, and controls related to research and licensing agreements are required to ensure they are managed in accordance with Government of Canada (GC) and NRC policy, while enabling the organization's innovation and research-based mandate. Notably, there is a lack of clarity related to roles and responsibilities for the agreement process, as well as gaps in the policy framework and weaknesses in business process controls (e.g., not documented, not automated). These issues, combined with a lack of compliance monitoring increases the risk of non-compliance to NRC policy. The NRC should strengthen its management practices through various ongoing improvements and risk-based considerations, as set out in this audit report.
Statement of conformance
In my professional judgement as Chief Audit Executive, the audit conforms to the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing (IIA Standards) and the Code of Ethics, as supported by the results of the Quality Assurance and Improvement Program.
Alexandra Dagger, CIA, Chief Audit Executive
Audit team
The audit was undertaken by Deloitte on behalf of the NRC Office of Audit and Evaluation.
Audit report
Introduction
The National Research Council (NRC) generated in excess of $180M in revenues in fiscal year 2017–2018 from research agreements and the provision of testing and technical services with industry, academia, and other government entities. An effective framework to manage research agreements is vital to ensure that arrangements with the NRC's clients protect taxpayer interests and are aligned with the NRC's mandate and renewed focus on research excellence and the enablement of research and development collaboration. The Audit of Research agreement management was identified as a high-priority project and was approved by the President of the NRC following the recommendations of the Departmental Audit Committee (DAC) as part of the NRC 2019 to 2021 Risk-Based Audit Plan.
Background and context
The NRC is an agency of the Government of Canada (GC), reporting to Parliament through the Minister of Innovation, Science and Economic Development (ISED). It is governed by a Council of appointees drawn from its client community. The NRC is led by a Deputy Head-equivalent President, and a number of Vice Presidents (VPs).
As outlined in the NRC's Priorities for FY2018–2019, the NRC has a renewed focus on research excellence and the enablement of research and development collaboration with other government entities, post-secondary institutions, and business innovators. In Budget 2017, the Government of Canada introduced a Skills and Innovation Plan, focused on helping Canada realize its potential as a global leader in innovation. In support of the federal government's plan, the NRC is looking to increase its focus on collaboration and accelerating research and development to support business innovation. Furthermore, in Budget 2018, the NRC received an additional $108 million per year to be used to meet these objectives.
In working towards these priorities, it is important to ensure that the NRC has a robust framework that supports the establishment and ongoing management of these agreements. This includes the implementation of appropriate governance, policy, business process, enabling IT systems, reporting, and overall risk management.
Client agreements are defined in the NRC's Client Agreement Policy as agreements whereby the NRC provides a service or a licence right to a client and business agreements including Non-Disclosure Arrangements (NDA), Memoranda of Understanding (MoU), and Letters of Intent (LoI). Client agreements can be categorized in many ways, but based on the NRC's Client Agreement Policy and for the purposes of this audit, client agreements are categorized as follows:
- Research agreements – custom work by the NRC that usually includes significant intellectual contribution by the NRC, and the development of intellectual property (IP) is a generally expected outcome. These may be Research Services Agreements (RSA) in which the client pays the full cost or Collaborative Research Agreements (CRA) which are joint research engagements with incremental intellectual contribution by both the NRC and the collaborator(s).
- Licensing agreements – used when the NRC wishes to grant rights to the NRC's intellectual property to a third party. Licensing agreements may be stand-alone agreements or may be embedded within a research agreement.
- Testing and Technical Services Agreements – standard services delivered using existing NRC technology, expertise and resources with a low level of technical risks.
For any of these agreement types, partners may include private sector companies, academic institutions, not-for-profit organizations, or government entities.
In 2012 the NRC moved from a decentralized Research Institute model to a research centre model that includes some centralization of corporate services, including aspects of the research and licensing agreement process, resulting in shared or joint responsibility between corporate services and research centres for some of the processes within the scope of this audit.The high-level accountabilities of key stakeholders as they relate to the research agreement process are outlined in Figure 2.
Figure 2: key stakeholders
Long description of Figure 2 - key stakeholders
This image represents 3 groups of key stakeholders, each with a short description of their accountability, using integrated circles:
- NRC's 14 Research Centres (RC's) which each report to a Research VP.
- Accountability for agreement development including defining project scope and pricing, risk assessment, mitigation, and acceptance, project and agreement approvals, and ongoing project management throughout the life of an agreement rests with each of the NRC Research Centre Directors General which report to a Research VP.
- National Programs and Business Services (NPBS) under VP, Business and Professional Services.
- Accountability for supporting NRC's Research Centres in the initiation and management of agreements rests with the Business Services Team (BST) under the DG, National programs and Business Services reporting to the VP, Business and Professional Services.
- Finance and Procurement Services Branch under the VP, Corporate Services and Chief Financial Officer (CFO).
- Accountability for the financial management of agreements with specific responsibilities for review and approval of financial terms and conditions within agreements, and managing billing and revenue collection activities for agreements rest with Finance and Procurement Services (FPS) under the DG, FPS reporting to the VP, Corporate Services and Chief Financial Officer.
The NRC has a number of IT systems in place to support the development and on-going management of research and licensing agreements through their lifecycle:
- An Agreement Management System (AMS) consisting of software built on the Microsoft Dynamics Client Relationship Management (CRM) platform that is primarily used by Business Management Support (BMS) to track interactions with potential partners from the identification of potential research and licensing opportunities to the execution of an agreement. The system is also used by the Business Services Team (BST) and research centres as a centralized document repository for the final versions of executed agreements.
- A cloud-based IP Management System (IPMS) used by the Business Services Team (BST) to record the terms and conditions of licensing agreements upon the execution of a licensing agreement. Throughout the life of each licensing agreement, the system is also used to track the collection of royalties and payment of awards.
- The Departmental Financial Management System (DFMS). The NRC utilizes a number of modules in support of the agreement process, including:
- The Sales and Distribution Module (SDM) is used by the NRC's Finance and Procurement Services Branch to record and manage all financial processes from order to cash including billing, collections, and financial reporting activities related to client agreements.
- The Project Systems Module (PSM) is used primarily by the NRC's project managers located in research centres to plan and manage research projects. The module supports the creation and execution of project plans, project accounting, and performance and progress reporting. When used as intended, project plans in the module are integrated with procurement, financial accounting, sales, and time recording with all information flowing to the respective project plan.
About the audit
Objective
The objective of the audit was to provide assurance that the appropriate governance, risk management, and controls have been implemented for the framework that supports the NRC's research and licensing agreement activities, to ensure they are managed according to Government of Canada (GC) and NRC policy, while enabling the organization's innovation and research-based mandate.
Scope
Based on the Planning Phase risk assessment, the Conduct Phase of the audit focused on the key foundational elements
- The NRC's governance and policy framework
- Business processes and supporting elements, and
- IT systems and reporting processes.
An assessment of the IP-related royalties and awards process has been included within the scope of a separate auditFootnote 1. Excluded from the scope of this audit were business development processes that occur prior to the agreement initiation process. The scope of the audit focused on testing the design and implementation, as well as the operating effectiveness, of the current state business process controls from April 1, 2017 to November 30, 2018. This includes detailed audit file testing of agreements that were in force during this time period, even if they originated prior to the audit period.
Approach
The audit was conducted in accordance with Institute of Internal Auditors (IIA) Standards and the Treasury Board Policy on Internal Audit and related policy instruments. The audit criteria was derived from a variety of sources, including TB policy (e.g., Policy on Financial Management), NRC policy and guidance, and leading practice.
Through the Conduct Phase of the audit, fieldwork was conducted through the review of documentation, system walkthroughs, and completion of interviews with key stakeholders including management and staff of the following groups within the NRC:
- Business Services Team (BST) and Business Management Support (BMS) under the DG, National Programs and Business Services reporting to the VP, Business and Professional Services
- Finance and Procurement Services Branch (Accounting Operations, Advisory Services and Resource Management, Accounts Receivable, Accounts Payable) reporting to the DG, Finance and Procurement Services under the VP, Corporate Services and Chief Financial Officer (CFO)
- Research centres (Management, Researchers, Project Managers) under Directors General reporting to a Research VP
Figure 3: scoped-in research
Long description of Figure 3: scoped-in research
Map of Canada indicating location of four NRC research centres:
- Energy, Mining and Environment (EME) in British Columbia
- Automotive and Surface Transportation (AST) in Ontario
- Human Health Therapeutics (HHT) in Quebec
- Medical Devices (MD) in Quebec
A total of 4 research centres were selected for site visits to ensure coverage across multiple research divisions and geographic locations. The volume and types of agreements generated by each research centre was also taken into consideration when selecting the sites.
In addition to conducting interviews with key stakeholders in the agreements process, a sample of 42 research and 42 licensing agreements were selected to assess the effectiveness of key business process controls. All agreements selected were "active" during the audit period. To ensure broad coverage of the agreements in scope for the audit, a number of attributes were taken into consideration when selecting the sample, including research centre, agreement type and type of client. Although they represent a smaller percentage of the NRC's research agreements, an emphasis was placed on research services and collaboration research agreements due to the higher degree of complexity and higher value of agreements. A smaller number of testing and technical services and other (master agreements) agreements were also selected.
Figure 4: research and licensing agreements by research centre in FY2018
Agreements by research centre
Long description of Agreements by research centre
| Research Centre | Other | Research and Collaboration | Testing and Technical Services | Selected for Detailed Testing |
|---|---|---|---|---|
| Human Health Therapeutics | 16 | 291 | 182 | 6 |
| Automotive and Surface Transportation | 13 | 203 | 461 | 7 |
| Energy Mining and Environment | 8 | 90 | 108 | 3 |
| Medical Devices | 7 | 87 | 27 | 3 |
| Advanced Electronics and Photonics | 0 | 23 | 73 | 0 |
| Aerospace | 25 | 144 | 307 | 7 |
| Aquatic and Crop Resource Development | 17 | 95 | 462 | 4 |
| Construction | 19 | 89 | 336 | 4 |
| Digital Technologies | 1 | 19 | 26 | 1 |
| Metrology | 9 | 28 | 594 | 3 |
| Nanotechnology | 27 | 28 | 49 | 1 |
| Ocean, Coastal and River Engineering | 7 | 42 | 118 | 2 |
| Security and Disruptive Technologies | 7 | 35 | 39 | 1 |
Licensing agreements by research centre
Long description of Licensing agreements by research centre
| Research Centre | Total License Agreements | Selected for Detailed Testing |
|---|---|---|
| Human Health and Therapeutics | 237 | 13 |
| Automotive and Surface Transportation | 17 | 3 |
| Energy Mining and Environment | 19 | 4 |
| Medical Devices | 31 | 7 |
| Advanced Electronics and Photonics | 13 | 1 |
| Aerospace | 12 | 2 |
| Aquatic and Crop Resource Development | 25 | 3 |
| Construction | 2 | 0 |
| Digital Technologies | 51 | 4 |
| Metrology | 30 | 2 |
| Nanotechnology | 0 | 0 |
| Ocean, Coastal and River Engineering | 6 | 2 |
| Security and Disruptive Technologies | 15 | 1 |
Audit findings and recommendations
The audit found well documented and robust processes around agreement initiation (e.g., project approvals, project pricing) in some of the research centres. In addition, through the centralization of the Contract Advisory Team (CAT), there has been an increase on the use of standard contract agreement templates. Furthermore, the Project Management (PM) Harmonization Working Group has put significant effort into the development of guidance, tools, and templates related to agreement initiation and project management to share across the organization. However, the group is informal, with no authority and as such, the tools and process improvements developed by them are not leveraged by all research centres.
In addition to the above, the audit identified several areas for improvement. An overview of these key findings and their impact on current business process controls and compliance to policy are outlined in Figure 1. Further details on these findings as well as recommendations intended to improve NRC research agreement management are explained further in this report.
Figure 1: key takeaways
Long description of Figure 1: key takeaways
This is a flow chart that describes the business process control weakness identified through detailed audit file testing.
- A lack of clarity on accountability, roles and responsibilities (Finding 1) has an impact of accountability weakness, which leads to 3 findings:
- There is lack of clarity in regards to policy requirements, and processes have not been standardized or aligned to requirements across the organization (Finding 2).
- Processes are "one size fits all" and are not currently related to the level of assessed risk (Finding 3).
- IT systems are currently not integrated and have limited automated workflow within each system (Finding 4).
These 4 findings have an impact on business process control weakness (including inefficiencies) and non-compliance with NRC policy.
Finding 1 - governance and accountability
It was expected that the NRC has clearly defined governance and accountability, as well as roles and responsibilities related to the management of research and licensing agreements. The audit found that there is a lack of clarity on the accountability for agreement management between research centres and BST/BMS, specifically related to the nature of how BST/BMS should be enabling these processes. Furthermore, the current governance structure and reporting that provides oversight over these processes have inhibited the ability for timely decision making.
Observations
The processes used in the management of research agreements cut across the NRC's current organizational structure, involving research centres, which are under the direction of a Research VP, as well as supporting corporate functions under the direction of the VP, Business and Professional Services and the VP, Corporate Services and Chief Financial Officer (CFO). Given this organizational structure, and unclear policy guidance related to authorities, accountabilities and responsibilities, decisions related to the management of research agreements are required to be taken to SEC (comprised of the President and VPs). This has limited the ability for decisions to be taken, including those related to defining roles and responsibilities for research agreements.
When the NRC moved to a more centralized research centre model in 2012, BST/BMS was expected to play an enabling function to research centres for research agreement management. The exact nature of this enabling function has not been defined. A responsibility and accountability matrix (i.e., RACIFootnote 2) was not developed, nor was there formal change management or communications to support understanding and acceptance of the role of BST/BMS. Contract coordinators that are part of the Contract Advisory Team that report to BST/BMS play different roles depending on the research centre that they support, from development of the contracts, to providing advice and oversight on contracts developed by others (i.e., Project Managers or researchers).
There is general agreement that ultimate accountability for most research agreement items rests with the research centres. However, without a formal definition of roles and responsibilities, there can be significant back and forth and holdups between corporate services and research centres when there are disagreements in relation to contract clauses and/or risk acceptance or mitigations. For instance, a research centre indicated that a contract was held up by a member of the Contract Advisory Team due to a disagreement on how to mitigate risk through the contract. Eventually the research centre went ahead without the support of the contract coordinator. However, by that time the opportunity with that client was no longer available.
Quarterly reports with key performance indicators related to NRC's Performance Measurement Framework (PMF), as of the end of the audit period, are presented to EXCO, which is comprised of SEC membership plus the Directors General (DGs). EXCO is a discussion forum and not a decision-making body. It is not possible to determine, what actions, if any, are taken based on the results of the Quarterly reporting provided to EXCO. These reports are not provided to SEC.
Recommendation
- The VP, Business and Professional Services, through the DG, National Programs and Business Services, should define a governance model for the management of research agreements for discussion and approval by SEC members. This should include clearly defined accountabilities, authorities and responsibilities for BST/BMS and the research centres. [Priority: Short-term]
Finding 2 – policy and processes
It was expected that the NRC has implemented a policy framework for agreements that is complete and clearly communicated, and is supported by policies, processes, guidelines, templates and staff training and awareness. The audit found that the NRC Client Agreement Policy does not provide enough direction to ensure effective and consistent "operationalization" of the requirements outlined within the policy. Although BST/BMS has developed a number of guidelines and templates related to research agreements, research centres are each using a variety of processes and templates/tools (i.e., project approvals, pricing). This represents a barrier when different areas within the NRC are working on the same project, and makes it difficult to implement enterprise training and/or a compliance program. A formal compliance program to ensure processes are adhering to requirements as well as to identify problematic files has not been established for research agreements. Project management practices vary widely across research centres.
Observations
The NRC Client Agreement Policy, published in 2014, is a high-level document that does not provide a complete set of requirements, nor enough direction to ensure effective "operationalization" of the requirements that are outlined within the policy. In some cases, the policy indicates that the applicable requirements are provided through research centre policies (e.g., project approval policies). However, through the site visits with research centres, the audit found that generally these types of formal policies do not exist at the research centres (e.g., practices for project approvals). The policy also references guideline documents that in some cases do not exist (e.g., pricing guidelines and recommendations). Furthermore, the policy does not represent the current agreement process. For instance, the policy contemplates a level of legal review that is no longer able to be done, given the NRC's agreement with the Department of Justice.
BST/BMS has developed guidelines and templates related to research agreements (e.g., Client Agreement Guidelines, statement of work [SOW] and Deliverable Drafting Guidelines); however, taken as a whole, these do not represent a comprehensive set of tools. Research centres have developed their own processes, and in some cases their own documentation to support these processes. For instance, 1 research centre has granted the Directors the ability to sign off on agreements, although they do not have this authority in the NRC Financial Signing Authority (FSA) matrix.
In some cases, there are differences in the processes and templates/tools (i.e., project approvals, pricing) not just within individual research centres, but between programs within the same research centre. This represents a barrier when different areas within the NRC are working on the same project. Furthermore, given the current environment of developing or redeveloping processes in isolation as well as the lack of a mechanism for the formal sharing of best practices between research centres, there is a risk that researchers may not be optimally utilized on research work, but involved in process-related/administrative activities.
There is no formal training and learning strategy or plan that has been developed for research and licensing agreement processes; although given the variety of processes practiced across research centres, and the lack of clarity on accountability, an enterprise plan would be difficult to develop. Research centres generally practice informal on-the-job training for research centre specific business processes. BST/BMS does provide system-specific guidance and communications in relation to the systems used within the research agreement process. Detailed user guides are available for the NRC's agreement management system, IP management system, and project management system. Some regular communication of system updates and features are provided to staff for these systems.
Through the performance of detailed audit file testing on the sample of agreements, a number of issues were identified, specifically related to the lack of documentation available to support agreements that were developed and approved in compliance with policy. This makes it difficult for those not involved in the development of the agreements to understand the context and any potential issues related to the agreement. Given the geographic dispersion of NRC operations, the absence of a central repository for project documentation increases risk of error through the lifecycle of a specific project when it transitions among research, BST and finance staff. Specific areas where documentation was not available or practices varied from the documented policy include:
- For the majority of research agreements, evidence that project approval was granted was not able to be provided.
- For the majority of research agreements, evidence that a risk assessment was performed was not able to be provided.
- For the majority of research agreements, evidence that agreement pricing was appropriately developed in accordance with NRC pricing practices was not able to be provided;
- For a smaller number of research and licensing agreements, legal reviews were not performed in accordance with NRC policy.
Based on the NRC Client Agreement Policy, the requirements for documentation (i.e., project approvals, agreement pricing, and risk assessments) are the same for research agreements as they are for licensing agreements; however, these practices are not followed for licensing agreements. None of the licensing agreements that were included in the audit file testing had this documentation on file.
A formal compliance program to ensure adherence to policy and business process requirements has not been established for research agreements.
Project management of research projects is required to ensure they are completed in accordance with the agreed upon statement of work, as well as contract terms and conditions. In relation to project management, there is no central project management office to provide project management oversight or support (i.e., common tools, processes or a forum for sharing best practices and lessons learned). A variety of Project Management Office (PMO) functions have been established in each research centre, with different project management tools, templates and processes. This represents both an inefficiency and a barrier to collaboration across the organization, increasing the risk of ineffective project management. Furthermore, this impacts the ability to understand and oversee the portfolio of projects across the organization. No recommendations for the improvement of project management practices are included in this report as this will be examined in a separate audit in fiscal year 2019–2020.
Recommendations
- Once a governance model is established (Recommendation #1), the VP, Business and Professional Services, through the DG, National Programs and Business Services, in collaboration and consultation with the Research DGs and VPs, should update existing agreement policies and ensure related tools and training are available to staff and management. This presents an opportunity to align key business controls across the NRC. [Priority: medium-term]
- The VP, Business and Professional Services, through the DG, National Programs and Business Services, should develop a process to monitor and report on policy compliance for the agreement process. This includes developing a formal process to identify and approve risk-based exceptions to policy requirements. [Priority: long-term]
Finding 3 – risk management
It was expected that the NRC has implemented an effective risk management framework for the approval and management of agreements. The audit found that a consistent, enterprise-wide risk-based framework for agreements has not been implemented for the research and licensing agreement processes. The process for the approval of agreements within individual research centres is generally "one size fits all", and is not currently tailored to the level of assessed risk.
Observations
BST/BMS has provided high-level guidance on risk management in relation to research and licensing agreement contracts. This includes the Business Risk Management best practices document published in 2014 as well as draft guidelines developed as part of the NRC Project Costing and Pricing Policy and Directive (i.e., Managing Project Risk and Determining Project Contingency Reserves). Note this guideline is focused on budget-related risks and required contingencies. Although a variety of risk-related guidance exists; a formal and consistent enterprise-wide risk-based framework for the management of research agreements, which defines the roles and responsibilities in relation to risk management, has not been implemented. Based on site visits, 3 of 4 research centres do not have formalized risk management processes related to agreement approval and management. The research centre that was the exception has risk management practices that varied from the guidance provided by BST/BMS. Through the performance of detailed file testing for a sample of research and licensing agreements, it was noted that risk assessments were not consistently documented and retained. Based on the site visits to research centres, risk assessment processes vary across research centres and no standardized framework or approach exists for the assessment and mitigation of risks.
Generally, the process for the approval of agreements and its level of rigour is not related to the level of risk. The agreement approval process within individual research centres is generally “one size fits all”. This has resulted in process inefficiencies that could be streamlined for routine or lower risk agreements, as well as the potential of the acceptance of projects that exceed management's risk appetite.
Recommendation
- Dependant on the implementation of recommendation 1, the VP, Business and Professional Services, through the DG, National Programs and Business Services, in collaboration and consultation with the Research DGs and VPs, should develop and implement an enterprise-wide risk-based framework for a consistent and effective approach to risk management. [Priority: medium-term]
Finding 4 – IT systems and reporting
It was expected that the NRC has the appropriate IT systems and tools to support the business requirements for the management of research and licensing agreements. The audit found that the IT systems used in the management of research agreements are currently not integrated and have limited automated workflow within each system, resulting in inefficient and more error prone manual processes and data entry. In some cases, users are maintaining separate or parallel systems, and using workarounds. Many documents and approvals are maintained outside of the official repositories, thus impacting the strength of the audit trail for research and licensing agreements. This also has an impact on the quality of the reporting and performance monitoring being conducted on research and licensing agreements.
Observations
The NRC utilizes a number of IT systems in its management of research agreements as outlined within the Background section of this report. The NRC's research agreement system is primarily used as a repository for finalized agreements. As demonstrated by file testing, there are many documents and approvals maintained outside of the agreement system, thus impacting the strength of the audit trail for research and licensing agreements. One (1) research centre out of 4 selected for a site visit continues to use a parallel database for the management of its research agreements.
Although the capability for automated workflows exists within the AMS, this functionality has currently not been enabled. The implementation of automated workflows are difficult in the current environment given the large variation in processes across research centres (e.g., different levels of approval, approval required at different stages of the process).
Once a licensing agreement is approved in the AMS, the royalty and payment of awards information is manually entered into the NRC's IPMS.
The PSM is intended to be used by the NRC's project managers located in research centres to plan and manage research projects. All agreements, regardless of whether or not they are projects, are required to be set up as a project within the PSM. This reduces efficiency and is labour intensive for smaller scale agreements. Given some users find the project management system overly cumbersome, a variety of other systems are being used by project managers within research centres for day to day project management, such as MS Project, while still manually inputting data into the PSM for corporate reporting purposes. Due to the absence of system integrations between AMS, IPMS, and PSM, processes related to the recording and tracking of financial information and transactions for agreements are inefficient and increase the likelihood of errors as users are required to manually enter information into multiple systems. Based on the data quality and completeness issues of information within the official repositories (e.g., AMS, IPMS, PSM) used for reporting in relation to the management of agreements, this results in difficulty in providing accurate reporting for decision-making.
Recommendation
- The VP, Business and Professional Services, through the DG, National Programs and Business Services, in consultation with the Research DGs and VPs should identify common agreement business requirements and implement the functionalities in the IT systems, automate workflows, provide integration between systems where required, and ensure data integrity. [Priority: long-term]
Appendix A – lines of enquiry
The Conduct Phase of the audit was completed based on the lines of enquiry below, the scope of which included both the management of research and licensing agreements as well as the management of IP-related royalties and awards (noting that the outcome of some licensing agreements are the payment of royalties and awards). For reporting purposes, 2 separate reports were prepared, 1 related to the management of research and licensing agreements (this report) and a separate Audit of IP related royalties and awards.
Line of enquiry
- There is clearly defined accountability for agreement and licensing management activities that is supported by an effective governance and risk management framework.
- A policy framework has been implemented that is complete and clearly communicated, which is supported by processes, guidelines, templates and staff training and awareness. Adherence to the policy framework is monitored.
- An internal control framework is in place to guide the research and licensing agreement management process. Key controls are appropriately designed and operating effectively.
- IT systems provide support and audit trail functionality for research and licensing agreements activities.
- Reporting has been established for research and licensing agreements and reporting provides the appropriate insight to staff and senior management for decision-making
Appendix B – management action plan
| Definition of priority of recommendations | |
|---|---|
| Short-term | Implementation is recommended within 6 months to reduce the risk of potential high likelihood and/or high impact events that may adversely affect the integrity of the NRC's governance, risk management and control processes. |
| Medium-term | Implementation is recommended within 1 year to reduce the risk of potential events that may adversely affect the integrity of the NRC's governance, risk management and control processes. |
| Long-term | Implementation is recommended within 2 years to reduce the risk of potential events that may adversely affect the integrity of the NRC's governance, risk management and control processes. |
| Recommendation | Corrective management action plan | Expected implementation date and responsible NRC contact |
|---|---|---|
|
|
Contact: VP, Business and Professional Services |
|
NPBS shall develop a directive that sets out procedures to ensure that client agreements are developed and negotiated in a disciplined manner. The procedures will be made available to all staff who are involved in client agreement development and negotiation, and training will be provided. |
Date: timeline TBC by March 31, 2020 Contact: VP, Business and Professional Services |
|
A policy compliance process will be developed that includes both escalation processes (with adherence to the NRC's delegation of financial authority) and monitoring/ reporting processes. Reporting shall be standardized and include both exception and non-compliance reporting. This structure will be defined within a risk tolerance matrix described in item 4 below. |
Date: timeline TBC by March 31, 2020 Contact: VP, Business and Professional Services |
|
A risk tolerance matrix will be developed in consultation with the research centre DGs and VPs. This matrix will describe key risks in client agreements and will set out responsibilities and accountabilities for making risk decisions. This risk matrix will be applied NRC-wide. |
Date: timeline TBC by March 31, 2020 Contact: VP, Business and Professional Services |
|
NPBS will do the following:
|
Date: timeline TBC by March 31, 2020 Contact: VP, Business and Professional Services |