Audit of major capital project management - August 2018

 

Alternative format

PDF version (546 KB)

Table of contents

Acronyms and Abbreviations

AMB Asset Management Boards
ASPM Administrative Services and Property Management
CBI Centres, Branches, and Industrial Research Assistance Program
CS Corporate Services
CS PMO Corporate Services Project Management Office
DBB Design-Bid-Build
FAA Financial Administration Act
FB Finance Branch
IMC Investment Management Committee
IPRG Investment Project Reference Guide
IRAP Industrial Research Assistance Program
KITS Knowledge and Information Technology Services
NRC National Research Council Canada
OPMCA Organizational Project Management Capacity Assessment
PC Project Charter
PCRA Project Complexity and Risk Assessment
PMP Project Management Plan
PRS Planning and Reporting Services
RMF Risk Management Framework
RPV Real Property and Vehicles
SAP PS SAP Project System
ScE Scientific Equipment and Laboratories
SE Strategic Enabler
SEC Senior Executive Committee
TB Treasury Board of Canada
TBS Treasury Board Secretariat

Executive Summary and Conclusion

Background

In 2016-17, the National Research Council of Canada (NRC) had an operating budget exceeding one billion dollars of which approximately $151M represented planned in-year major capital expenditures. An effective and integrated major capital project management framework that applies across NRC strengthens the likelihood that NRC's major capital projects are planned and managed successfully.

The Audit of Major Capital Project Management was approved by the President following the recommendation of the Departmental Audit Committee as part of the NRC 2017-18 to 2019-20 Risk-Based Internal Audit Plan.

Audit Objective

The objective of the audit was to provide assurance that NRC's management framework and internal controls are adequate to support the delivery of major capital projects. Specifically, the audit examined governance structures in place to support major capital project accountability, processes in place to plan and execute major capital projects, and major capital project oversight and monitoring processes.

Scope

The audit scope included NRC's overall major capital project management framework (the Framework) and applicable internal controls. The audit assessed the application of the Framework across a sample of 34 major capital projects. The audit reviewed the governance bodies and business processes in place between April 2015 and August 2017.

The audit scope excluded general budgeting or financial forecasting methodologies. It did not assess investment planning processes, minor capital project management, or management rationale for project selection. The audit examined the use of SAP Project System (SAP PS) for major capital projects (excluding an assessment of information technology general controls (ITGC) for the purposes of the Policy on Internal Control).

Strengths

In general, building blocks of a management framework are in place for major capital project management at NRC. Strengths identified as part of the audit include:

  • Governance structures exist and continue to evolve with respect to major capital project planning and oversight
  • Policy instruments are in place and aligned with Treasury Board requirements under the Policy on Management of Projects and its related instruments
  • Processes and practices have been adapted to strengthen stakeholder input in major capital project planning and development, as well as monitoring
  • Data in support of information for senior management decision-making is generally available
  • A corporate project management office for major capital projects exists to support continuous improvement of major capital project management (especially in the areas of cost estimation and risk management)

Areas for Improvement

Key opportunities to improve and strengthen NRC's major capital project management framework include:

  • Defining and communicating a consistent major capital project management governance model that identifies project-related roles, responsibilities, accountabilities, ongoing monitoring and oversight functions and activities, and project performance expectations
  • Improving consistency and bringing clarity to processes and practices through up-to-date policy and guidance instruments
  • Defining and communicating a consistent approach for project cost estimation, including the identification of all planning costs and the use of feasibility studies where appropriate
  • Strengthening project oversight and control with timely project performance information that reflects the needs of management and stakeholders
  • Updating guidance in key project planning and execution areas to capture and reflect lessons learned and industry best practices
  • Reviewing and assessing project outcomes relative to plans

Recommendations

The Vice-President, Corporate Services and Chief Financial Officer, should:

  1. Clarify the major capital project management governance model by defining and communicating:
    1. An organization-wide major capital project management model with associated project decision-making authorities, accountabilities, roles and responsibilities, and project related performance expectations. [Priority: High]
    2. Major capital oversight structures (corporate and Division-level and below) and their associated mandates. [Priority: High]
    3. Major capital project management performance standards and expectations. [Priority: High]
  2. Introduce additional required building blocks to strengthen major capital project management by:
    1. Updating the framework to align requirements for project level planning and oversight structures and related tools based on project scope, complexity, and risk. [Priority: High]
    2. Providing guidance on project delivery strategies and their associated risks and benefits to match desired project outcomes with project constraints. [Priority: Moderate]
    3. Defining a consistent approach to major capital project cost estimation, incorporating the identification of all applicable planning costs and the use of feasibility studies, which aligns with project lifecycle stage gates and cost estimate maturity standards. [Priority: Moderate]
  3. Ensure that project close out reports are completed in a timely fashion and project lessons learned are consistently collected and shared with the major capital project community in support of continuous improvement. [Priority: Moderate]
  4. Refine the project oversight process by:
    1. Clarifying expectations of the project team and senior management for information required to support oversight, including non-financial metrics. [Priority: Moderate]
    2. Linking project financials with applicable project deliverables and/or milestones to improve project financial forecasting, objectivity of project reporting, and enable project progress assessment at any point within the project lifecycle. [Priority: High]
  5. Define and implement a consistent process to assess major capital project results and deliverables against planned objectives, outcomes and plans. [Priority: Moderate]

Audit Opinion and Conclusion

In my opinion as Chief Audit and Evaluation Executive, improvements to NRC's current major capital project management framework and internal controls are required to support the delivery of major capital projects. NRC should strengthen its management practices through various ongoing improvements and risk-based considerations, as set out in this audit report.

Statement of Conformance

In my professional judgement as Chief Audit Executive, the audit was conducted in conformance with the Institute on Internal Auditors' International Standards for the Professional Practice of Internal Auditing and the Government of Canada Policy on Internal Audit as supported by the results of the quality assurance and improvement program.

Alexandra Dagger, CIA, Chief Audit Executive

 

Acknowledgements

The audit team would like to thank those who collaborated in this effort to highlight NRC's strengths and opportunities for improvement as they relate to this project.

1.0 Introduction

In 2016-17, the National Research Council of Canada (NRC) had an operating budget exceeding one billion dollars of which approximately $151MFootnote 1 represented planned in-year major capital expenditures. A comprehensive approach to managing major capital projects that is appropriate for the level of project risk and complexity supports the realization of NRC's project outcomes within identified scope, timelines, and budgets. It also supports NRC's senior management in ensuring sound stewardship, accountability, and risk management on a project of any scale, duration, or complexity.Footnote 2

Project management is about improving the likelihood of success of time-limited initiatives by applying certain practices. These practices include establishing clear accountabilities, defining objectives and outcomes, establishing the scope, planning, monitoring, and reporting controls for project activitiesFootnote 3. The Audit of Major Capital Project Management was approved by the President following the recommendation of the Departmental Audit Committee as part of the NRC 2017-18 to 2019-20 Risk-Based Internal Audit Plan.

2.0 Background and Context

Under the Treasury Board of Canada (TB) Policy on the Management of Projects, organizations are responsible for ensuring their projects are managed in a manner consistent with assessed levels of complexity and risk. Within the context of this policy:

  • The Organizational Project Management Capacity Assessment (OPMCA) determines organizational capacity to manage projects
  • A Project Complexity and Risk Assessment (PCRA) determines individual project risk and complexity
  • Project expenditure authority differs from the authority to enter into contracts and precedes the authority to commit funds (Financial Administration Act, Section 32)Footnote 4
  • NRC has a Level 2 "Tactical" ratingFootnote 5 under the OPMCA with delegated project expenditure authority for projects with a PCRA score of two or lower

Within NRC's context, major capital projects are valued over $350K and predominately fall within one of the four investment capability categoriesFootnote 6 as noted in Figure 1 below.

Figure 1: NRC investment capability categories defined

Figure 1: NRC investment capability categories defined. Long description follows.
Long description of Figure 1: NRC investment capability categories defined

Figure 1 describes the four NRC investment capability categories used to classify NRC's major capital investments: scientific equipment and laboratories; real property and vehicles; information management and information technology; and strategic enablers (internal business improvement initiatives).

In 2016-17, NRC planned to undertake 92 major capital projects, representing a combination of new and unfinished projects from previous fiscal years. Planned major capital expenditures exceeded $150M. Figure 2 shows projects by phase at FY2017 year-end.

Figure 2: Projects by phase at FY-End 2017

Figure 2: Projects by phase at FY-End 2017. Long description follows.
Long description of Figure 2: Projects by phase at FY-End 2017

Figure 2 provides a breakdown of the proportion of projects across NRC's project phases. In fiscal year 2017, NRC planned to undertake work on 92 major capital projects. At the closure of the fiscal year, 59% of projects were in execution phase; 27% of planned projects were in planning phase; 12% of planned projects were closed; and 2% of planned projects were cancelled.

 

Figure 3 shows NRC's budgeted-to-actual spending on major capital projects under the departmental investment plan.

Figure 3: Major capital budget-to-actuals across fiscal years

Figure 3: Major capital budget-to-actuals across fiscal years. Long description follows.
Long description of Figure 3: Major capital budget-to-actuals across fiscal yearsFootnote 7

Figure 3 provides a comparison of budgeted to actual spending from fiscal year 2015 to fiscal year 2017. The figure identifies a consistent gap between actual spending and planned major capital spending.

Challenges in delivering major capital projects on budget, schedule, and scope, as shown in the gap between budgets and actual expenditures, led in part, to the decision to undertake this audit.

3.0 Audit Objective and Scope

Objective

The objective of this audit was to provide assurance that NRC's management framework and internal controls are adequate to support the delivery of major capital projects. Specifically, the audit examined:

  • Governance structures in place to support major capital project accountability
  • Processes in place to plan and execute major capital projects
  • Major capital project oversight and monitoring processes

Scope

Within the context of this audit, major capital projects are defined as initiatives that comprise any combination of capital, labour, and/or operations and maintenance funding. Figure 4 below depicts the scope of this audit.

Figure 4 – Audit Scope – NRC Project Management Framework (PMF) May 2015

Figure 4 – Audit Scope – NRC Project Management Framework (PMF) May 2015. Long description follows.
Long description of Figure 4 – Audit Scope – NRC Project Management Framework (PMF) May 2015

Figure 4 describes the scope of the audit across NRC's defined project stages according to the NRC's Project Management Framework v2015. The figure provides a description of NRC's four defined project phases, the steps within each phase, and the key documents applicable to each step.

The scope of the audit included NRC's overall major capital project management framework (the Framework) and applicable internal controls at the corporate and research centre levelsFootnote 8. The audit focused on activities and the processes, procedures, tools, and templates provided by Planning and Reporting Services (PRS) to Research Centres, Branches, and Industrial Research Assistance Program (CBIs). The audit reviewed a sample of 34 major capital projects across NRC to assess the application of the Framework. The audit assessed the governance bodies and business processes in place between April 2015 and August 2017.

Table 1: Projects sampled according to NRC PMF project stage
Sampled Projects by TypeFootnote 9 Approved Project Charter
(Stage 1 Approval)
Approved Project Management Plan
(Stage 2 Approval)
Scientific equipment and laboratories 22 16
Real property and vehicles 9 8
Information management / Technology 1 0
Strategic enabler 2 2
Total Projects Sampled 34 26

In October 2017, the NRC investment plan management team and major capital project delivery team transitioned to Finance Branch (FB) and the Corporate Services Project Management Office (CS PMO) respectively.

The audit scope excluded general budgeting and financial forecasting methodologies. The audit touched upon elements of investment planning and minor capital project management to the extent that the topics influenced major capital project management. It did not assess investment planning processes or minor capital project management. The audit touched upon project selection to the extent that information is available to support decision-making but did not assess management rationale for project selection. The audit examined the use of SAP Project System (SAP PS), NRC's enterprise project management system, for major capital projects. Audit procedures excluded an assessment of information technology general controls (ITGC) for the purposes of the Policy on Internal Control.

4.0 Audit Findings and Recommendations

NRC's framework for major capital projects was assessed through three lines of enquiry: governance (Section 4.1); processes for planning and execution (Section 4.2); and project oversight and monitoring (Section 4.3). Each section below provides a summary of findings supported by detailed observations, a description of the risk(s) and impact(s) of associated findings, and recommendations to address areas for improvement.

4.1 Major capital project management governance

Summary Finding

NRC lacks a defined and understood major capital project management governance model. While governance bodies exist and continue to evolve, there is a lack of clarity over decision-making, accountabilities, responsibilities, and expectations across major capital project process participants and stakeholders. Insufficient challenge of planning activities and inconsistent ongoing project oversight hampered opportunities to proactively manage risks or take corrective action.

There is a need for clarification and reinforcement of roles, responsibilities, accountabilities, and expectations of NRC's various governance bodies involved in major capital project management.

NRC generally has the policies and instruments in place to support major capital project management. However, these need to be updated, consolidated, and communicated to ensure that NRC is implementing an effective and integrated major capital project management framework.

Observations

4.1.1 Governance structures, roles, responsibilities, and accountabilities

We examined whether NRC has governance bodies in place, with defined roles, responsibilities, and accountabilities, to support clarity of decision-making authorities, control of resources, and strategic and operational alignment for project planning and delivery. We found that governance bodies exist to oversee major capital projects. However, there is a need to clarify roles, responsibilities, accountabilities, and performance related expectations across the governance bodies.

The following entities comprise NRC's corporate major capital governance bodies:

  • Senior Executive Committee (SEC) – approves departmental investment plan (list of major capital projects)
  • Investment Management Committee (IMC) [reconstituted in FY2017] – reviews, prioritizes, and recommends projects (NRC-wide) for the NRC investment plan to SEC
  • Asset Management Boards (AMB) [dis-banded in FY2017 with duties absorbed by IMC] – reviewed, ranked, and endorsed projects within asset categories (Figure 1) for IMC review

Division-level and below governance bodies include:

  • Division Vice-President's Office – prioritizes and proposes projects for the investment plan and provides ongoing management of approved projects
  • Project steering committees (within or across CBIs) – formal and informal entities to provide project-level oversight, guidance, and direction
  • Project Sponsor – accountable for project success and realization of project business case outcomes

Corporate major capital governance bodies have defined and communicated terms of reference that describe their roles, responsibilities, and authorities. For example, the IMC terms of reference includes provisions to monitor adherence to approved project plans and to take corrective action when projects are off-track. We found corporate governance bodies were active in the review and prioritization of projects for inclusion on NRC's Investment Plan. However, we noted that governance bodies are not exercising their mandates to provide ongoing oversight of project progress and the need for corrective action.

NRC's major capital project management framework does not identify the governance bodies responsible for ongoing project monitoring and oversight. In general, we found that high complexity and/or risk projects lack project-level oversight bodies to support monitoring and control. Unclear roles and responsibilities have created a gap with respect to ongoing project monitoring. As discussed further in Section 4.3, the lack of ongoing and effective stakeholder involvement, oversight of plans and progress, and monitoring of resources adversely affected project results (Table 3).

We noted that in FY2017, changes to governance bodies and major capital process owners have put new controls in place intended to increase strategic alignment, project stakeholder involvement, and proactive planning and risk management of major capital projects. From a governance perspective, NRC's investment plan management team provides IMC with information for decision-making. Their monitoring and control roles require clarification to provide senior management with timely information when projects are not being implemented as planned.

4.1.2 Policies and supporting instruments

We examined whether NRC had up-to-date policies and related instruments to guide and support the implementation of major capital project management. We found that existing policy instruments require updating, streamlining, and clear communication across NRC to clarify accountabilities, roles, responsibilities, and expectations.

The following NRC policy instruments represent the key documents to implement major capital project management governance:

  • NRC Project Approval Authority Directive
    • NRC Project Management Framework (January 2011) [Framework 2011]
      • Guide to Costing of Investment Projects
    • NRC Project Complexity Risk Assessment (PCRA) Guide
    • NRC's Project Management Framework v1.0 (May 2015) [Framework 2015]
      • [Draft] Investment Project Reference Guide (IPRG)

The NRC Project Approval Authority Directive defines NRC's authorities to undertake projects in accordance with the TB Policy on the Management of Projects and its related instruments.

We found that NRC has two authoritative policy instruments in place related to major capital project management. Table 2 provides a summary of the similarities and differences of the instruments.

Table 2: Comparison of Framework 2011 and Framework 2015
Framework 2011 Framework 2015
  • Description of major capital related governance bodies (generally focused on investment planning aspects – project concept review, endorsement, and prioritization process)
  • Description of investment plan (major capital) process and related tools and templates
  • Does not reflect updates to related processes (increase in major capital threshold, project approval and expenditure processes/timelines, organizational changes, lessons learned)
  • Designed to support implementation of NRC Investment Plan
  • High-level stewardship, accountability, and risk management principles for consistency in major capital project management
  • Major capital focused costing principles based on project stage gates
  • Available on NRC intranet (widely accessible)
  • Over-arching high-level overview of NRC (general) project management principles
  • Costing principles focused on labour components for program-related (revenue generating) projects
  • Defined roles, responsibilities, and project-related controls across management levels
  • Not available on NRC intranet (not easily accessible to NRC community)

We found that neither Framework 2011 nor Framework 2015 provides adequate guidance related to major capital project assessment for project viability or performance expectations. We also found that the instruments do not identify specific responsibilities or provide guidance on problem indicators when a project is off-track.

The lack of a single authoritative document for major capital related project management is hindering effective and integrated major capital project management at NRC. As well, we found that Framework 2015 refers to an Investment Project Reference Guide (IPRG)Footnote 10 that was not available to the NRC community at the time of the audit. The lack of consistent, up-to-date, guidance has adversely impacted project planning and cost estimate development as illustrated in 17 of 34 projects examined (Table 3).

Major Capital Project Management Governance Risk and Impact

There is a risk that the lack of clarity around accountabilities, roles, responsibilities, and project performance expectations will continue to hinder effective project oversight and direction for NRC's major capital projects.

Recommendation 1

The Vice-President, Corporate Services and Chief Financial Officer, should clarify the major capital project management governance model by defining and communicating:

  1. An organization-wide major capital project management model with associated project decision-making authorities, accountabilities, roles and responsibilities, and project related performance expectations. [Priority: High]
  2. Major capital oversight structures (corporate and Division-level and below) and their associated mandates. [Priority: High]
  3. Major capital project management performance standards and expectations. [Priority: High]

4.2 Processes for effective planning and project execution

Summary Finding

Some foundational processes are in place to support major capital planning and execution.

Challenges in contingency definition, cost estimate development, risk management, and execution strategy, have resulted in delayed, off-scope, and/or over-budget projects. Inconsistent approaches and interpretations of processes stemming from the lack of a single source of guidance, hampered project teams across planning and execution phases. Best practices and lessons learned are not being shared within the community to drive improvement.

Targeted guidance is required to better support the development of information related to project estimation, contingency planning, and risk management.

Observations

4.2.1 Management models, processes, practices, and tools

We examined whether NRC had defined processes, practices, and tools to support project planning and execution to facilitate achievement of project objectives and outcomes. We found that processes are generally in place to support major capital planning and execution.

Thirty-four capital projects were analyzed for project issues adversely impacting scope, schedule, or budgetFootnote 11. These issues were classified across seven root-cause categories (Table 3). Projects may have experienced more than one issue throughout their lifecycle. Further details are discussed following the table.

Table 3:Project Delivery Strategies Root cause categories with associated examples
  Root causes by Project CategoryFootnote 12Footnote 13 Impact  
Root Cause Category and Sampled Projects Impacted (n=34) RPV ScE SE Over Budget Delayed Reduced Scope ExamplesFootnote 14
1. Insufficient planning / estimate development 1 17 -     Some building sections (interior) left incomplete
1. Insufficient planning / estimate development 1 17 -     Vendor lead time for production not sufficiently factored into schedule
1. Insufficient planning / estimate development 1 17 -     Insufficient contingency in budget and planning cost considerations
2. Mid-project scope change 1 6 2     Change in technical requirements
3. External influence 1 11 -     Lack of access to technical planning documents to support procurement activities
3. External influence 1 11 -     Lowest submitted bid exceeded cost estimates (i.e. market labour shortage)
4. Third-party issues / contractor delays 3 8 -     Lack of central purchasing agent staff to undertake procurement activities
4. Third-party issues / contractor delays 3 8 -     NRC work is contingent on third-party activities
5. Lack of stakeholder involvement - 7 1     Unclear ownership to advance project
5. Lack of stakeholder involvement - 7 1     Large number of stakeholders whose input is required in scope definition
6. Insufficient resourcing - 8 1     Insufficient internal capacity to deliver
6. Insufficient resourcing - 8 1     No project manager available
6. Insufficient resourcing - 8 1     Project team unable to commit time due to operational priorities
7. Missed deadline / milestone (e.g., completion of negotiations) 2 8 -     Inability to settle negotiations on legal requirements with project stakeholder

Vice-Presidents and Directors General are responsible for providing ongoing oversight and supervision of their respective major capital projects. We observed that NRC has a number of major capital project management models:

  • Part-time project managers (researchers and/or CBI project management office staff)
  • Part-time project managers, at the CBI level, who are also responsible for (major and minor) capital planning and asset management activities
  • Full-time project managers from the CS PMO

The various management models introduce planning and resourcing challenges, as noted in Table 3, including:

  • Insufficient capacity to deliver projects while also performing their regular duties
  • Higher priority for operational commitments (e.g. revenue generating projects)
  • Project team availability when projects have multiple stakeholders
  • The capacity of NRC common services to support CBI project needs
  • For Administrative Services and Property Management (ASPM), balancing internal service delivery with contracting-out
  • For the CS PMO in particular, the capacity to deliver numerous projects while supporting corporate initiatives (i.e. the development of the major capital project management community)

Major capital project charters (PC) and Project Management Plans (PMP) generally identified individual project roles, responsibilities, accountabilities, and stakeholders. However, across the 34 sampled projects, the project sponsor was identified as responsible for the project in 23 instances, as opposed to being accountable for project success. This is contrary to Framework 2015 principles and major capital project management best practices, highlighting a lack of clarity between project responsibility and accountability and the risk of inadequate control and oversight.

In FY2016, the investment plan management team introduced a simplified major capital project management process based on project complexity and risk. This is a best practice to ensure oversight and control match project complexity (scope), schedule, and budget. NRC policy instruments were not updated to reflect the inclusion of this simplified process, increasing the likelihood of misunderstanding of framework requirements and an inconsistent application of tools, templates, and framework principles.

Lessons learned were not shared with the broader major capital project management community. No forum exists for the major capital project management community to share best practices, and/or expertise. The sharing of lessons learned and best practices would support better planning and cost estimation throughout the major capital project lifecycle. For example, the consideration of labour constraints in risk management plans, the inclusion of different "soft costs"Footnote 15 as part of cost estimate development, and evolving planning requirements, and associated timelines in the National Capital Region would all support better project planning.

We found a lack of guidance to support project managers in the identification of project delivery approaches or construction methods, based on operating constraints. Across sampled projects, project delivery strategies were not defined to demonstrate the associated risks and opportunities of selected approaches. For example, while not an NRC requirement, the projects sampled generally made use of the Design-Bid-Build (DBB)Footnote 16 model. According to the Construction Management Association of America (CMAA), the DBB model is not consistently appropriate for projects where design elements are not mature due to the creation of completely new capabilities or schedule constraints resulting from government budgeting requirements. In general, project delivery strategies fall into the following categories defined in Table 4 with variations based on project constraints.

Table 4: Project Delivery StrategiesFootnote 17
  Design-Build Design-Bid-Build Construction Management
Description Single entity provides architectural, engineering design, and construction performance services Separate contracts with a designer and a contractor generally leading to a fix-price contract Construction manager advises during planning phase and acts as general contractor during construction phase (execution phase) within defined schedule and pricing parameters
Risk and Control Lower Risk to NRC Control by NRC Higher
Higher Risk to Contractor Control by Contractor Lower
4.2.2 Management models Processes for resources assumptions, estimates and stakeholders' engagement

We examined whether NRC had defined processes to support major capital project teams in the development of reasonable and mature project cost estimates relative to the project stage that reflected stakeholder input. We found roles and responsibilities for cost estimate development are not defined and there is no common NRC methodology reflective of best practices. Guidelines require clarification and improvement to ensure that project budgets consider all reasonable costs, contingency standards are applied, and assumptions reflect lessons learned. Engagement of stakeholders earlier in project concept development is required to support more mature project definition and cost estimates.

We found that project teams were not consistently considering common service requirements during the project planning process. For example, in two sampled projects, the lack of stakeholder input during the planning phase resulted in coordination issues developing during project execution. The NRC investment plan management team made efforts to improve stakeholder engagement through additional guidance materials and sign-offs on plans.

NRC does not have up-to-date project-costing guidelines that define a common NRC methodology with related roles, responsibilities, and accountabilities, and reflects lessons learned. For example, one project experienced a funding shortfall when the local airport authority required a work permit fee. The lack of complete planning, sufficient budget contingency, and understanding of applicable regulatory requirements for the work site had adverse impacts to the project and management perception of performance.

We found that Framework 2011 includes major capital project contingency guidelines based on a project's assessed PCRA level, but these are not consistently being applied. Twenty of the 26 sampled projects with an approved PMP defined financial contingency within the approved project budget. Seven of these 20 projects examined with defined contingencies had quantified (i.e. specified dollar amount) contingency amounts. The remaining 13 projects only identified the existence of a contingency built into the overall budget. This prevented any oversight and challenge of cost estimate assumptions.

Insufficient planning has resulted in projects being delivered late, over-budget, and or with reduced scope as noted in 17 of the 34 projects examined. As well, interviews noted a lack of consideration for "soft costs" and the benefits of feasibility studies to inform cost estimates. We noted instances of project teams unwilling to use funding on project definition activities due to the risk of concept rejection, (i.e. funding is "wasted" when the project is not approved). Guidance materials defined the purpose of feasibility studies but did not identify the associated benefits. No decision criteria exist to require or recommend feasibility studies, especially for high-complexity and/or risk projects. NRC guidance aligns cost estimate standards with project stage gates but it does not reflect best practices, incorporate lessons learned, or assign responsibility for estimate verification and integrity.

NRC's investment plan management team defined guidelines for budget maturity and project contingency to address some of the aforementioned concerns in FY2017. The guidelines were not yet generally available to the NRC community at-large as processes have been in a state of change. Opportunities for improvement exist with respect to aligning project stage gates with project cost estimate maturity, contingency, and scope definition standards to enable consistent management decision-making.

As noted in Table 3, nine of the 34 major capital projects examined experienced resourcing issues, including five that had specific human resource constraints. No mitigating resource management strategies were identified or implemented for these projects. The lack of mechanisms to ensure projects are adequately resourced (with project management and technical expertise, and facility or equipment availability) hindered NRC's ability to deliver projects. However, the NRC investment plan management team took action to increase project planning maturity and mitigate overly optimistic project assumptions. This was done by implementing stakeholder sign-offs on plans, as well as quality assurance reviews of project submissions throughout the project lifecycle.

4.2.3 Project Changes, Risks and Complexities

We examined whether NRC had defined processes to manage project changes and risks relative to their complexity to ensure adequate oversight and control. We found that NRC major capital project risk management activities are based on NRC's Risk Management Framework (RMF).

Planning templates incorporate NRC RMF approaches and guidelines to support a common understanding for managing risk. However, we identified inconsistent application of NRC's RMF. Specifically, we observed project risks that were predominately qualitative and generic in nature and focused on planning stage issues. Defined risks also lacked triggers or responsibilities for mitigation. Across 34 projects examined, 23 identified a consistent approach to risk tracking and management. The lack of a consistent approach across all major capital projects to manage risk throughout the project lifecycle increases the likelihood of project failure from unmitigated risks and precludes trend analysis for cross-NRC lessons learned.

We noted that tools and templates are generally available and used to manage project changes. All sampled projects are managing change orders according to framework principles although guidance on change order authorities is unclear.

NRC is completing PCRAs according to NRC and TBS guidance. We found that NRC has not exceeded its project expenditure authorities and processes exist to escalate projects to TBS when higher authorities are required. In 25 of the 26 sampled projects with approved PMPs, we found that PCRA scores were reflective of project scope, cost, schedule, and plan details. The NRC PCRA Guide requires evidence to support each question. Projects did not consistently demonstrate adherence to Guide instructions. For example, 11 of the 34 sampled projects did not reference supporting information to justify their scoring.

Processes for Effective Planning and Project Execution Risk and Impact

There is a risk that inadequate guidance, direction, and up-to-date tools, will continue to contribute to an inconsistent application of defined processes that impact project planning and execution, as well as information for decision-making.

There is also a risk that inconsistent approaches to major capital project cost estimation will result in inadequate budgeting and resource allocation, impacting project planning and decision-making.

Recommendation 2

The Vice-President, Corporate Services and Chief Financial Officer should introduce additional required building blocks to strengthen major capital project management by:

  1. Updating the framework to align requirements for project level planning and oversight structures and related tools based on project scope, complexity, and risk. [Priority: High]
  2. Providing guidance on project delivery strategies and their associated risks and benefits to match desired project outcomes with project constraints. [Priority: Moderate]
  3. Defining a consistent approach to major capital project cost estimation, incorporating the identification of all applicable planning costs and the use of feasibility studies, which aligns with project lifecycle stage gates and cost estimate maturity standards. [Priority: Moderate]

Recommendation 3

The Vice-President, Corporate Services and Chief Financial Officer should ensure that project close out reports are completed in a timely fashion and that project lessons learned are consistently collected and shared with the major capital project community to support continuous improvement. [Priority: Moderate]

4.3 Effective project monitoring and oversight

Summary Finding

Project monitoring and oversight processes are in place but are not being used effectively to support decision-making or exercise control of projects. While data and reporting capabilities exist, management is not receiving the information necessary to effectively assess risks and to make informed decisions.

Project benefits are defined and documented. Project outcomes are not revisited as schedule, budget, or scope change, and are not assessed after project completion, precluding a clear demonstration of stewardship of taxpayer funds or opportunities for lessons learned.

Observations

4.3.1 Information for project monitoring and decision-making

We examined whether complete and up-to-date project schedule, budget, and scope information was available and accessible to project team members, management, and stakeholders for monitoring and decision-making purposes. Reporting activities are not providing senior management with sufficient (in quality) information to identify issues for management's attention.

Senior management and project teams both expressed concerns with reporting effectiveness for monitoring and decision-making purposes. While we observed best practice examples of tailored reporting for project sponsor needs, interviews with senior management noted the absence of information for oversight. Interviews with project managers identified a lack of direction with respect to reporting needs and expectations.

Finance Branch comptrollers support management oversight with general monthly financial reports that include major capital spending and provide financial expertise. Interviews noted that comptrollers, as the key financial resource to CBIs, are not consistently involved or engaged in supporting major capital project oversight. Only one project in our sample expressly identified a FB representative as a project stakeholder within the project governance structure highlighting inadequate consideration for cross-functional expertise at NRC when developing project plans.

In FY2017, the investment plan management team introduced a quality assurance process for key major capital project documentation, including the PC, PMP, and change requests, to increase the consistency of submission documentation. Project management and financial experts jointly review submitted documentation and maintain an action and response log to bring visibility to issues and considerations raised. The log provides an overview of actions taken by the project team to reinforce project responsibilities and accountabilities. As of the conclusion of the audit, log information has not been used to support cross-project trend analyses for continuous improvement.

Project financial forecasting is independent of deliverables or milestones. This limits the ability to assess project progress beyond a comparison of expenditures-to-date. As well, projects examined did not consistently demonstrate the use of milestones, deliverables, or critical success factors, to support management oversight of project progress, (i.e. missed milestones or unrealized deliverables did not always trigger increased oversight or remedial action). We found that opportunities exist to better integrate project scope with scheduling and cost elements to provide management and stakeholders with a fulsome perspective of project progress. Integration of project scope, schedule, and cost elements would support project forecasting through the development of project performance trends and enable NRC to optimize its major capital management reserve (i.e. minimize unspent funds).

Across 26 projects examined with approved PMPs, 24 had sufficiently detailed and tangible milestones to enable project monitoring. That said, quarterly status reports are inconsistently reporting on milestones and not facilitating the identification of project issues and oversight. Specifically, reported milestones did not always reflect those defined in the PMP nor did target dates match PMP targets or updated schedules. While we found that project risks were defined in each quarterly status report, these are not being consistently reported on until after they are addressed, realized, or expire. In addition, updates and changes to plans were being captured across different project documents thus increasing the risk of using dated information for reporting purposes. Guidance on status report requirements to address aforementioned issues was not defined in either framework.

While capabilities exist in SAP PS to manage milestones with assigned degrees of project completion, NRC is not using SAP PS milestone capabilities for reporting purposes. For example, across 33 of the 34 sampled projects created in SAP PS, 22 were using milestone capabilities. These 22 projects defined 336 milestones of which 215 were past due an average of 340 days at the time the audit was completed, demonstrating inconsistent use of system capabilities.

Project Monitoring and Oversight Risk and Impact

There is a risk that the absence of timely and sufficient information will continue to hinder monitoring and oversight to challenge forecasts and assumptions, and proactively manage project risks and issues.

Recommendation 4

The Vice-President, Corporate Services and Chief Financial Officer should refine the project oversight process by:

  1. Clarifying expectations of the project team and senior management for information required to support oversight, including non-financial metrics [Priority: Moderate]
  2. Linking project financials with applicable project deliverables and/or milestones to improve project financial forecasting, objectivity of project reporting, and enable project progress assessment at any point within the project lifecycle. [Priority: High]
4.3.2 Project risk monitoring and reporting processes

We examined whether processes were defined to support ongoing corporate and project-level risk monitoring and reporting throughout the project lifecycle. In general, we found project risks defined in the PC and PMP reflected planning stage concerns, and were generic in nature.

Framework 2011 and 2015 delegate risk management practices to project managers. Planning and execution phase templates provide general structure to identify and report on project risks. In 19 of the 26 sampled projects with approved PMPs, we found that risk management plans identified a consistent approach, with a risk register or log, to help manage project risks. Communication of project risks in quarterly status reports were inconsistent precluding a coherent view of emerging and ongoing risks. It was also unclear how identified issues and risks were used to support corporate oversight. As illustrated in Table 3 above, insufficient and immature project risk management (in terms of cost and schedule assumptions, resourcing availability, and stakeholder involvement) resulted in poor project results.

4.3.3 Project benefits and outcomes

We examined whether NRC had defined processes to assess project impacts, outcomes, and benefits. We found the quantitative measures being used did not include suitable metrics to demonstrate project strategic or operational value.

No formal structures or processes existed to undertake post-project benefits analysis. NRC does not revisit planned outcomes at project conclusion for lessons learned in plan development. None of the sampled projects that were complete had assessed project deliverables against business case benefits. For example, a new research asset requires a costing exercise to determine an activity rate for internal financial reporting and asset utilization measurement. Four of the seven completed projects with revenue generating assets did not include a cost review as part of their project transition plans and consequently did not update cost rates in NRC's financial system. Not reviewing facility and equipment cost rates after new assets are added hinders the ability to realize planned project revenue targets impacting internal financial reporting and performance metrics (i.e. asset utilization).

We found that project outcomes were not being consistently updated with revised budget, schedule, and scope information as project plans mature (i.e. from PC to PMP). Specifically, in six of the11 projects examined for revenue generating assets, project benefits were not updated between the PC and PMP to reflect more mature cost estimates. Not updating expected project outcomes as plans mature limits the opportunity to adjust project resourcing, including project termination, based on project business value.

NRC has defined financial measures to demonstrate project benefits. However, these measures were not consistently applicable to projects for lifecycle management or safety and health. We found that the lack of fit-for-purpose metrics prevents quantitatively supported decision-making and project selection. Aligning project benefits metrics tailored to NRC's research capability investment categories would support decision-making with financial and non-financial metrics.

Project Benefits and Outcomes Risk and Impact

There is a risk that without formal processes to assess benefits and outcomes to plans, NRC will continue to be unable to demonstrate investment impacts and value to taxpayers in a timely and effective manner.

Recommendation 5

The Vice-President, Corporate Services and Chief Financial Officer should define and implement a consistent process to assess major capital project results and deliverables against planned objectives and outcomes. [Priority: Moderate]

Appendix A: About the Audit

Approach and Methodology

The audit was conducted in accordance with generally accepted professional auditing standards of the Institute of Internal Auditors (the IIA) and the standards and requirements set out in the Treasury Board Policy on Internal Audit. A risk assessment was conducted to define the audit objective, scope and criteria.

The audit criteria were primarily derived from the TB Policy on the Management of Projects with consideration toTBS Management Accountability Framework areas of management and Office of the Comptroller General's (OCG) Audit Criteria related to the Management Accountability Framework: A Tool for Internal Auditors (2011). Consideration was also given to related TB and NRC policies, standards, and directives.

The criteria were discussed with management in advance of the audit. Senior management or their delegates were consulted on an on-going basis throughout the audit process. The audit methodology included documentation review and interviews. A sample of 34 major capital projects was drawn from across the organization. Our selection methodology incorporated quantitative and qualitative factors including major capital budgets and expenditures, project lifecycle stage, and other professional judgement informed criteria.

Audit Criteria

Line of Enquiry Audit Criteria
1.0 Governance structures have been designed and implemented to provide and support accountability for major capital projects 1.1 Governance structures exist, with defined roles, responsibilities and accountabilities to support project delivery, and work as intended
1.2 Capital project management policies and supporting instruments are established, communicated, maintained, and reviewed as necessary
2.0 Processes are in place to support effective planning and execution of major capital projects 2.1 Processes, practices, and tools are available to support project planning and execution for the achievement of project objectives and outcomes
2.2 A defined process exists to support the development of reasonable resourcing assumptions, project estimates, and engagement with relevant stakeholders
2.3 Processes exist to identify, assess, and manage project changes and risks relative to their complexity
3.0 Processes are in place to support effective monitoring and oversight of major capital projects 3.1 Accurate and complete information is available to enable project monitoring and decision-making
3.2 Processes exist to monitor and report on project risks throughout the project lifecycle
3.3 Processes are in place to support the identification, assessment, measurement, and reporting of project benefits and outcomes

Appendix B: Management Action Plan

Definition of Priority of Recommendations

High Implementation is recommended within six months to reduce the risk of potential high likelihood and/or high impact events that may adversely affect the integrity of NRC's governance, risk management and control processes.
Moderate Implementation is recommended within one year to reduce the risk of potential events that may adversely affect the integrity of NRC's governance, risk management and control processes.
Low Implementation is recommended within one year to adopt best practices and/or strengthen the integrity of NRC's governance, risk management and control processes.

Recommendation 1

The Vice-President, Corporate Services and Chief Financial Officer, should clarify the major capital project management governance model by defining and communicating:

  1. An organization-wide major capital project management model with associated project decision-making authorities, accountabilities, roles and responsibilities, and project related performance expectations. [Priority: High]
  2. Major capital oversight structures (corporate and Division-level and below) and their associated mandates. [Priority: High]
  3. Major capital project management performance standards and expectations. [Priority: High]

Corrective Management Action Plan

In 2016, the Planning and Reporting Services office underwent an intensive review of the Major Capital Investment Projects framework and associated Stage-Gate templates. Organizational changes and capacity challenges delayed the final approvals. In 2017, Investment planning moved under the VP, CS/CFO resulting in additional minor organizational revisions.

Finance Branch will prioritize a final review and update the framework for the management of Major Capital Investment Projects. This priority activity will include a review of the responsibility matrix and instructions provided in investment project templates.

Finance Branch will also develop a project related Performance Expectation Standard applicable to all stakeholders, including oversight committees.

Finance Branch will develop and deliver targeted information sessions to communicate the individual roles and responsibilities of each project team member.

Expected Implementation Date and Responsible NRC Contact

Completion of the initial review by June 30, 2018 with implementation by March 31, 2019. Updates on the actions and status will be done at each Investment Management Committee Meeting through 2018-19.

Completion of the framework within the division by September 30, 2018 with dissemination to and discussions with divisions through Fall 2018.

Training and information sessions to be developed by September 30, 2018 and delivered by December 31, 2018.

Contact: Vice-President, Corporate Services and Chief Financial Officer

Recommendation 2

The Vice-President, Corporate Services and Chief Financial Officer should introduce additional required building blocks to strengthen major capital project management by:

  1. Updating the framework to align requirements for project level planning and oversight structures and related tools based on project scope, complexity, and risk. [Priority: High]
  2. Providing guidance on project delivery strategies and their associated risks and benefits to match desired project outcomes with project constraints. [Priority: Moderate]
  3. Defining a consistent approach to major capital project cost estimation, incorporating the identification of all applicable planning costs and the use of feasibility studies, which aligns with project lifecycle stage gates and cost estimate maturity standards. [Priority: Moderate]

Corrective Management Action Plan

  1. Finance Branch will update the Major Capital Investment Projects management framework to incorporate a tiered approach to the management of capital investments that aligns to the specific investment scope, complexity and risk. For the management of low complexity, low risk investments, a set of criteria was approved by NRC's Senior Executive Committee, prior to re-organization, and will be used to pilot a Major Capital Investment in 2018-19. Finance Branch will further outline a second criteria based tier for medium complexity and risk investment projects to further advantage alignment of governance and resources to project complexities and risks.
  2. Finance Branch will outline a priority activity to leverage project delivery strategies available from TB website specific to Construction projects. Finance Branch will incorporate models into the Major Capital Investment Projects governance tools, and provide education though communication channels.
  3. Last year, NRC rolled out a new approach for funding Major Capital Investment Projects that granted planning fund approval to carry out necessary pre-planning project activities, such as feasibility studies. In addition, Finance Branch has recently undertaken an external review to identify opportunities to improve its current project lifecycle. The inclusion of pre-defined project planning activities (feasibility study) and increased gate reviews will allow for improved cost estimations and earlier Capital investment decisions. Finance Branch will continue to interact with OGD's and Center's of Excellence to further refine a consistent approach to Major Capital Investment Project cost estimation.

Expected Implementation Date and Responsible NRC Contact

  1. Implement the low complexity and risk approach by September 30, 2018, which will include any new investment projects approved by June 30, 2018. The second tier will be developed and presented to NRC's Senior Executive Committee for approval by December 31, 2018.
  2. Implementation of leveraged TB strategies within NRC's guides and tools by December 31, 2018.
  3. To be completed by December 31, 2018.

Contact: Vice-President, Corporate Services and Chief Financial Officer

Recommendation 3

The Vice-President, Corporate Services and Chief Financial Officer should ensure that project close out reports are completed in a timely fashion and project lessons learned are consistently collected and shared with the major capital project community in support of continuous improvement. [Priority: Moderate]

Corrective Management Action Plan

Finance Branch acknowledges project close out documents are challenging to obtain in a timely manner. As projects close, PM's are assigned new activities. Conflicting schedules and priorities ensue, hindering the submission of timely, meaningful project reports.

Finance Branch will include Close out report responsibilities in the development of a project related Performance Expectation Standard, which will be used to document and measure on a number of performance metrics.

Lessons Learned are currently submitted at the end of the project life cycle. Finance Branch will incorporate processes to accumulate and disseminate lessons learned to the project management community across the stages of the projects lifecycle (pre-planning, planning, execution, and close out).

Expected Implementation Date and Responsible NRC Contact

Aligned with the response to Recommendation 1, the draft Standard will be completed by 30 September 2018 and implemented by December 31, 2018.

Contact: Vice-President, Corporate Services and Chief Financial Officer

Recommendation 4

The Vice-President, Corporate Services and Chief Financial Officer should refine the project oversight process by:

  1. Clarifying expectations of the project team and senior management for information required to support oversight, including non-financial metrics; [Priority: Moderate]
  2. Linking project financials with applicable project deliverables and/or milestones to improve project financial forecasting, objectivity of project reporting, and enable project progress assessment at any point within the project lifecycle. [Priority: High]

Corrective Management Action Plan

  1. Consistent with recommendation 1, Finance Branch will finalize the definition of roles, responsibilities and accountabilities. This deliverable will address non-financial expectations for the project team, as well as senior management (i.e. sponsor). Finance Branch will define expectations that are consistent with the project complexity and risk tiers. Finance Branch will work with Project Sponsors to clearly define specific quantitative key deliverables and/or milestones in project documents.
  2. Finance Branch will leverage the NRC project harmonization working group to enhance the use of milestone functionality and/ or work package deliverables within SAP PS. Finance Branch will also consult with the small working group tasked with reviewing the application of Earned Value management within its project management practices.

Expected Implementation Date and Responsible NRC Contact

  1. Definitions will be completed by September 30, 2018 with implementation by December 31, 2018
  2. Finance Branch will work project managers associated with new investments, to be selected by June 30, 2018 to incorporate milestone functionality and / or work package deliverable reporting within SAP PS to enable project progress assessment at any point within the project lifecycle. Assessments will be done on existing projects by September 30, 2018 to determine what action can be taken and work with those teams to implement as appropriate.

Contact: Vice-President, Corporate Services and Chief Financial Officer

Recommendation 5

The Vice-President, Corporate Services and Chief Financial Officer should define and implement a consistent process to assess major capital project results and deliverables against planned objectives, outcomes and plans. [Priority: Moderate]

Corrective Management Action Plan

Finance Branch will work with appropriate NRC business partners (e.g. Policy, Strategy and Performance) to develop and implement a Benefits Realization and Outcome Management Framework. Finance Branch will leverage Treasury Board Secretariat (TBS) Outcome Management Guide and Tools, as well as engage with OGD's to incorporate lessons learned and best practises.

Expected Implementation Date and Responsible NRC Contact

A working group will be created by October 31, 2018. Initial findings are to be presented by January 31, 2019 and with an identification of required actions to be developed by March 31, 2019.

Contact: Vice-President, Corporate Services and Chief Financial Officer