Evaluation of the Aerospace Research Centre

 
Information
Status: Active
Effective date: June 26, 2020
Prepared by: Office of Audit and Evaluation, National Research Council Canada
Approval: NRC's President
Related document: Summary - Evaluation of the Aerospace Research Centre
ISBN: 978-0-660-34897-1
Alternate format: Evaluation of the Aerospace Research Centre (PDF, 1.64 MB)

Acronyms

AIAA
American Institute of Aeronautics and Astronautics
AFRL
Air Force Research Laboratory
DLR
Deutsches Zentrum für Luft-und Raumfahrt (German aerospace centre)
DND
Department of National Defence
DST
Defence Science and Technology Grou
FAA
Federal Aviation Administration
FWCI
Field-Weighted Citation Index
GBA+
ender-Based Analysis Plus
ICAS
International Council of the Aeronautical Sciences
IRAP
Industrial Research Assistance Program
NASA
National Aeronautics and Space Administration
NRC
National Research Council of Canada
OGD
Other government department (Canadian federal)
ONERA
Office National d’Études et de Recherches Aérospatiales (French national aerospace research centre)
PSES
Public Service Employee Survey
R&D
Research and Development
SME
Small-to-Medium Enterprise
TRL
Technology Readiness Level
UAV
Unmanned Aerial Vehicle

Executive Summary

The National Research Council of Canada (NRC)’s Aerospace Research Centre supports industry and other government departments to create, advance and apply world-leading aerospace capabilities in Canada for the benefit of society. Its research is organized into five laboratories: Aerodynamics, Aerospace Manufacturing, Flight Research, Gas Turbines, and Structures and Materials Performance. It is one of the NRC’s largest research centres in terms of both budget and staff. This evaluation covered 2012-13 to 2018-19 inclusively, and drew on a bibliometric study, data review, document review, internal and external interviews, a client survey, and a peer review by experts from industry, government, and academia.

Areas of demonstrated strength

Performance

The research centre is an important partner for its industry clients and government partners in contributing to advancements in the field of aerospace. It fosters economic growth for Canadian companies, supports government operations (in particular for the military) as well as policy and regulations, and contributes to social and environmental impacts.

Uniqueness

The research centre’s combined competencies,facilities, and ability to perform end to end research (from conception to in-flight testing) are unique within Canada.

Areas for improvement

Integration and alignment

The research centre’s new areas of focus are well aligned to stakeholder needs. Ensuring that objectives are met in these areas will require increased integration of staff across laboratories and the NRC, as well as alignment of projects to objectives.

Recommendation 1: Increase collaboration and integration across its five laboratories and other research centres to facilitate interdisciplinary work and integrated solutions to future aerospace problems.

Recommendation 2: Clearly articulate expectations regarding research objectives and priorities to staff. Project selection and prioritization (including technical services vs. strategic research services) should align with these expectations and the research centre’s strategic objectives.

Engagement

The research centre has recognized a need to increase collaboration with SMEs, universities, and Transport Canada. These relationships are critical elements of future success.

Recommendation 3: Better articulate and market its unique value proposition for key stakeholder groups (i.e., SMEs, original equipment manufacturers and Tier 1 companies, universities, other government departments, international organizations and associations). The research centre’s specific role and contribution in each of its strategic research areas should also be clearly identified. This should be communicated to both internal and external stakeholders.

Recommendation 4: Identify clear mechanisms (including an identified internal champion/resource) to build strategic and successful relationships with universities, SMEs, and Transport Canada.

Capabilities

The research centre has unique and diverse facilities that require significant upgrades and renewal to ensure sustained operations. Prioritization and timely decisions supported by the NRC’s senior management are crucial.

Research centre staff are dedicated and driven but, in recent surveys and interviews, also report being overworked and stressed. The level of stress has improved from last year, according to the PSES survey, but to retain and support current staff, as well as recruit staff to new areas of research, this issue requires further action.

Recommendation 5: Identify specific timelines and decision points, along with clear accountabilities for investments in facilities upkeep, upgrade and replacement.

Recommendation 6: Identify and implement actions to reduce stress and overwork for employees.

Metrics

Many facets of the research centre’s work on improving aircraft components and materials contribute to downstream environmental benefits, as do much of its atmospheric and UAV work. These important contributions should be captured by the research centre.

Recommendation 7: Explore different means or metrics to better capture the impact of its work, especially with regards to environmental benefits.

Introduction

The evaluation of the National Research Council (NRC)’s Aerospace Research Centre covered the 2012-13 to 2018-19 period. It was carried out in accordance with the NRC’s approved evaluation plan and Treasury Board policies. The research centre was last evaluated in 2010 for the 2004-05 to 2008-09 period.

This report begins by providing a profile of the Aerospace Research Centre. It then presents the evaluation findings on the appropriateness of research, stakeholder engagement, capabilities, scientific excellence and performance. Following the conclusion are seven recommendations for improvements within the research centre.

Evaluation approach

Methods

Mixed methods were used to maximize the generation of useful, valid and relevant evaluation findings. This approach also allowed for convergence of results across methods and contributed to a better understanding ofcomplex issues by exploring different facets.

  • Bibliometric study (publication citation analyses)
  • Data review (financial, administrative and performance data)
  • Document review
  • Client survey (26% response rate)
  • Internal and external interviews (N=34)
  • External expert peer review committee

For more detailed information on the methods, including challenges and limitations, refer to Appendix A.

Questions

The evaluation questions were developed based on consultations and a review of key documents. The questions were:

  1. Is the Aerospace Research Centre’s strategic plan appropriate to ensure the continued relevance and success of the Research Centre?
  2. Has the Aerospace Research Centre engaged the most appropriate clients and collaborators in the most effective ways?
  3. To what extent does the Aerospace Research Centre have the capacities, competencies, and facilities to achieve its objectives?
  4. To what extent is the Aerospace Research Centre a leader in scientific excellence in the areas of aerodynamics, flight research, structures and materials, manufacturing and propulsion?
  5. To what extent has the Aerospace Research Centre contributed to longer term outcomes?
    1. the economic growth and prosperity of the Canadian aerospace industry?
    2. government operations, policy and regulations?
    3. social and environmental impacts?

Profile

The Aerospace Research Centre provides the aerospace industry and government agencies with ideas, new technologies and the development and demonstration of new products and processes that target the challenges faced by the aerospace sector. The Aerospace Research Centre works in five areas: aerodynamics, propulsion, flight research, structures and materials, and manufacturing.

Research centre structure

Figure 1.

""
The Aerospace Research Centre is divided into five distinct laboratories.
Long description of Figure 1

This illustrates the Research Centre Structure, in a form of a three-tiered pyramid.

At the top of the pyramid is the “Vice-President of Transportation and Manufacturing Division”.

The second tier of the pyramid, under the Vice-President, is the Director General Aerospace Research Centre.

The last tier of the pyramid, divided into two rows, includes the Aerospace Research Centre’s five distinct laboratories and the Director of Operations. The first row includes the Director of R&D Aerodynamics Lab, the Director of R&D Structures and Materials Performance Lab, and the Director of R&D Flight Research Lab. The second row includes the Director of R&D Gas Turbines Lab, the Director of R&D Aerospace Manufacturing Technology Centre, and the Director of Operations.

Research activities

Consolidation of programs and activities

The Aerospace Research Centre had six active research programs between 2013-14 and 2018-19. Following its strategic planning for 2019-2024, the research centre moved forward with two programs, plus strategic client services, and support to another NRC research centre’s program in advanced manufacturing. Several elements of previous programs and services are integrated into these new programs or the research centre’s strategic client services. For more detailed information on the activities of the research centre, refer to the logic model in Appendix B.

Programs before 2019 (2013-14 to 2018-19)

Civilian Unmanned Aircraft Systems worked to develop and demonstrate the technology necessary to address key impediments to the large-scale adoption of unmanned aerial vehicles.

Aeronautical Product Development Technologies worked to reduce the time that Canadian aerospace companies spend on the development of products by ensuring efficient passage through the certification process.

Working and Travelling on Aircraft worked to accelerate the development of new cabins, environmental controls and avionics to provide airline passengers with a comfortable yet economical travel experience.

Air Defence Systems provided the dominant source of aerospace research and development for the Department of National Defence.

Reducing Aviation Icing Risk worked to develop technologies and processes that would reduce the costs of production and icing product qualification; and helped regulators to develop standards and tools for demonstrating compliance with these standards.

Aeronautics for the 21st Century aimed to de-risk the innovation process for aerospace original equipment manufacturers in areas such as manufacturing and fuel efficiency. The program ended in 2016.

New programs and services (2019 to 2024)

Aeronautical Product Development and Certification Program aims to reduce the time-to-market for aerospace products by solving complex problems of product development and certification, with a focus on aircraft icing and innovative methods for physical and “virtual” testing.

Integrated Autonomous Mobility Program seeks to accelerate technology development in unmanned aerial mobility and assist with the development of certification and qualification standards.

Support to other NRC programs

The research centre continues to support other programs such as the Advanced Manufacturing Program (housed within the NRC’s Automotive and Surface Transportation Research Centre) as it aims to increase manufacturing, efficiency and agility by using emerging technologies.

Strategic client services

Defence technologies and sustainment intends to enhance defence platform capabilities, mission readiness and fleet affordability and sustainability of the Royal Canadian Airforce.

Air travel research aims to improve the end-to-end travel experience, looking at issues related to passenger comfort, health and safety and aircrew efficiency and fatigue.

Other technical services and research and development projects (including Ideation) are also conducted under strategic client services.

Financial resources

Revenue and expenses

The Aerospace Research Centre is one of the NRC’s largest research centres in terms of the number of staff and facilities. It is also the NRC’s top revenue generating research centre, earning $196.7 million over the past six years with direct expenses totaling $193.4 million. Between 2013-14 and 2018-19, the research centre earned an average of $32.8 million per fiscal year.

Over the past three years (2016-17 to 2018-19), the research centre generated more revenues than planned, with its highest earnings in 2018-19 ($37.8 million).

Figure 2.

""
*Note: Data is based on NRC project coding. However, the distinction between technical services and strategic research is not always so binary. Within the Aerospace Research Centre, technical services often serve strategic purposes or lead into strategic research projects. The research centre also engages in non-revenue generating strategic research projects.
Long description of Figure 2

A bar graph depicting the revenue of the Aerospace Research Centre in millions of dollars (ranging between $0 to $40 million) by fiscal year. Seven fiscal years were examined, between 2013-14 to 2018-19. A line is superimposed on the bars to indicate Aerospace Research Centre’s target revenue.

  • For 2013-14, the earned revenue was approximately $34.8 million and the target revenue was $36.2 million.
  • For 2014-15, the earned revenue was approximately $31.2 million, and the target revenue was $36.8 million.
  • For 2015-16, the earned revenue was approximately $31.3 million, and the target revenue was $36 million.
  • For 2016-17, the earned revenue was approximately $28.7 million, and the target revenue was, $27.2 million.
  • For 2017-18, the earned revenue was approximately $31.4 million, and the target revenue was $29.7 million.
  • For 2018-19, the earned revenue was $37.8 million, and the target revenue was $31.4 million.

The figure revealed that over the past three years, between 2016-17 to 2018-19, the research centre generated more revenues than planned, with the highest earnings in 2018-19 ($37.8 million).

A donut chart is found on the right, further breaking down the sources of the revenue as shown in the previously discussed bar graph. The donut chart indicates that the majority of the revenues were generated from technical services ($143 million), while $53.6 million in revenues were generated from strategic research services. A note is appended beneath the donut chart, further explaining the source of the data.

Figure 3.

""
Long description of Figure 3

A chart summarizes the total expenses and illustrates the yearly average, the lowest year and the highest year. The total expenses yearly average is $62.5 million, and the lowest year is 2013-14 at $59.8 million, whereas the highest year is 2018-19 at $69.6 million.

What’s the difference between strategic research and development (R&D) projects and technical services?

Strategic R&D consists of collaborative research projects undertaken with partners to de-risk R&D and accelerate commercial development timelines or to advance knowledge that is of interest to the NRC. Technical services make use of NRC IP to assist clients in solving immediate technical problems (including maturation of client technology) through the delivery of specialized fee-for-service support.

Human resources

Workforce

In 2018-19, the Aerospace Research Centre employed 354 employees across five laboratories. Of these, 152 were research council officers or research officers and 106 were technical officers. The majority (84%) of research centre staff were permanent fulltime employees. Most staff were located at the research centre’s four Ottawa laboratories, with the balance in Montreal at the Aerospace Manufacturing Technology Centre. The Flight Research Laboratory had the largest number of employees with 76 staff.

Figure 4.

""
Long description of Figure 4

An illustration of the workforce distribution of the Aerospace Research Centre. The illustration is of five separate boxes, one box per laboratory.

  • Box 1: Gas Turbines Laboratory has 55 employees.
  • Box 2: Structures and Material Performance Laboratory has 40 employees.
  • Box 3: Aerodynamics Laboratory has 53 employees.
  • Box 4: Flight Research Laboratory has 76 employees.
  • Box 5: Aerospace Manufacturing Technology Centre has 47 employees.

All of the employees from the first four mentioned laboratories (Gas Turbines, Structures and Material Performance, Aerodynamics, and Flight Research Laboratories) are located in Ottawa. Whereas 34 of the 47 employees from the Aerospace Manufacturing Technology Centre are located in Montreal, and the remaining 13 are located in Ottawa.

Gender, diversity and inclusion

When compared to workforce availability, the Aerospace Research Centre has good representation of women in professional and administrative roles. However, women are under-represented in management and technical officer positions. Visible minorities are well-represented in professional roles but under-represented in management, administrative and technical roles. Indigenous people and people with disabilities are under-represented in all categories in the Aerospace Research Centre, as is the case across the NRC.

If the research centre hired two women as directors and four as technical officers, it would be on par with workforce availability for representation of women.

(Workforce availability estimates are based on the 2016 Census.)

Projects and clients / collaborators

The Aerospace Research Centre undertook 714 unique projects between 2013-14 and 2018-19. Most of the research centre’s work, 518 projects (73%), focused on the provision of technical services with 191 projects (27%) focused on strategic research and development (R&D).

Industry

  • 86% (292) of the research centre’s clients
  • 47% Canadian and 39% foreign
  • Responsible for 56% of the research centre’s revenues

Other Government Departments

  • 7% (25) of the research centre’s clients
  • 3% Canadian, 1% provincial and 2% foreign (does not add up to 7% due to rounding)
  • Responsible for 38% of the research centre’s revenues

Academia

  • 4% (13) of the research centre’s clients
  • 3% Canadian and 1% foreign

Other

  • 3% (10) of the research centre’s client
  • Includes non-profit organizations such as Speed Skating Canada

Appropriateness of research

Overall finding: The Aerospace Research Centre’s new strategic focus on emerging technology areas is well-aligned with industry and government needs. To ensure clarity of purpose and alignment of efforts within the research centre, increased communication and direction to staff is required.

Uniqueness

Finding

While other global research and technology organizations conducting similar research are much larger in size and funding, the Aerospace Research Centre delivers important and unique services to the Canadian government and private sectors.

Sources: Document review, client survey, internal and external interview, peer review committee

Similar areas of research compared to other research and technology organization

The Aerospace Research Centre’s areas of research are similar to those of other major research and technology organizations, such as NASA (US), DLR (Germany), DSTL (UK), and ONERA (France). However those organizations are much larger in size, scope and budget.

End to end capabilities unique in Canada

Within Canada, the combination of research expertise, capabilities and facilities provided by the research centre is unique. Providing the ability to conduct research from models to in-flight testing is viewed positively by the research centre’s clients.

Safeguarding Canadian information

Canadian entities like Bombardier and the Department of National Defence rely on the Aerospace Research Centre, as opposed to other organizations, for their research and testing needs. As part of the government of Canada, the NRC is better able to ensure the safeguarding of their intellectual property and national defence secrets.

"I have a great deal of confidence in the facility at NRC to keep the data confidential. There aren’t any other facilities in Canada that could provide the data we need."
OGD Client

Relevance

Finding

The Aerospace Research Centre’s new strategic focus on emerging areas aligns well with industry needs and government priorities. Achieving impact in the new areas will require a greater integration of the research centre’s five laboratories, which, according to the peer review committee, are currently still operating in silos.

Sources: Document review, client survey, internal and external interview, peer review committee

Industry needs and trends

The Aerospace Research Centre’s new strategic focus on disruptive technologies aligns well with identified sector needs in the emerging areas of:

  • digital twin
  • additive / subtractive manufacturing
  • autonomous aerial mobilit
  • disruptive architecture (includes electrification)

The peer review committee supported the research centre’s strategic alignment to these areas provided that is:

  • Identify, clearly communicate, and limit focus to the areas where it can add the most value in these highly populated research areas
  • Further integrates its five laboratories to address these aerospace problems in a unified and interdisciplinary way

Government priorities and needs

The Aerospace Research Centre’s work aligns with the priority the federal government places on the aerospace industry as a driver of domestic research and development. For example, the research centre’s work aligns well with three superclusters (Digital Technology, Next Generation Manufacturing and Artificial Intelligence-Powered Supply Chains). It also supports major federal investments under the Strategic Innovation Fund.

The Aerospace Research Centre is supporting federal departments through its work. In particular, it is helping the Canadian military with the maintenance and repair of its aging aircraft, development of new technologies, and procurement of new equipment and technologies. The research centre also plays a key supporting role to Transport Canada in the provision of evidence required to certify and regulate the aerospace sector. Transport Canada’s focus on the certification of new manufacturing methods and the regulation of unmanned aerial vehicles also require close collaboration with the Aerospace Research Centre.

Competing priorities

Finding

The Aerospace Research Centre has been challenged in trying to balance the financial imperatives presented by revenue targets with strategic research priorities, and building a pipeline from low to high TRL. Clear direction to staff is required to align efforts.

Sources: Document review, client survey, internal and external interview, peer review committee

Technical services

Activities within the Aerospace Research Centre are heavily focused on the provision of technical services or technology maturation. This is mainly due to the following realities:

  • Companies and OGDs in the aerospace sector mainly require high TRL support from the research centre
  • Technical services lend themselves better to higher Technology Readiness Level (TRL) work
  • Of note, plans to increase collaboration with universities were encouraged by the peer review committee –especially with regards to new areas like autonomous mobility where lower TRL research will be undertaken.
  • Technical services generate significant revenue needed for the research centre to achieve its financial targets.
  • Technical services drive utilization of the research centre’s large facilities (e.g., wind tunnels) and contribute to the recovery of their significant operational and maintenance costs

The Aerospace Research Centre is aware of the need to conduct strategic research and development projects within this context and aims to direct such projects within its four priority areas.

Revenue targets

The NRC has moved away from a focus on revenue growth and has stabilized revenue targets in recent years. However, staff continue to feel pressure to meet these targets, and continue to exceed them. In a record-breaking year in 2018-19, the research centre exceeded its revenue target by $6.4 million or 20%.

Communication

Achieving a balance, between technical services and strategic research priorities, that works within existing constraints is an ongoing struggle for the Aerospace Research Centre’s management team. Research centre staff would benefit from clear communication of expectations and priorities from senior management. Consistency of messaging when selecting and approving projects or research initiatives would also serve to reinforce these priorities.

Stakeholder engagement

Overall finding: The Aerospace Research Centre has effectively engaged with its major industry clients and the majority of its other government departments. While the research centre plans to enhance outreach to, and collaboration with, universities and SMEs to achieve its objectives, the peer review committee found that there is no clear plan or mechanism to ensure successful engagement with these two stakeholder groups.

Clients and collaborators

Finding

The Aerospace Research Centre has engaged with multiple stakeholders across the aerospace sector with a specific focus on large industry players and other government departments.

Sources: Data review, client survey, internal and external interviews

Overall, the research centre’s project revenues are concentrated around a small number of major projects from ten clients.

Total revenue (2013-14 to 2018-19)

""

Almost ¾ of the research centre’s revenue comes from ten clients. Revenue figures from these clients, range from just over $2 million to over $56 million.

*Total revenue in this figure differs from the total revenue reported on slide 13 because this figure excludes revenues from the discontinued Aero21 program.

Long description of Total revenue (2013-14 to 2018-19)

A single bar graph depicting the total revenue between 2013-14 to 2018-19. The total revenue shown is $174 million (excluding revenues from the discontinued Aero21 program), and the graphic shows that 72% of the research centre’s revenue comes from ten clients. Revenue figures from these clients range from $2 to $56 million.

Industry: The Aerospace Research Centre works with leading Canadian aerospace companies and is well-connected to large original equipment manufacturers and Tier 1 suppliers*. The research centre has also maintained relationships with key industry associations.

*Original equipment manufacturers (OEMs) are companies whose goods are used as components in the products of another company. Tier 1 suppliers supply parts or systems directly to OEMs.

Government departments and regulators: The Department of National Defence was by far the research centre’s largest revenue generating client. Client interviews indicate that a positive and collaborative relationship has developed, making the research centre an engaged enabler that helps the department push boundaries.

Collaborations with Transport Canada and Environment and Climate Change Canada have taken place, and continue to be essential on several different fronts including anti-icing research, alternative fuels, oil sands monitoring, autonomous vehicle regulations, and aircraft certification.

The research centre also collaborates with the American (Federal Aviation Administration) and European (European Union Aviation Safety Agency) certification and regulation bodies in several safety areas. The relationships are positive and the research centre is viewed as a respected and important contributor.

Engagement and awareness

Finding

Overall, engagement activities with key industry players and several other government departments have been effective. Positive relationships have grown between the research centre and its clients and collaborators. In terms of the research centre reputation, the peer review committee found that the research centre is not as well-known as other national research organizations, such as NASA and ESA, within the international aerospace community.

Sources: Data review, client survey, internal and external interviews

Engagement

The Aerospace Research Centre has positive relationships with clients and collaborators.

Awareness

The research centre is well known within the areas of aviation safety, aircraft research and engine icing. Most clients surveyed (74%) rated awareness of the research centre as medium to high.

Clients indicated the following activities would have the greatest impact on increasing knowledge of the research centre and the services it provides:

  • Increased participation in academic conferences
  • More frequent outreach by research centre executives
  • Greater participation of staff in national and international committees

The peer review committee supported the need for an increase in these activities, indicating that researchers may not be as well-known globally as they should be.

There is “low international awareness of some of the research centre’s areas of expertise and facilities that should be better known.”

Peer review committee

Increased engagement needed with key stakeholder groups

Finding

In its new strategic plan, the Aerospace Research Centre has recognized the need to increase engagement with universities, SMEs, and Transport Canada. Implementation of this plan is at risk without a proper understanding of collaboration mechanisms, clear communication, and dedicated resources to develop and maintain meaningful relationships.

Sources: Data review, client survey, internal and external interviews

Universities Academic institutions represented a small proportion of the research centre’s clients and collaborators over the evaluation period. This is not unique to the Aerospace Research Centre as there is a desire across the NRC to increase engagement with academia, though there are not yet any clear mechanisms to do so. The Aerospace Research Centre’s new strategic plan prioritizes university collaborations, especially to fill capacity gaps and connect with research at lower TRLs to provide a pipeline for future projects. The peer review committee indicated this relationship is critical and recommended the identification of “a dedicated resource to be the focal point for the strategic development and support of relationships with university faculties.”

Small and Medium Enterprises (SMEs) represent only a small portion (12%) of the research centre’s Canadian industry clients. The research centre is working to increase this number through collaboration with the National Research Council Industrial Research Assistance Program (NRC IRAP). However, the peer review committee encouraged the research centre to better articulate its value-add and increase visibility/promote its work to attract SMEs.

Transport Canada (TC) is a key department with whom the research centre needs to solidify collaborations. The success of the research centre’s two new programs (Integrated Autonomous Mobility and Aeronautical Product Development and Certification) is directly tied to TC regulations and certification. To date, interactions have been more productive for autonomous mobility than certification. Engagement with Transport Canada at the strategic level has not been particularly successful. The research centre acknowledges the importance of its relationship with TC, and is working towards an MOU with the department. However, progress on this has been slow.

Universities: “There does not seem to be a clear understanding, within the research centre, of how to collaborate with universities and academic researchers.”

SMEs: “It was unclear to members of the committee whether there is a specific communication plan to reach SMEs and to promote the research centre’s capabilities and the available financial incentives.”

Peer review committee

Capabilities

Overall finding: The Aerospace Research Centre has had the capacities, competencies and facilities to achieve its objectives, but requires continued action as it plans to shift competencies to address new strategic objectives and address overwork and stress. Timely decisions with regards to facilities (i.e., which to keep, upgrade or divest) and the aircraft fleet (i.e., matching current and future research needs to aircraft, time and cost of instrumenting and training) are critical to the achievement of the research centre’s objectives. Facility investments and a clearly communicated way forward are essential.

Capacity

Finding

Clients are satisfied with the research centre’s capacity to meet their needs, although research centre staff have identified heavy workloads and a lack of capacity as factors causing stress. Research centre staff feel pressure to meet workload and revenue targets. These factors may also be reducing staff’s ability to engage in professional development and scientific knowledge dissemination activities.

Sources: Data review, document/literature review, internal interviews, peer review

Adequate capacity

The Aerospace Research Centre has a current count of 354 employees. Generally, clients are satisfied with staff capacity. Clients praised their responsiveness and flexibility to put in the hours and work necessary to successfully complete projects.

However, overwork and stress were a concern expressed by nine out of the ten senior managers interviewed. Employees are also experiencing high levels of stress. Interviews indicated that publications or strategic research projects are sometimes done on employees’ own time. The peer review committee noted that “The current workload provides little time, if any, for writing publications, participating in technical associations, and conducting (possibly) more rewarding research.” The Public Service Employee Survey (PSES) also identified workload as an issue.

Selected factors causing stress at work (PSES)

""
Long description of Selected factors causing stress at work (PSES)

Bar graphs were used to illustrate the proportion of employees reporting experiencing three selected work-related stressors in 2017, 2018, and 2019.

  1. Not enough employees. In 2017, 33% of employees reported ‘not enough employees’ as a factor that caused stress at work. This proportion increased to 56% in 2018, but it lowered to 56% in 2019. A pink link across the chart shows the 2019 NRC average for this factor is 28%.
  2. Heavy workload. In 2017, 27% of employees reported heavy workload as a factor that caused stress at work. In 2018, that proportion rose to 48%. The proportion of employees reporting heavy workload as a stressor in 2019 was comparable to that of 2017 (28%). A pink line across the chart shows the 2019 NRC average for this factor is 21%.
  3. Overtime or long work hours. In 2017, 11% of employees reported overtime or long work hours as a factor that caused stress at work. In 2018, 21% of employees reported overtime or long workhours as a factor that caused stress. In 2019, 13% of employees reported overtime or long work hours as a factor that caused stress. A pink line across the chart shows the 2019 NRC average for this factor is 10%.

Stressors

Although 71% of the Aerospace Research Centre’s respondents to the 2019 Public Service Employee Survey (PSES) indicated they have support at work to balance their work and personal life, they also highlighted several important stressors. In particular, only 55% of the research centre’s employees reported that they can complete their assigned workload during their regular working hours (although this reflects an improvement from the 2018 result of 51%). Administrative workload was identified as a contributing factor to employees’ stress in sessions conducted by the research centre following the 2017 PSES results. Several actions have since been taken by the research centre to address various elements of this administrative burden.

While the research centre is aware that change is needed and, as mentioned above, some actions have already been taken, more work is needed. This may be challenging given the research centre’s dependency on revenue generation, competing priorities, and a general aerospace culture of working long hours.

PSES Results 2017 2018 2019 2019 NRC Average
I have support at work to balance my work and personal life 74% 71% 1% 78%
I can complete my assigned workload during my regular working hours 67% 51% 55% 68%

Competencies

Finding

The research centre has a strong workforce with unique expertise. A shift to new strategic areas (i.e., digital twin, additive/subtractive manufacturing, autonomous aerial mobility, and disruptive architecture) require hiring in new competency areas, particularly in digital technologies. The research centre will face competition in recruiting and retaining staff in these highly sought areas.

Sources: Data review, document review, client survey, internal and external interviews, peer review committee

Competent and adaptable staff

Overall, research centre clients were highly satisfied with the expertise of staff. Staff are considered highly competent and even world class experts in certain areas such as engine icing and advanced manufacturing.

The peer review committee found that the research centre has a “highly productive and flexible workforce with unique technical capabilities.”

Competing for resources in new areas

It is imperative that the research centre continue to develop competencies in its new areas of focus. In particular, the peer review committee highlighted the importance of developing competencies in high performance computing, machine learning, and multidisciplinary design optimization. However, the research centre’s ability to attract and retain staff may be hampered by fierce competition across the aerospace industry in Canada, where there is a large shortage of qualified graduates. Additionally, if unaddressed, workload and stress issues affecting current research centre staff will further affect recruitment and retention of a new generation of researchers who value work-life balance.

Increased collaboration with universities and with other NRC research centres, especially with regards to digital technologies and advanced manufacturing, will be important to the Aerospace Research Centre’s strategic success. However, as expertise in digital technologies is in high demand across NRC research centres, and advanced manufacturing capacity is growing across Canada, the research centre may also face internal competition for resources.

Facilities

Finding

The research centre facilities are leading edge or unique in many of its areas of focus. Decisions by NRC senior management and the research centre are required, without delay, on which facilities to maintain, operate, upgrade and retire, given the significant time and financial investments required to purchase and equip new aircraft and to maintain the relevance and competitiveness of existing facilities.

Sources: Data review, document review, client survey, internal and external interviews, peer review committee

Unique facilities

Many of the Aerospace Research Centre facilities have unique capabilities. For example:

  • Altitude Icing Wind Tunnel is unique in simulating altitudes up to 40,000 feet and temperatures down to -40ºC.
  • Gas Turbine Engine Test Facility has state-of-the-art icing test systems accepted for engine certification by major regulators.
  • Convair 580 the NRC’s world-class airborne lab, is used internationally for icing and satellite research.

Overall, the research centre has the types of facilities needed to fulfil the project objectives of their clients and they are price competitive with their peers.

Aging infrastructure

Many of the research centre’s facilities are several decades old. While the majority have been well maintained and are still functioning well, they require regular maintenance and upgrades. For example, in the next 5 to 10 years, the Aerodynamics, Gas Turbine and Flight Research Laboratories together, will require capital investments ranging from $24 to $60 million. This includes major investments in wind tunnels, engine test facilities and aircraft. Additionally, the replacement of aircraft, such as the Convair 580 (a key aircraft used to support a wide range of research projects) require much advanced planning due to the significant financial investments required to purchase, re-instrument, and train staff. This is a multi-year, multi-million-dollar project.

Utilization

While some facilities are heavily utilized, a number of the facilities are underutilized. Fluctuating demand due to industry cycles is a contributing factor, as is a lack of technical staff. For example, several wind tunnels share staff, as do some aircraft, so when one facility is in use another must sit idle.

Multiple inputs into facilities decision-making

In addition to the current evaluation, the research centre has received, and will receive additional recommendations and advice regarding its facilities, namely through the NRC-wide facilities review process and its own fleet renewal initiative. These multiple reviews on different schedules, combined with the special financial planning required to fund aircraft acquisitions, and facilities maintenance and upgrades, add complexity to an already multifaceted process. The evaluation did not find evidence of a plan to integrate these, nor a detailed timeline on decision-making by the research centre and the NRC. Nor was there evidence of how the research centre will implement any identified actions.

Performance

Overall finding: In areas of icing and materials fatigue, the research centre is a leader in scientific excellence, as measured by publications. Using this same metric (publications), the research centre is not a leader in scientific excellence in other areas. However, this is to be expected given its operating model, which emphasizes technology maturation.

There are multiple examples of how the Aerospace Research Centre has contributed to its longer-term outcomes of fostering economic growth and prosperity of the Canadian aerospace industry; advancing government operations, policy and regulations; and social and environmental impacts.

Scientific excellence

Finding

The Aerospace Research Centre’s publication count was lower than that of the NRC and other international organizations in this sector. This is not unexpected given the research centre’s service model that emphasizes technology maturation and proprietary client work. According to bibliometric analyses, the research centre is a Canadian leader in the topic areas of ice, ice particle, and fatigue of materials.

Sources: Bibliometric study, document review, peer review committee

Publications

Bibliometric analysis showed that the research centre publishes less and is cited less frequently than the NRC as a whole (FWCI of 1.5) and its global comparators. The research centre’s publication output dropped over the evaluation period, from a high of 129 publications in 2013 to a low of 94 publications in 2018.

Figure 5.

""
Long description of Figure 5

A horizontal bar graph illustrating the Field-Weighted Citation Index (FWCI) during 2013-18 of NRC-Aerospace Research Centre and five other comparable research centres/organizations.

In descending order, the FWCI of the six research centres and organizations is as follows:

  1. NASA (USA): 1.9
  2. DLR (Germany): 1.6
  3. AFRL (USA): 1.5
  4. ONERA (France): 1.3
  5. DST (Australia): 1.2
  6. NRC-Aerospace Research Centre: 1.1

In two areas, gas turbines and aerodynamics, the research centre’s FWCI is above the Canadian and Global FWCI.

What is the Field-Weighted Citation Index (FWCI)? The FWCI is a normalized indicator to gauge relative performance of a publication in terms of citations. It takes into consideration the number of publications and publication norms in different research domains.

Awards

Several Aerospace Research Centre researchers have won prestigious awards for their work. These include awards from NASA, NATO and Manitoba Aerospace for the research centre’s work on engine icing. Other researchers have won NATO awards for their work on gas turbine engines, structural fatigue in military aircraft and the aerodynamics of helicopter launch and recovery.

“Publishing should be viewed as both an asset to the Aerospace Research Centre to promote its visibility, and as a concrete way to give the researchers a sense of accomplishment.”

Peer review committee

Finding

With the exception of a few areas of research (i.e., icing research, studying cloud physics, emission measurements at high altitude, and fatigue) the peer review committee found that the research centre’s researchers and scientific achievements are not well known on the global stage.

Sources: Bibliometric study, document review, peer review committee

The peer review committee found that the Aerospace Research Centre has demonstrated excellence in:

  • Reduction in icing risk
  • Hyper-spectral sensing
  • Special use aircraft that perform several types of reconnaissance for the Government of Canada
  • Fatigue and fracture studies for the Canadian Defense Forces

However, the committee also noted that due to the Aerospace Research Centre’s model of higher TRL and revenue-generating projects, the greater aerospace community may not view the research centre as world class. The committee noted in particular that the research centre publishes less papers and technical memoranda than its peers within NASA, for example. Combined with lower attendance at scientific conferences and fewer university collaborations, the committee warned that this may prevent researchers and the Aerospace Research Centre itself from enhancing a reputation for scientific excellence.

Industry growth and prosperity

Finding

The Aerospace Research Centre has contributed to its industry clients’ abilities to develop, adopt, and commercialize new technologies; grow; improve productivity; and reduce costs.

Sources: Bibliometric study, document review, peer review committee

Developing new technologies

The research centre has assisted clients in developing new or improved products or solutions. That assistance has been important in de-risking new aircraft development and cabin configurations. New technologies have been developed, for example, in:

  • Icing: ultrasound ice accretion sensor to detect ice build-up on aircraft engines
  • UAVs: UAV sense and avoid prototype technologies
  • Manufacturing: automated welding and machining of next generation fuel injector assembly, and cold spray structural repair

Commercializing new technologies

There are several examples of new technologies commercialized by the research centre:

  • NRC patented profilometer: a tool that provides faster and more accurate inspection of large aircraft components was licensed to an industrial engineering group
  • Particle detection probe: a device designed to detect dangerous ice crystals found at high altitudes on aircraft engines has been commercialized by an American company

Company growth

Examples of the research centre’s impact on company growth are:

  • An international original equipment manufacturer experienced a 20% growth in employment after working with the research centre on product development
  • A Canadian Tier 1 company won a $100 million contract, in part, due to its collaboration with the research centre

Improved productivity and cost savings

Several of the research centre’s clients indicated that through optimization of products or processes, and model testing, they reduced costs or improved productivity.

Government solutions

Finding

The work of the Aerospace Research Centre is contributing to government operations, policies and regulations. Its most significant contributions were in extending the life of military equipment and providing operational savings for the Department of National Defence; and in the areas of icing, cleaner fuels, and air mobility.

Sources: Bibliometric study, document review, peer review committee

Support for DND

The Department of National Defence (DND) is the research centre’s largest single client. As the following examples show, the research centre’s projects have extended the life of military equipment and paved the way for significant cost savings.

  • Fatigue Life Extension Test for CF-18 Horizontal Stabilizer: allowed the military to extend the life of stabilizers across the CF-18 fleet which are estimated to cost more than $100 million to replace.
  • Ship-Helicopter Operating Limits: Modeling and simulation tools contributed to more than $10 million in costs savings and months of time saved for flight trials and de-risking ship design. This was highlighted as a valuable contribution by the peer review committee.
  • Residual Strength Testing of CF188 Aileron: The research centre’s tests extended the retirement time of the aircraft aileron, a hinged control surface on an aircraft wing. Potential savings are estimated at up to $24 million.

Air mobility

The research centre has an ongoing relationship with Transport Canada to provide advice and generate the scientific data required to inform the development of regulations for drones and unmanned aerial vehicles.

Icing

The Aerospace Research Centre’s work on aircraft icing has had a positive impact on aviation safety and aircraft certification in Canada and around the world. Its collaboration with Environment and Climate Change Canada provided key data for the development of a new appendix by the United States Federal Aviation Administration (FAA) on the operation of aircraft engines under high altitude conditions. As well, the research centre’s anti-icing fluid testing contributes to Transport Canada updates of regulated holdover times for aircraft to reflect new anti-icing fluids. These holdover timetables are used internationally to determine how long an aircraft can be on the ground following de-icing.

Fuel

The research centre’s work with Transport Canada on the FAA’s Piston Aviation Fuel Initiative could eventually lead to policy changes by Environment and Climate Change Canada to ban lead in aviation fuel, as Canada has done with other vehicles. Its work with DND on alternative aviation fuels has also allowed the Canadian Airforce to maintain interoperability with its allies who are increasing the use of alternative fuels.

Social and environmental impacts

Finding

The Aerospace Research Centre is contributing to social benefits through its support to the Department of National Defence, government regulations, and also through the research conducted at its Centre for Air Travel Research on air travel for people with differing needs. It is also contributing to environmental impacts (often indirectly through improvements to aircraft components). Overall, the research centre does not systematically collect data on these benefits.

Sources: Bibliometric study, document review, peer review committee

Social/health impacts

Multiple research centre projects with DND have the potential to contribute to better health outcomes for military personnel. For example:

  • Verification of Personal Protective Equipment: the research centre used computed radiography to assess whether ceramic body armour plates used by Canadian Special Operations Forces were in safe condition
  • Airforce Hearing Protectors: NRC research led to policy updates at DND that could reduce the incidence of noise-induced hearing loss in the Canadian Forces

Gender-Based Analysis Plus (GBA+)

The Aerospace Research Centre opened a new facility in 2018. The Centre for Air Travel Research is a unique facility that allows researchers to examine the passenger travel experience from arrival at the airport through a simulated flight including lighting, noise and vibration. This facility is now being used for leading edge collaborative research with partners such as Carleton University and Obesity Canada on the diverse needs of air travelers from older adults to people living with obesity.

What is GBA+?

“GBA+ is an analytical process used to assess how diverse groups of women, men and non-binary people may experience policies, programs and initiatives. The “plus” in GBA+ acknowledges that GBA goes beyond biological (sex) and socio-cultural (gender) differences. We all have multiple identity factors that intersect to make us who we are; GBA+ also considers many other identity factors, like race, ethnicity, religion, age, and mental or physical disability.”

Department for Women & Gender Equality

Environmental impacts

The Aerospace Research Centre does not systematically track environmental benefits, such as green house gas emissions reductions realized through its work. However, much of its research focusing on reducing the cost of air transport through lighter materials and more fuel-efficient engines will translate into reduced greenhouse gas emissions. For example:

  • The Aerospace Research Centre achieved a 30% reduction in fuselage weight through manufacturing using composite structures
  • The introduction of a finlet on C130 and C17 military aircraft reduced fuel consumption by up to 5%

Increasing use of unmanned aerial vehicles (UAVs) for many applications also has the potential to reduce emissions as these can be used instead of larger piloted aircraft, which use more fuel. However, UAVs could also create new environmental issues, for example from noise pollution. The research centre is working on how to reduce the noise.

Finally, the research centre is adopting more sustainable manufacturing processes. For example, by replacing the use of cutting fluids in machine processes with liquid nitrogen or micro-lubrication, the research centre is reducing the risk of environmental contamination.

Conclusions and recommendations

Conclusions

Relevance

The Aerospace Research Centre provides important support to the aerospace sector in terms of research and technology maturation. In particular, the research centre provides capabilities to other government departments and Canadian companies that are unique in Canada. Its new strategic focus aimed to develop emerging areas (i.e., digital twin, additive/subtractive manufacturing, autonomous aerial mobility and disruptive architectures) is relevant. To ensure success, the research centre must clearly identify its value-add, communicate its priorities with regards to strategic research, and ensure greater integration of its five laboratories.

Capacity

Although the research centre has been meeting the needs of its clients and exceeding its revenue targets, employees have identified workload and a lack of capacity as factors causing stress. The level of stress has improved from last year, according to the PSES survey, but further action is required. As well, to achieve its new strategic priorities, the research centre needs to recruit new staff in areas that touch on digital technologies and artificial intelligence. As the entire aerospace industry faces a shortage of workers, the research centre is looking, in part, to universities to help fill the gap. However, this requires strong partnerships, which the research centre has not yet developed.

With regards to facilities, the research centre has several unique and important facilities that provide important support to clients and partners. However, as multiple reviews provide information on facility upgrades and replacements needed, along with costing and usage data, the research centre needs to establish a clear pathway to decision-making that will ensure the research centre is able to successfully meet its objectives without delay.

Performance

There are several notable examples of excellent research among the Aerospace Research Centre’s projects, both published and unpublished. Although publications are not the only mark of scientific excellence, they are an important element in knowledge dissemination and international reputation. If the research centre wishes to be world-leading for its work, as indicated in its mission statement, it needs to increase its publications and presence on international committees and at conferences. This will also serve the purpose of helping to retain and attract top talent within its fold.

The Aerospace Research Centre is achieving its objectives. Aerospace industry clients and collaborators respect and appreciate the research centre’s contribution in ensuring they are developing leading edge, quality and safe aircraft components, systems, and materials. Government departments also rely on the Aerospace Research Centre for high quality scientific data to support their work, including regulations. In particular, the Canadian military has benefited greatly from an operational perspective in the maintenance and enhancement of its aircraft. Additionally, much of the research centre’s work contributes to social and environmental benefits, though this is not systematically recorded or quantified due to the often indirect nature of the work.

Recommendations

Recommendation 1
The Aerospace Research Centre should increase collaboration and integration across its five laboratories and other research centres to facilitate interdisciplinary work and integrated solutions to future aerospace problems.

Rationale: The research centre’s emerging areas of research (digital twin, additive/subtractive manufacturing, autonomous aerial mobility and disruptive architectures) will require increased integration across laboratories to benefit from interdisciplinary expertise. The peer review committee indicated the siloed approach to the five laboratories is a model of the past. Additionally, increased collaboration across the NRC (e.g., with the Digital Technologies Research Centre and Advanced Manufacturing Program) will further enhance a multidisciplinary approach and provide needed capabilities in the research centre’s new areas of focus.

Recommendation 2
The Aerospace Research Centre’s management team should clearly articulate expectations regarding research objectives and priorities to staff. Project selection and prioritization (including technical services vs. strategic research services) should align with these expectations and the research centre’s strategic objectives.

Rationale: Research centre staff feel a continued pressure to meet revenue targets, sometimes to the exclusion of other strategic research priorities. A consistent and clearly defined approach to project selection and prioritization will help the research centre staff prioritize their work and manage stress.

Recommendation 3
The Aerospace Research Centre should better articulate and market its unique value proposition for key stakeholder groups (i.e., SMEs, original equipment manufacturers and Tier 1 companies, universities, other government departments, international organizations and associations). The research centre’s specific role and contribution in each of its strategic research areas should also be clearly identified. This should be communicated to both internal and external stakeholders.

Rationale: The research centre has made important contributions to the field of aerospace and its industry and OGD clients. However those contributions are not all as well know as they could be, especially if they are to be internationally recognized for their scientific excellence. Additionally, to avoid loss of focus and dilution of efforts and impact, the research centre needs to clearly identify its niche within the already crowded space of its new areas of focus.

Recommendation 4
The Aerospace Research Centre should identify clear mechanisms (including an identified internal champion/resource) to build strategic and successful relationships with universities, SMEs, and Transport Canada.

Rationale: The research centre’s expressed desire to increase collaboration with universities and SMEs in its 2019-24 strategic plan is important. To focus the relationships on areas where the research centre has specific capabilities and capacity needs, and to prevent ad hoc relationship building with these partners, it is important to designate specific resources who can assist the research centre build the appropriate relationships. With regards to Transport Canada, the success of the research centre’s two new programs is intertwined with regulatory approval and the department’s active involvement, which is why a strong collaboration is key.

Recommendation 5
The Aerospace Research Centre should identify specific timelines and decision points, along with clear accountabilities for its investments in facilities upkeep, upgrade and replacement.

Rationale: The research centre is engaged in multiple initiatives that are reviewing different aspects of its facilities. Upgrades to facilities, purchases of new aircraft, and decisions regarding the use and divestiture of facilities is at a critical point for the research centre as it launches its new strategic plan and two new programs. Even though accountability for investment planning and budgets rests with NRC senior management, the research centre has an important role to play in establishing priorities to ensure its continued health and success. A clear roadmap of the decision process, timelines, and accountabilities, for the research centre and the NRC, was not evident as it relates to facilities.

Recommendation 6
The Aerospace Research Centre should identify and implement actions to reduce stress and overwork for employees.

Rationale: The research centre’s employees are driven and dedicated individuals, but are stretched to meet the high demand of work and advance their careers. Although the research centre has started to address issues affecting workload, additional action is required. As the research centre faces significant competition in recruiting staff to its new areas of focus, addressing this issue effectively will be important to ensure the research centre is able to recruit and retain staff.

Recommendation 7
The Aerospace Research Centre should explore different means or metrics to better capture the impact of its work, especially with regards to environmental benefits.

Rationale: The environment is an important priority for the Government of Canada. As many of the research centre’s projects have the potential to contribute to environmental benefits in the field of aerospace, it should find ways to measure and highlight this important contribution.

Management Response and Action Plan

 

Recommendation Risk-level Associated with not Addressing Recommendation Management Response Measure of Achievements Proposed Person(s) Responsible Expected Date of Completion

1. The Aerospace Research Centre should increase collaboration and integration across its five laboratories and other research centres to facilitate interdisciplinary work and integrated solutions to future aerospace problems.

low

Response: Accepted

The Aerospace Research Centre recognizes this opportunity to exploit a key differentiating capability of the NRC.

Action 1: Drive interdisciplinary collaboration across the five laboratories using Aerospace Programs: Integrated Autonomous Mobility (IAM), Aeronautical Product Development and Certification (APDC). Master projects within programs are designed to achieve these goals.

Action 2: Review scoring metrics for project selection under the Aerospace Futures Initiative to determine if weighting for inter-laboratory collaboration should be increased.

Action 1:

  • IAM and APDC Program leaders are integrated members of the Aerospace management team
  • Master project outcomes are developed and communicated across the five laboratories and other relevant NRC Research Centres
  • A minimum of three technical communities of practice focused on niche technology areas are designed and launched with participation from Aerospace labs, across NRC and stakeholders.

Action 2:

  • Scoring metrics reviewed
  • New weighting used for project selection (if applicable)

Aerospace Research Centre Director General

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Aerospace Research Centre Operations Director

June 2021

2. The Aerospace Research Centre’s management team should clearly articulate expectations regarding research objectives and priorities to staff. Project selection and prioritization (including technical services vs. strategic research services) should align with these expectations and the research centre’s strategic objectives.

Medium

Response: Accepted

Research Centre’s top-level planning documents, in particular, the 2019-2024 Strategic Plan and annual Operations Plans define how resources can be deployed to direct available capacity for internally funded/ collaborative work toward long-term priorities, while continuing to respect our revenue commitments and support industry needs. We will continue to take all available opportunities to ensure regular and clear communication of these plans with staff.

Action 1: Finalize and communicate lower-level plans that translate research centre-level objectives and strategies into actions that may be more meaningful to lab teams, and individual staff members, including:

  • Master project definitions for Aerospace Research Centre-hosted research programs (Integrated Autonomous Mobility(IAM), Aeronautical Product Development and Certification (APDC))
  • Technology roadmaps and project selection criteria for Strategic Client Services, including Air Travel Research and Defence Technologies and Sustainment.

Action 1:

  • Master projects defined for IAM, APDC by Program Leaders
  • Technology roadmaps and project selection criteria defined for Strategic Client Services
  • Briefing on expectations regarding research objectives and priorities for all Aerospace Research Centre staff at Town Halls
  • Expectations regarding research objectives and priorities reinforced with follow up communications at a laboratory level.

Aerospace Research Centre Director General

December 2020

3. The Aerospace Research Centre should better articulate and market its unique value proposition for key stakeholder groups (i.e., SMEs, original equipment manufacturers and Tier 1 companies, universities, other government departments, international organizations and associations). The research centre’s specific role and contribution in each of its strategic research areas should also be clearly identified. This should be communicated to both internal and external stakeholders.

low

Response: Accepted

Success with core clients / collaborators has been demonstrated by the high degree of engagement achieved to date. We will build on this approach to target our new priorities (SMEs, academia, other government departments). The stakeholder management strategies in response to Recommendation 4 will further complement our approach.

Action 1: Access NRC corporate business advisory and communications services to develop targeted messaging and marketing materials highlighting the unique roles and expected contributions of the Aerospace Research Centre in each of the four strategic focus areas.

Action 1:

  • Value offerings targeting SME’s, OGD’s and academia are articulated
  • Communications products are developed and available for use
  • Communications products used to foster enhanced engagement with SMEs, academia, OGDs and other research centres

Aerospace Research Centre Operations Director

June 2021

4. The Aerospace Research Centre should identify clear mechanisms (including an identified internal champion/resource) to build strategic and successful relationships with universities, SMEs, and Transport Canada.

Low

Response: Accepted

Growing these relationships has been identified as a priority in the 2019-2024 strategic plan. We plan to follow a team approach involving Aerospace management, IRAP resources and business management resources of the National Programs and Business Services Branch to achieve the objective.

Action 1: Charge the recently-formed Key Account Management team for Transport Canada with developing and nurturing a long term strategic relationship with this priority stakeholder with support from Aerospace and NRC senior management.

Action 2: Develop or select a mechanism and an action plan to facilitate, promote and enhance engagement with one or more Canadian universities to drive innovation in priority areas.

Action 3: Pilot an Aerospace Research Centre-sponsored collaboration network for research in electrification and expand the model to address other strategic focus areas, if successful.

Action 4: Consolidate existing initiatives for SME engagement under the IAM and APDC programs into a single integrated strategy in partnership with IRAP.

Action 1:

  • Key contacts within Transport Canada identified, engagement priorities and objectives are established
  • Evidence of multi-level, regular interaction advancing mutually identified priority areas

Action 2:

  • University engagement action plan defined and a formal collaboration mechanism in place for at least one University

Action 3:

  • Collaboration network for research in electrification is formalized

Action 4:

  • Integrated strategy for SME engagement documented and in use for two research programs

1. Aerospace Research Centre Operations Director

 

 

 

 

 

 

 

2. Director, Aerodynamics Laboratory

 

 

 

 

3 / 4. IAM Program Leader

June 2021

5. The Aerospace Research Centre should identify specific timelines and decision points, along with clear accountabilities for its investments in facilities upkeep, upgrade and replacement.

High

Response: Accepted

Facility sustainment has been identified as a key risk and significant resources have been dedicated to timely completion of Aerospace Research Centre facility reviews. The scale of investment required exceeds the internal capacity of the research centre, and even the NRC itself, so multiple opportunities for cost-sharing and external funding are currently being pursued.

Action 1: Develop a long-term plan for Aerospace Research Centre facility renewal and seek the endorsement of the Aerospace Research Centre Advisory Board, and NRC senior management

Action 1:

  • Aircraft fleet renewal plan submitted to Senior Executive Committee
  • Overall summary of facility renewal requirements and timeline presented to Aerospace Research Centre Advisory Board for external validation
  • Facility sustainment priorities documented in FY2022 Operations Plan
  • Renewal plan for the Aerospace Research Centre’s top 4 highest priority facilities (excluding aircrafts) complete

Aerospace Research Centre Director General

 

 

 

Overall summary of facility renewal requirements and timeline - December 2020

Facility sustainment priorities - June 2021

6. The Aerospace Research Centre should identify and implement actions to reduce stress and overwork for employees.

High

Response: Accepted

Although traditional indicators of overwork (increase in overtime and/or general labour expenditure on projects) show contradictory results, the evaluation findings corroborate the results of the 2018 employee engagement survey and the direct observations of the Aerospace Research Centre management team. We believe that actions already taken are having a beneficial effect (see PSES 2019 results), but it is recognized that more work is needed in this area.

Action 1: Establish an Aerospace human resources committee in order to track, and drive progress on HR initiatives and priorities

Action 2: In consultation with HR, empower employees to undertake a thorough root cause analysis of staff stress and workload, informed by the results of the past three Public Service Employee Surveys

Action 3: Develop and implement an action plan to address issues identified in the root cause analysis.

Action 1:

  • Aerospace Human Resources Committee in operation.

Action 2:

  • Employee-led root cause analysis completed and documented.

Action 3:

  • Action plans developed and tracked (expected improvement on PSES results).

Aerospace Research Centre Director General

 

 

 

 

 

December 2020

 

 

June 2021

7. The Aerospace Research Centre should explore different means or metrics to better capture the impact of its work, especially with regards to environmental benefits.

low

Response: Accepted

Measuring impact is a challenge that we share with all other research centres, due to the nature of the NRC’s mandate. We will support corporate initiatives to improve in this area.

Action 1: Develop mechanisms for tracking projects with environmental impact across all Aerospace Research Centre labs and programs to enable independent, external assessment of outcomes achieved.

Action 1:

  • Tracking mechanism developed and implemented.

Aerospace Research Centre Director General

June 2021

Appendices

Appendix A – Methodology

Document review
Internal and external documents were reviewed to provide context and to complement other lines of evidence in assessing relevance and performance. Internal documents included program business plans, research centre operational and strategic plans, presentations, progress reports, etc. External documents included industry studies, market assessments, as well as documents related to government priorities. Also included in the document review were facility review self-assessments of the Gas Turbines Laboratory, the Flight Research Laboratory, and the Aerodynamics Laboratory. These reviews were completed as part of an NRC-wide facilities review exercise, and were used to inform questions related to capabilities.

Key informant interviews
Interviews were conducted with a total of 34 stakeholders. This included 20 external clients and partners, and 14 research centre senior staff comprised of directors of research and development and program leads. This information was used to complement other lines of evidence and to contextualize quantitative information.

Client survey
A total of 247 external clients and collaborators were invited to complete an emailed client survey which included questions related to the relevance of the research centre activities, satisfaction with the availability of competencies and facilities, and the achievement of outcomes. 63 responded for a response rate of 26%. The breakdown of survey respondents was 83% from industry and 17% from government departments.

Bibliometric study
NRC’s Library and Information Management Services conducted a bibliometric analysis of peer-reviewed publications affiliated with the research centre and indexed in Scopus for the period 2013-2018. This analysis was used to answer questions of scientific excellence.

Data review
Research centre and program administrative and performance data for 2012-13 to 2018-19 were reviewed to provide information on program inputs (i.e., resources), outputs, and client reach. This included financial data, human resource data, project data and intellectual property data.

Peer review
A peer review committee was convened to assess the research centre along four dimensions: relevance, stakeholder engagement, capabilities and performance. The committee was composed of six members with expertise in each of the research centre’s laboratories. Members included national and international representatives from academia, research organizations and industry. To ensure objectivity and avoid conflicts of interest, peer review committee members signed a confidentiality and conflict of interest agreements.

The process included:

  1. Reviewing background material produced by the research centre and by the NRC evaluation team.
  2. Participating in a pre-site visit teleconference to discuss the committee’s initial assessment of the research centre, information gaps and questions.
  3. Participating in a two day site visit to the NRC.
  4. Writing a peer review report.

Limitations and Mitigation Strategies

Availability of interviewees
Despite attempts by the evaluation team to reach certain key external stakeholders, some individuals were not available for interviews within the timelines allowed for the evaluation, and thus could not be interviewed.

Mitigation
Where possible, alternate stakeholders from key client organizations and associations were identified and included in interviews.

Challenges identifying comparable organizations
No comparable organization was identified for the aerospace research centre. Other global research and technology organizations conducting similar research are much larger in size and funding, making a comprehensive comparative analysis ineffective.

Mitigation
Where applicable, organizations working in the same areas, despite differences in size, were included to benchmark performance (or to introduce some degree of comparison). For instance, the bibliometric assessment included comparisons with NRC overall and other international organizations in the field. Some administrative and performance data compared Aerospace Research Centre results to other NRC research centres.

Use of publications to measure scientific excellence
The challenge with bibliometric analysis is that there is a time lag prior to citation of published work. As a result, the actual use of more recent publications is likely underestimated in the current study. Also, not all research is published in academic peer-reviewed journals.

Mitigation
To mitigate these limitations, other lines of evidence were used to assess the excellence and scientific impact of the Aerospace Research Centre, including the use of an expert peer review.

Client survey response bias
The NRC client relationship management system did not include client contact information for all research centre clients and, for some clients, available contact information was for individuals in the organization’s finance or accounts payable units, who would have been unable to answer the survey questions. An email distribution list of clients, maintained by the research centre’s quality management team, was therefore used to reach more appropriate individuals within the client firms. The extent to which this list included all research centre clients over the evaluation period is unknown.

In addition, as with any survey, the fact that only a subset of the entire population responded to the survey introduces possible error into the study, since the respondent population may not entirely reflect the larger total population.

Mitigation
To mitigate this limitation, no findings were based solely on survey results. Survey findings were examined in conjunction with findings from other lines of evidence.

Challenges associated with peer review committee
In order to provide an objective, independent assessment of the research centre, arms-length experts were sought to participate in the peer review process. Given the need to be independent and objective, experts selected for peer review committees may not have in-depth knowledge of the research centre and its activities, or the NRC. As well, each member comes to the peer review exercise with their own experience, expertise and associated biases. Finally, one enlisted peer review committee member was, in the end, unable to attend the site visit, and did not participate in the peer review process.

Mitigation
In order to ensure the needed expertise on the committee, and to select the best possible committee composition, the project team consulted with the research centre’s senior management and advisory board, and with the Knowledge, Information and Technology Services branch, to identify and invite the peer review chair and members. All peer review committee members were vetted and approved by the research centre VP and DG.

To mitigate any bias, in constituting peer review committees, the evaluation team tries to ensure the inclusion of experts from various areas of expertise, including different genders, geographic locations, and representatives from various types of organizations (industry, academia and other governmental organizations). In addition, at least one Canadian representative is sought in order to bring an understanding of the Canadian context. In the case of the Aerospace peer review committee, member expertise covered all five Aerospace Research Centre laboratories, in spite of the loss of the one member prior to the visit, and included representatives from academic institutions, industry and other research-type organizations.

Peer Review Committee members

Dimitri Mavris, Chair
S. P. Langley Distinguished Professor, National Institute of Aerospace; Regents Professor, Boeing Chair in Advanced Aerospace Systems Analysis; Director, Aerospace Systems Design Laboratory, Executive Director, Professional Master’s in Applied Systems Engineering, Georgia Institute of Technology

Yolanda Hicks
Research Engineer, Combustion Branch, NASA Glenn Research Centre

Rakesh Kapania
Mitchell Professor of Aerospace and Ocean Engineering, Virginia Polytechnic Institute and State University

David Rancourt
Assistant Professor, Director of Aerospace, l’Université de Sherbrooke

Jeffrey Stith
Manager, Research Aviation Facility and Senior Scientist, National Centre for Atmospheric Research Earth Observing Laboratory

Alexander Velicki
Structural Design Engineer, Boeing Research and Technology Group

Appendix B – Aerospace research centre logic model

 

 

6. Ultimate
(10+ years)

Economic impacts

6.1 NRC fosters a healthy, vibrant and competitive Canadian aerospace industry

Social impacts

6.2 Improved quality of life for Canadians (e.g., improved in-flight comfort, connectivity and safety)

Environmental impacts

6.3 Improved Canadian environment position (e.g., environmentally friendly aircrafts)

5. Intermediate
(6-10 years)

Advancement of knowledge

5.1 NRC contributes to the future direction of the Aerospace industry

Business innovation

5.2 Creation & growth of Canadian firms (increased competitiveness & productivity)

Advancement of government policies and priorities

5.3 Regulation of aviation in Canada benefits from NRC’s expert contribution

4. Immediate
(1-5 years)

4.1 NRC is recognized as a national leader and hub for aerospace research & technical excellence

4.2 New scientific or engineering discoveries advance aerospace R&TD (aerodynamic, flight research, gas turbines, structures & materials, manufacturing)

4.3 Extended customer base and reach within the Canadian aerospace industry (increased engagement with SME clients)

4.4 NRC is integrated with other government entities (federal, provincial) in matters relating to air transport and the environment

4.5 Scientific / technical advances contribute to effective stewardship of DND’s airborne assets

3. Outputs

Enhanced / maintained capabilities

3.1 Enhanced / maintained competencies

3.2 Enhanced / maintained world-class R&DT infrastructure

Scientific & technical achievements

3.3 Scientific & technical knowledge

3.4 Intellectual property

Collaborations

3.5 Strategic relationships / partnerships

2. Activities

Facilitate scientific, technical & organizational capability

2.1 HR management (expertise / workforce development, staffing & strategic succession planning)

2.2 Facility management (development & management, health and safety)

2.3 Strategic planning (foresight, market intelligence)

2.4 Program & portfolio management / coordination & project management

Build scientific & technical capability

2.5 Self-funded R&TD projects

Deploy scientific & technical capability

2.6 Collaborative R&TD projects

2.7 Technical services

2.8 Participation in scientific / technical committees

1. Inputs

1.1 NRC investment

1.2 Earned revenues

1.3 Contributions from other NRC portfolios & collaborators / partners (in-kind & cash contributions)

1.4 NRC common services & SSC

1.5 People (HQP, specialized expertise & experience)

1.6 Facilities (specialized facilities & equipment)