Evaluation of the Aerospace Research Centre

Information

Status:	Active
Effective date:	June 26, 2020
Prepared by:	Office of Audit and Evaluation, National Research Council Canada
Approval:	NRC's President
Related document:	Summary - Evaluation of the Aerospace Research Centre
ISBN:	978-0-660-34897-1
Alternate format:	Evaluation of the Aerospace Research Centre (PDF, 1.64 MB)

Acronyms

AIAA

American Institute of Aeronautics and Astronautics

AFRL

Air Force Research Laboratory

DLR

Deutsches Zentrum für Luft-und Raumfahrt (German aerospace centre)

DND

Department of National Defence

DST

Defence Science and Technology Grou

FAA

Federal Aviation Administration

FWCI

Field-Weighted Citation Index

GBA+

ender-Based Analysis Plus

ICAS

International Council of the Aeronautical Sciences

IRAP

Industrial Research Assistance Program

NASA

National Aeronautics and Space Administration

NRC

National Research Council of Canada

OGD

Other government department (Canadian federal)

ONERA

Office National d’Études et de Recherches Aérospatiales (French national aerospace research centre)

PSES

Public Service Employee Survey

R&D

Research and Development

SME

Small-to-Medium Enterprise

TRL

Technology Readiness Level

UAV

Unmanned Aerial Vehicle

Executive Summary

The National Research Council of Canada (NRC)’s Aerospace Research Centre supports industry and other government departments to create, advance and apply world-leading aerospace capabilities in Canada for the benefit of society. Its research is organized into five laboratories: Aerodynamics, Aerospace Manufacturing, Flight Research, Gas Turbines, and Structures and Materials Performance. It is one of the NRC’s largest research centres in terms of both budget and staff. This evaluation covered 2012-13 to 2018-19 inclusively, and drew on a bibliometric study, data review, document review, internal and external interviews, a client survey, and a peer review by experts from industry, government, and academia.

Areas of demonstrated strength

Performance

The research centre is an important partner for its industry clients and government partners in contributing to advancements in the field of aerospace. It fosters economic growth for Canadian companies, supports government operations (in particular for the military) as well as policy and regulations, and contributes to social and environmental impacts.

Uniqueness

The research centre’s combined competencies,facilities, and ability to perform end to end research (from conception to in-flight testing) are unique within Canada.

Areas for improvement

Integration and alignment

The research centre’s new areas of focus are well aligned to stakeholder needs. Ensuring that objectives are met in these areas will require increased integration of staff across laboratories and the NRC, as well as alignment of projects to objectives.

Engagement

The research centre has recognized a need to increase collaboration with SMEs, universities, and Transport Canada. These relationships are critical elements of future success.

Capabilities

The research centre has unique and diverse facilities that require significant upgrades and renewal to ensure sustained operations. Prioritization and timely decisions supported by the NRC’s senior management are crucial.

Research centre staff are dedicated and driven but, in recent surveys and interviews, also report being overworked and stressed. The level of stress has improved from last year, according to the PSES survey, but to retain and support current staff, as well as recruit staff to new areas of research, this issue requires further action.

Metrics

Many facets of the research centre’s work on improving aircraft components and materials contribute to downstream environmental benefits, as do much of its atmospheric and UAV work. These important contributions should be captured by the research centre.

Introduction

The evaluation of the National Research Council (NRC)’s Aerospace Research Centre covered the 2012-13 to 2018-19 period. It was carried out in accordance with the NRC’s approved evaluation plan and Treasury Board policies. The research centre was last evaluated in 2010 for the 2004-05 to 2008-09 period.

This report begins by providing a profile of the Aerospace Research Centre. It then presents the evaluation findings on the appropriateness of research, stakeholder engagement, capabilities, scientific excellence and performance. Following the conclusion are seven recommendations for improvements within the research centre.

Evaluation approach

Methods

Mixed methods were used to maximize the generation of useful, valid and relevant evaluation findings. This approach also allowed for convergence of results across methods and contributed to a better understanding ofcomplex issues by exploring different facets.

• Bibliometric study (publication citation analyses)
• Data review (financial, administrative and performance data)
• Document review
• Client survey (26% response rate)
• Internal and external interviews (N=34)
• External expert peer review committee

For more detailed information on the methods, including challenges and limitations, refer to Appendix A.

Questions

The evaluation questions were developed based on consultations and a review of key documents. The questions were:

1. Is the Aerospace Research Centre’s strategic plan appropriate to ensure the continued relevance and success of the Research Centre?
2. Has the Aerospace Research Centre engaged the most appropriate clients and collaborators in the most effective ways?
3. To what extent does the Aerospace Research Centre have the capacities, competencies, and facilities to achieve its objectives?
4. To what extent is the Aerospace Research Centre a leader in scientific excellence in the areas of aerodynamics, flight research, structures and materials, manufacturing and propulsion?
5. To what extent has the Aerospace Research Centre contributed to longer term outcomes?
   1. the economic growth and prosperity of the Canadian aerospace industry?
   2. government operations, policy and regulations?
   3. social and environmental impacts?

Profile

The Aerospace Research Centre provides the aerospace industry and government agencies with ideas, new technologies and the development and demonstration of new products and processes that target the challenges faced by the aerospace sector. The Aerospace Research Centre works in five areas: aerodynamics, propulsion, flight research, structures and materials, and manufacturing.

Research centre structure

Research activities

Consolidation of programs and activities

The Aerospace Research Centre had six active research programs between 2013-14 and 2018-19. Following its strategic planning for 2019-2024, the research centre moved forward with two programs, plus strategic client services, and support to another NRC research centre’s program in advanced manufacturing. Several elements of previous programs and services are integrated into these new programs or the research centre’s strategic client services. For more detailed information on the activities of the research centre, refer to the logic model in Appendix B.

Programs before 2019 (2013-14 to 2018-19)

Civilian Unmanned Aircraft Systems worked to develop and demonstrate the technology necessary to address key impediments to the large-scale adoption of unmanned aerial vehicles.

Aeronautical Product Development Technologies worked to reduce the time that Canadian aerospace companies spend on the development of products by ensuring efficient passage through the certification process.

Working and Travelling on Aircraft worked to accelerate the development of new cabins, environmental controls and avionics to provide airline passengers with a comfortable yet economical travel experience.

Air Defence Systems provided the dominant source of aerospace research and development for the Department of National Defence.

Reducing Aviation Icing Risk worked to develop technologies and processes that would reduce the costs of production and icing product qualification; and helped regulators to develop standards and tools for demonstrating compliance with these standards.

Aeronautics for the 21st Century aimed to de-risk the innovation process for aerospace original equipment manufacturers in areas such as manufacturing and fuel efficiency. The program ended in 2016.

New programs and services (2019 to 2024)

Aeronautical Product Development and Certification Program aims to reduce the time-to-market for aerospace products by solving complex problems of product development and certification, with a focus on aircraft icing and innovative methods for physical and “virtual” testing.

Integrated Autonomous Mobility Program seeks to accelerate technology development in unmanned aerial mobility and assist with the development of certification and qualification standards.

Support to other NRC programs

The research centre continues to support other programs such as the Advanced Manufacturing Program (housed within the NRC’s Automotive and Surface Transportation Research Centre) as it aims to increase manufacturing, efficiency and agility by using emerging technologies.

Strategic client services

Defence technologies and sustainment intends to enhance defence platform capabilities, mission readiness and fleet affordability and sustainability of the Royal Canadian Airforce.

Air travel research aims to improve the end-to-end travel experience, looking at issues related to passenger comfort, health and safety and aircrew efficiency and fatigue.

Other technical services and research and development projects (including Ideation) are also conducted under strategic client services.

Financial resources

Revenue and expenses

The Aerospace Research Centre is one of the NRC’s largest research centres in terms of the number of staff and facilities. It is also the NRC’s top revenue generating research centre, earning $196.7 million over the past six years with direct expenses totaling $193.4 million. Between 2013-14 and 2018-19, the research centre earned an average of $32.8 million per fiscal year.

Over the past three years (2016-17 to 2018-19), the research centre generated more revenues than planned, with its highest earnings in 2018-19 ($37.8 million).

Human resources

Workforce

In 2018-19, the Aerospace Research Centre employed 354 employees across five laboratories. Of these, 152 were research council officers or research officers and 106 were technical officers. The majority (84%) of research centre staff were permanent fulltime employees. Most staff were located at the research centre’s four Ottawa laboratories, with the balance in Montréal at the Aerospace Manufacturing Technology Centre. The Flight Research Laboratory had the largest number of employees with 76 staff.

Gender, diversity and inclusion

When compared to workforce availability, the Aerospace Research Centre has good representation of women in professional and administrative roles. However, women are under-represented in management and technical officer positions. Visible minorities are well-represented in professional roles but under-represented in management, administrative and technical roles. Indigenous people and people with disabilities are under-represented in all categories in the Aerospace Research Centre, as is the case across the NRC.

Projects and clients / collaborators

The Aerospace Research Centre undertook 714 unique projects between 2013-14 and 2018-19. Most of the research centre’s work, 518 projects (73%), focused on the provision of technical services with 191 projects (27%) focused on strategic research and development (R&D).

Industry

• 86% (292) of the research centre’s clients
• 47% Canadian and 39% foreign
• Responsible for 56% of the research centre’s revenues

Other Government Departments

• 7% (25) of the research centre’s clients
• 3% Canadian, 1% provincial and 2% foreign (does not add up to 7% due to rounding)
• Responsible for 38% of the research centre’s revenues

Academia

• 4% (13) of the research centre’s clients
• 3% Canadian and 1% foreign

Other

• 3% (10) of the research centre’s client
• Includes non-profit organizations such as Speed Skating Canada

Appropriateness of research

Overall finding: The Aerospace Research Centre’s new strategic focus on emerging technology areas is well-aligned with industry and government needs. To ensure clarity of purpose and alignment of efforts within the research centre, increased communication and direction to staff is required.

Uniqueness

Sources: Document review, client survey, internal and external interview, peer review committee

Similar areas of research compared to other research and technology organization

The Aerospace Research Centre’s areas of research are similar to those of other major research and technology organizations, such as NASA (US), DLR (Germany), DSTL (UK), and ONERA (France). However those organizations are much larger in size, scope and budget.

End to end capabilities unique in Canada

Within Canada, the combination of research expertise, capabilities and facilities provided by the research centre is unique. Providing the ability to conduct research from models to in-flight testing is viewed positively by the research centre’s clients.

Safeguarding Canadian information

Canadian entities like Bombardier and the Department of National Defence rely on the Aerospace Research Centre, as opposed to other organizations, for their research and testing needs. As part of the government of Canada, the NRC is better able to ensure the safeguarding of their intellectual property and national defence secrets.

"I have a great deal of confidence in the facility at NRC to keep the data confidential. There aren’t any other facilities in Canada that could provide the data we need."

Relevance

Sources: Document review, client survey, internal and external interview, peer review committee

Industry needs and trends

The Aerospace Research Centre’s new strategic focus on disruptive technologies aligns well with identified sector needs in the emerging areas of:

• digital twin
• additive / subtractive manufacturing
• autonomous aerial mobilit
• disruptive architecture (includes electrification)

The peer review committee supported the research centre’s strategic alignment to these areas provided that is:

• Identify, clearly communicate, and limit focus to the areas where it can add the most value in these highly populated research areas
• Further integrates its five laboratories to address these aerospace problems in a unified and interdisciplinary way

Government priorities and needs

The Aerospace Research Centre’s work aligns with the priority the federal government places on the aerospace industry as a driver of domestic research and development. For example, the research centre’s work aligns well with three clusters (Digital Technology, Next Generation Manufacturing and Artificial Intelligence-Powered Supply Chains). It also supports major federal investments under the Strategic Innovation Fund.

The Aerospace Research Centre is supporting federal departments through its work. In particular, it is helping the Canadian military with the maintenance and repair of its aging aircraft, development of new technologies, and procurement of new equipment and technologies. The research centre also plays a key supporting role to Transport Canada in the provision of evidence required to certify and regulate the aerospace sector. Transport Canada’s focus on the certification of new manufacturing methods and the regulation of unmanned aerial vehicles also require close collaboration with the Aerospace Research Centre.

Competing priorities

Sources: Document review, client survey, internal and external interview, peer review committee

Technical services

Activities within the Aerospace Research Centre are heavily focused on the provision of technical services or technology maturation. This is mainly due to the following realities:

• Companies and OGDs in the aerospace sector mainly require high TRL support from the research centre
• Technical services lend themselves better to higher Technology Readiness Level (TRL) work
• Of note, plans to increase collaboration with universities were encouraged by the peer review committee –especially with regards to new areas like autonomous mobility where lower TRL research will be undertaken.
• Technical services generate significant revenue needed for the research centre to achieve its financial targets.
• Technical services drive utilization of the research centre’s large facilities (e.g., wind tunnels) and contribute to the recovery of their significant operational and maintenance costs

The Aerospace Research Centre is aware of the need to conduct strategic research and development projects within this context and aims to direct such projects within its four priority areas.

Revenue targets

The NRC has moved away from a focus on revenue growth and has stabilized revenue targets in recent years. However, staff continue to feel pressure to meet these targets, and continue to exceed them. In a record-breaking year in 2018-19, the research centre exceeded its revenue target by $6.4 million or 20%.

Communication

Achieving a balance, between technical services and strategic research priorities, that works within existing constraints is an ongoing struggle for the Aerospace Research Centre’s management team. Research centre staff would benefit from clear communication of expectations and priorities from senior management. Consistency of messaging when selecting and approving projects or research initiatives would also serve to reinforce these priorities.

Stakeholder engagement

Overall finding: The Aerospace Research Centre has effectively engaged with its major industry clients and the majority of its other government departments. While the research centre plans to enhance outreach to, and collaboration with, universities and SMEs to achieve its objectives, the peer review committee found that there is no clear plan or mechanism to ensure successful engagement with these two stakeholder groups.

Clients and collaborators

Sources: Data review, client survey, internal and external interviews

Overall, the research centre’s project revenues are concentrated around a small number of major projects from ten clients.

Industry: The Aerospace Research Centre works with leading Canadian aerospace companies and is well-connected to large original equipment manufacturers and Tier 1 suppliers*. The research centre has also maintained relationships with key industry associations.

*Original equipment manufacturers (OEMs) are companies whose goods are used as components in the products of another company. Tier 1 suppliers supply parts or systems directly to OEMs.

Government departments and regulators: The Department of National Defence was by far the research centre’s largest revenue generating client. Client interviews indicate that a positive and collaborative relationship has developed, making the research centre an engaged enabler that helps the department push boundaries.

Collaborations with Transport Canada and Environment and Climate Change Canada have taken place, and continue to be essential on several different fronts including anti-icing research, alternative fuels, oil sands monitoring, autonomous vehicle regulations, and aircraft certification.

The research centre also collaborates with the American (Federal Aviation Administration) and European (European Union Aviation Safety Agency) certification and regulation bodies in several safety areas. The relationships are positive and the research centre is viewed as a respected and important contributor.

Engagement and awareness

Sources: Data review, client survey, internal and external interviews

Engagement

The Aerospace Research Centre has positive relationships with clients and collaborators.

Awareness

The research centre is well known within the areas of aviation safety, aircraft research and engine icing. Most clients surveyed (74%) rated awareness of the research centre as medium to high.

Clients indicated the following activities would have the greatest impact on increasing knowledge of the research centre and the services it provides:

• Increased participation in academic conferences
• More frequent outreach by research centre executives
• Greater participation of staff in national and international committees

The peer review committee supported the need for an increase in these activities, indicating that researchers may not be as well-known globally as they should be.

There is “low international awareness of some of the research centre’s areas of expertise and facilities that should be better known.”

Increased engagement needed with key stakeholder groups

Sources: Data review, client survey, internal and external interviews

Universities Academic institutions represented a small proportion of the research centre’s clients and collaborators over the evaluation period. This is not unique to the Aerospace Research Centre as there is a desire across the NRC to increase engagement with academia, though there are not yet any clear mechanisms to do so. The Aerospace Research Centre’s new strategic plan prioritizes university collaborations, especially to fill capacity gaps and connect with research at lower TRLs to provide a pipeline for future projects. The peer review committee indicated this relationship is critical and recommended the identification of “a dedicated resource to be the focal point for the strategic development and support of relationships with university faculties.”

Small and Medium Enterprises (SMEs) represent only a small portion (12%) of the research centre’s Canadian industry clients. The research centre is working to increase this number through collaboration with the National Research Council Industrial Research Assistance Program (NRC IRAP). However, the peer review committee encouraged the research centre to better articulate its value-add and increase visibility/promote its work to attract SMEs.

Transport Canada (TC) is a key department with whom the research centre needs to solidify collaborations. The success of the research centre’s two new programs (Integrated Autonomous Mobility and Aeronautical Product Development and Certification) is directly tied to TC regulations and certification. To date, interactions have been more productive for autonomous mobility than certification. Engagement with Transport Canada at the strategic level has not been particularly successful. The research centre acknowledges the importance of its relationship with TC, and is working towards an MOU with the department. However, progress on this has been slow.

Universities: “There does not seem to be a clear understanding, within the research centre, of how to collaborate with universities and academic researchers.”

SMEs: “It was unclear to members of the committee whether there is a specific communication plan to reach SMEs and to promote the research centre’s capabilities and the available financial incentives.”

Capabilities

Overall finding: The Aerospace Research Centre has had the capacities, competencies and facilities to achieve its objectives, but requires continued action as it plans to shift competencies to address new strategic objectives and address overwork and stress. Timely decisions with regards to facilities (i.e., which to keep, upgrade or divest) and the aircraft fleet (i.e., matching current and future research needs to aircraft, time and cost of instrumenting and training) are critical to the achievement of the research centre’s objectives. Facility investments and a clearly communicated way forward are essential.

Capacity

Sources: Data review, document/literature review, internal interviews, peer review

Adequate capacity

The Aerospace Research Centre has a current count of 354 employees. Generally, clients are satisfied with staff capacity. Clients praised their responsiveness and flexibility to put in the hours and work necessary to successfully complete projects.

However, overwork and stress were a concern expressed by nine out of the ten senior managers interviewed. Employees are also experiencing high levels of stress. Interviews indicated that publications or strategic research projects are sometimes done on employees’ own time. The peer review committee noted that “The current workload provides little time, if any, for writing publications, participating in technical associations, and conducting (possibly) more rewarding research.” The Public Service Employee Survey (PSES) also identified workload as an issue.

Stressors

Although 71% of the Aerospace Research Centre’s respondents to the 2019 Public Service Employee Survey (PSES) indicated they have support at work to balance their work and personal life, they also highlighted several important stressors. In particular, only 55% of the research centre’s employees reported that they can complete their assigned workload during their regular working hours (although this reflects an improvement from the 2018 result of 51%). Administrative workload was identified as a contributing factor to employees’ stress in sessions conducted by the research centre following the 2017 PSES results. Several actions have since been taken by the research centre to address various elements of this administrative burden.

While the research centre is aware that change is needed and, as mentioned above, some actions have already been taken, more work is needed. This may be challenging given the research centre’s dependency on revenue generation, competing priorities, and a general aerospace culture of working long hours.

PSES Results	2017	2018	2019	2019 NRC Average
I have support at work to balance my work and personal life	74%	71%	1%	78%
I can complete my assigned workload during my regular working hours	67%	51%	55%	68%

Competencies

Sources: Data review, document review, client survey, internal and external interviews, peer review committee

Competent and adaptable staff

Overall, research centre clients were highly satisfied with the expertise of staff. Staff are considered highly competent and even world class experts in certain areas such as engine icing and advanced manufacturing.

The peer review committee found that the research centre has a “highly productive and flexible workforce with unique technical capabilities.”

Competing for resources in new areas

It is imperative that the research centre continue to develop competencies in its new areas of focus. In particular, the peer review committee highlighted the importance of developing competencies in high performance computing, machine learning, and multidisciplinary design optimization. However, the research centre’s ability to attract and retain staff may be hampered by fierce competition across the aerospace industry in Canada, where there is a large shortage of qualified graduates. Additionally, if unaddressed, workload and stress issues affecting current research centre staff will further affect recruitment and retention of a new generation of researchers who value work-life balance.

Increased collaboration with universities and with other NRC research centres, especially with regards to digital technologies and advanced manufacturing, will be important to the Aerospace Research Centre’s strategic success. However, as expertise in digital technologies is in high demand across NRC research centres, and advanced manufacturing capacity is growing across Canada, the research centre may also face internal competition for resources.

Facilities

Sources: Data review, document review, client survey, internal and external interviews, peer review committee

Unique facilities

Many of the Aerospace Research Centre facilities have unique capabilities. For example:

• Altitude Icing Wind Tunnel is unique in simulating altitudes up to 40,000 feet and temperatures down to -40ºC.
• Gas Turbine Engine Test Facility has state-of-the-art icing test systems accepted for engine certification by major regulators.
• Convair 580 the NRC’s world-class airborne lab, is used internationally for icing and satellite research.

Overall, the research centre has the types of facilities needed to fulfil the project objectives of their clients and they are price competitive with their peers.

Aging infrastructure

Many of the research centre’s facilities are several decades old. While the majority have been well maintained and are still functioning well, they require regular maintenance and upgrades. For example, in the next 5 to 10 years, the Aerodynamics, Gas Turbine and Flight Research Laboratories together, will require capital investments ranging from $24 to $60 million. This includes major investments in wind tunnels, engine test facilities and aircraft. Additionally, the replacement of aircraft, such as the Convair 580 (a key aircraft used to support a wide range of research projects) require much advanced planning due to the significant financial investments required to purchase, re-instrument, and train staff. This is a multi-year, multi-million-dollar project.

Utilization

While some facilities are heavily utilized, a number of the facilities are underutilized. Fluctuating demand due to industry cycles is a contributing factor, as is a lack of technical staff. For example, several wind tunnels share staff, as do some aircraft, so when one facility is in use another must sit idle.

Multiple inputs into facilities decision-making

In addition to the current evaluation, the research centre has received, and will receive additional recommendations and advice regarding its facilities, namely through the NRC-wide facilities review process and its own fleet renewal initiative. These multiple reviews on different schedules, combined with the special financial planning required to fund aircraft acquisitions, and facilities maintenance and upgrades, add complexity to an already multifaceted process. The evaluation did not find evidence of a plan to integrate these, nor a detailed timeline on decision-making by the research centre and the NRC. Nor was there evidence of how the research centre will implement any identified actions.

Performance

Overall finding: In areas of icing and materials fatigue, the research centre is a leader in scientific excellence, as measured by publications. Using this same metric (publications), the research centre is not a leader in scientific excellence in other areas. However, this is to be expected given its operating model, which emphasizes technology maturation.

There are multiple examples of how the Aerospace Research Centre has contributed to its longer-term outcomes of fostering economic growth and prosperity of the Canadian aerospace industry; advancing government operations, policy and regulations; and social and environmental impacts.

Scientific excellence

Sources: Bibliometric study, document review, peer review committee

Publications

Bibliometric analysis showed that the research centre publishes less and is cited less frequently than the NRC as a whole (FWCI of 1.5) and its global comparators. The research centre’s publication output dropped over the evaluation period, from a high of 129 publications in 2013 to a low of 94 publications in 2018.

In two areas, gas turbines and aerodynamics, the research centre’s FWCI is above the Canadian and Global FWCI.

Awards

Several Aerospace Research Centre researchers have won prestigious awards for their work. These include awards from NASA, NATO and Manitoba Aerospace for the research centre’s work on engine icing. Other researchers have won NATO awards for their work on gas turbine engines, structural fatigue in military aircraft and the aerodynamics of helicopter launch and recovery.

“Publishing should be viewed as both an asset to the Aerospace Research Centre to promote its visibility, and as a concrete way to give the researchers a sense of accomplishment.”

Sources: Bibliometric study, document review, peer review committee

The peer review committee found that the Aerospace Research Centre has demonstrated excellence in:

• Reduction in icing risk
• Hyper-spectral sensing
• Special use aircraft that perform several types of reconnaissance for the Government of Canada
• Fatigue and fracture studies for the Canadian Defense Forces

However, the committee also noted that due to the Aerospace Research Centre’s model of higher TRL and revenue-generating projects, the greater aerospace community may not view the research centre as world class. The committee noted in particular that the research centre publishes less papers and technical memoranda than its peers within NASA, for example. Combined with lower attendance at scientific conferences and fewer university collaborations, the committee warned that this may prevent researchers and the Aerospace Research Centre itself from enhancing a reputation for scientific excellence.

Industry growth and prosperity

Sources: Bibliometric study, document review, peer review committee

Developing new technologies

The research centre has assisted clients in developing new or improved products or solutions. That assistance has been important in de-risking new aircraft development and cabin configurations. New technologies have been developed, for example, in:

• Icing: ultrasound ice accretion sensor to detect ice build-up on aircraft engines
• UAVs: UAV sense and avoid prototype technologies
• Manufacturing: automated welding and machining of next generation fuel injector assembly, and cold spray structural repair

Commercializing new technologies

There are several examples of new technologies commercialized by the research centre:

• NRC patented profilometer: a tool that provides faster and more accurate inspection of large aircraft components was licensed to an industrial engineering group
• Particle detection probe: a device designed to detect dangerous ice crystals found at high altitudes on aircraft engines has been commercialized by an American company

Company growth

Examples of the research centre’s impact on company growth are:

• An international original equipment manufacturer experienced a 20% growth in employment after working with the research centre on product development
• A Canadian Tier 1 company won a $100 million contract, in part, due to its collaboration with the research centre

Improved productivity and cost savings

Several of the research centre’s clients indicated that through optimization of products or processes, and model testing, they reduced costs or improved productivity.

Government solutions

Sources: Bibliometric study, document review, peer review committee

Support for DND

The Department of National Defence (DND) is the research centre’s largest single client. As the following examples show, the research centre’s projects have extended the life of military equipment and paved the way for significant cost savings.

• Fatigue Life Extension Test for CF-18 Horizontal Stabilizer: allowed the military to extend the life of stabilizers across the CF-18 fleet which are estimated to cost more than $100 million to replace.
• Ship-Helicopter Operating Limits: Modeling and simulation tools contributed to more than $10 million in costs savings and months of time saved for flight trials and de-risking ship design. This was highlighted as a valuable contribution by the peer review committee.
• Residual Strength Testing of CF188 Aileron: The research centre’s tests extended the retirement time of the aircraft aileron, a hinged control surface on an aircraft wing. Potential savings are estimated at up to $24 million.

Air mobility

The research centre has an ongoing relationship with Transport Canada to provide advice and generate the scientific data required to inform the development of regulations for drones and unmanned aerial vehicles.

Icing

The Aerospace Research Centre’s work on aircraft icing has had a positive impact on aviation safety and aircraft certification in Canada and around the world. Its collaboration with Environment and Climate Change Canada provided key data for the development of a new appendix by the United States Federal Aviation Administration (FAA) on the operation of aircraft engines under high altitude conditions. As well, the research centre’s anti-icing fluid testing contributes to Transport Canada updates of regulated holdover times for aircraft to reflect new anti-icing fluids. These holdover timetables are used internationally to determine how long an aircraft can be on the ground following de-icing.

Fuel

The research centre’s work with Transport Canada on the FAA’s Piston Aviation Fuel Initiative could eventually lead to policy changes by Environment and Climate Change Canada to ban lead in aviation fuel, as Canada has done with other vehicles. Its work with DND on alternative aviation fuels has also allowed the Canadian Airforce to maintain interoperability with its allies who are increasing the use of alternative fuels.

Social and environmental impacts

Sources: Bibliometric study, document review, peer review committee

Social/health impacts

Multiple research centre projects with DND have the potential to contribute to better health outcomes for military personnel. For example:

• Verification of Personal Protective Equipment: the research centre used computed radiography to assess whether ceramic body armour plates used by Canadian Special Operations Forces were in safe condition
• Airforce Hearing Protectors: NRC research led to policy updates at DND that could reduce the incidence of noise-induced hearing loss in the Canadian Forces

Gender-Based Analysis Plus (GBA+)

The Aerospace Research Centre opened a new facility in 2018. The Centre for Air Travel Research is a unique facility that allows researchers to examine the passenger travel experience from arrival at the airport through a simulated flight including lighting, noise and vibration. This facility is now being used for leading edge collaborative research with partners such as Carleton University and Obesity Canada on the diverse needs of air travelers from older adults to people living with obesity.

What is GBA+?

“GBA+ is an analytical process used to assess how diverse groups of women, men and non-binary people may experience policies, programs and initiatives. The “plus” in GBA+ acknowledges that GBA goes beyond biological (sex) and socio-cultural (gender) differences. We all have multiple identity factors that intersect to make us who we are; GBA+ also considers many other identity factors, like race, ethnicity, religion, age, and mental or physical disability.”

Environmental impacts

The Aerospace Research Centre does not systematically track environmental benefits, such as green house gas emissions reductions realized through its work. However, much of its research focusing on reducing the cost of air transport through lighter materials and more fuel-efficient engines will translate into reduced greenhouse gas emissions. For example:

• The Aerospace Research Centre achieved a 30% reduction in fuselage weight through manufacturing using composite structures
• The introduction of a finlet on C130 and C17 military aircraft reduced fuel consumption by up to 5%

Increasing use of unmanned aerial vehicles (UAVs) for many applications also has the potential to reduce emissions as these can be used instead of larger piloted aircraft, which use more fuel. However, UAVs could also create new environmental issues, for example from noise pollution. The research centre is working on how to reduce the noise.

Finally, the research centre is adopting more sustainable manufacturing processes. For example, by replacing the use of cutting fluids in machine processes with liquid nitrogen or micro-lubrication, the research centre is reducing the risk of environmental contamination.

Conclusions and recommendations

Conclusions

Relevance

The Aerospace Research Centre provides important support to the aerospace sector in terms of research and technology maturation. In particular, the research centre provides capabilities to other government departments and Canadian companies that are unique in Canada. Its new strategic focus aimed to develop emerging areas (i.e., digital twin, additive/subtractive manufacturing, autonomous aerial mobility and disruptive architectures) is relevant. To ensure success, the research centre must clearly identify its value-add, communicate its priorities with regards to strategic research, and ensure greater integration of its five laboratories.

Capacity

Although the research centre has been meeting the needs of its clients and exceeding its revenue targets, employees have identified workload and a lack of capacity as factors causing stress. The level of stress has improved from last year, according to the PSES survey, but further action is required. As well, to achieve its new strategic priorities, the research centre needs to recruit new staff in areas that touch on digital technologies and artificial intelligence. As the entire aerospace industry faces a shortage of workers, the research centre is looking, in part, to universities to help fill the gap. However, this requires strong partnerships, which the research centre has not yet developed.

With regards to facilities, the research centre has several unique and important facilities that provide important support to clients and partners. However, as multiple reviews provide information on facility upgrades and replacements needed, along with costing and usage data, the research centre needs to establish a clear pathway to decision-making that will ensure the research centre is able to successfully meet its objectives without delay.

Performance

There are several notable examples of excellent research among the Aerospace Research Centre’s projects, both published and unpublished. Although publications are not the only mark of scientific excellence, they are an important element in knowledge dissemination and international reputation. If the research centre wishes to be world-leading for its work, as indicated in its mission statement, it needs to increase its publications and presence on international committees and at conferences. This will also serve the purpose of helping to retain and attract top talent within its fold.

The Aerospace Research Centre is achieving its objectives. Aerospace industry clients and collaborators respect and appreciate the research centre’s contribution in ensuring they are developing leading edge, quality and safe aircraft components, systems, and materials. Government departments also rely on the Aerospace Research Centre for high quality scientific data to support their work, including regulations. In particular, the Canadian military has benefited greatly from an operational perspective in the maintenance and enhancement of its aircraft. Additionally, much of the research centre’s work contributes to social and environmental benefits, though this is not systematically recorded or quantified due to the often indirect nature of the work.

Recommendations

Management Response and Action Plan

 

Recommendation	Risk-level Associated with not Addressing Recommendation	Management Response	Measure of Achievements	Proposed Person(s) Responsible	Expected Date of Completion
1. The Aerospace Research Centre should increase collaboration and integration across its five laboratories and other research centres to facilitate interdisciplinary work and integrated solutions to future aerospace problems.	low	Response: Accepted The Aerospace Research Centre recognizes this opportunity to exploit a key differentiating capability of the NRC. Action 1: Drive interdisciplinary collaboration across the five laboratories using Aerospace Programs: Integrated Autonomous Mobility (IAM), Aeronautical Product Development and Certification (APDC). Master projects within programs are designed to achieve these goals. Action 2: Review scoring metrics for project selection under the Aerospace Futures Initiative to determine if weighting for inter-laboratory collaboration should be increased.	Action 1: IAM and APDC Program leaders are integrated members of the Aerospace management team Master project outcomes are developed and communicated across the five laboratories and other relevant NRC Research Centres A minimum of three technical communities of practice focused on niche technology areas are designed and launched with participation from Aerospace labs, across NRC and stakeholders. Action 2: Scoring metrics reviewed New weighting used for project selection (if applicable)	Aerospace Research Centre Director General                               Aerospace Research Centre Operations Director	June 2021
2. The Aerospace Research Centre’s management team should clearly articulate expectations regarding research objectives and priorities to staff. Project selection and prioritization (including technical services vs. strategic research services) should align with these expectations and the research centre’s strategic objectives.	Medium	Response: Accepted Research Centre’s top-level planning documents, in particular, the 2019-2024 Strategic Plan and annual Operations Plans define how resources can be deployed to direct available capacity for internally funded/ collaborative work toward long-term priorities, while continuing to respect our revenue commitments and support industry needs. We will continue to take all available opportunities to ensure regular and clear communication of these plans with staff. Action 1: Finalize and communicate lower-level plans that translate research centre-level objectives and strategies into actions that may be more meaningful to lab teams, and individual staff members, including: Master project definitions for Aerospace Research Centre-hosted research programs (Integrated Autonomous Mobility(IAM), Aeronautical Product Development and Certification (APDC)) Technology roadmaps and project selection criteria for Strategic Client Services, including Air Travel Research and Defence Technologies and Sustainment.	Action 1: Master projects defined for IAM, APDC by Program Leaders Technology roadmaps and project selection criteria defined for Strategic Client Services Briefing on expectations regarding research objectives and priorities for all Aerospace Research Centre staff at Town Halls Expectations regarding research objectives and priorities reinforced with follow up communications at a laboratory level.	Aerospace Research Centre Director General	December 2020
3. The Aerospace Research Centre should better articulate and market its unique value proposition for key stakeholder groups (i.e., SMEs, original equipment manufacturers and Tier 1 companies, universities, other government departments, international organizations and associations). The research centre’s specific role and contribution in each of its strategic research areas should also be clearly identified. This should be communicated to both internal and external stakeholders.	low	Response: Accepted Success with core clients / collaborators has been demonstrated by the high degree of engagement achieved to date. We will build on this approach to target our new priorities (SMEs, academia, other government departments). The stakeholder management strategies in response to Recommendation 4 will further complement our approach. Action 1: Access NRC corporate business advisory and communications services to develop targeted messaging and marketing materials highlighting the unique roles and expected contributions of the Aerospace Research Centre in each of the four strategic focus areas.	Action 1: Value offerings targeting SME’s, OGD’s and academia are articulated Communications products are developed and available for use Communications products used to foster enhanced engagement with SMEs, academia, OGDs and other research centres	Aerospace Research Centre Operations Director	June 2021
4. The Aerospace Research Centre should identify clear mechanisms (including an identified internal champion/resource) to build strategic and successful relationships with universities, SMEs, and Transport Canada.	Low	Response: Accepted Growing these relationships has been identified as a priority in the 2019-2024 strategic plan. We plan to follow a team approach involving Aerospace management, IRAP resources and business management resources of the National Programs and Business Services Branch to achieve the objective. Action 1: Charge the recently-formed Key Account Management team for Transport Canada with developing and nurturing a long term strategic relationship with this priority stakeholder with support from Aerospace and NRC senior management. Action 2: Develop or select a mechanism and an action plan to facilitate, promote and enhance engagement with one or more Canadian universities to drive innovation in priority areas. Action 3: Pilot an Aerospace Research Centre-sponsored collaboration network for research in electrification and expand the model to address other strategic focus areas, if successful. Action 4: Consolidate existing initiatives for SME engagement under the IAM and APDC programs into a single integrated strategy in partnership with IRAP.	Action 1: Key contacts within Transport Canada identified, engagement priorities and objectives are established Evidence of multi-level, regular interaction advancing mutually identified priority areas Action 2: University engagement action plan defined and a formal collaboration mechanism in place for at least one University Action 3: Collaboration network for research in electrification is formalized Action 4: Integrated strategy for SME engagement documented and in use for two research programs	1. Aerospace Research Centre Operations Director               2. Director, Aerodynamics Laboratory         3 / 4. IAM Program Leader	June 2021
5. The Aerospace Research Centre should identify specific timelines and decision points, along with clear accountabilities for its investments in facilities upkeep, upgrade and replacement.	High	Response: Accepted Facility sustainment has been identified as a key risk and significant resources have been dedicated to timely completion of Aerospace Research Centre facility reviews. The scale of investment required exceeds the internal capacity of the research centre, and even the NRC itself, so multiple opportunities for cost-sharing and external funding are currently being pursued. Action 1: Develop a long-term plan for Aerospace Research Centre facility renewal and seek the endorsement of the Aerospace Research Centre Advisory Board, and NRC senior management	Action 1: Aircraft fleet renewal plan submitted to Senior Executive Committee Overall summary of facility renewal requirements and timeline presented to Aerospace Research Centre Advisory Board for external validation Facility sustainment priorities documented in FY2022 Operations Plan Renewal plan for the Aerospace Research Centre’s top 4 highest priority facilities (excluding aircrafts) complete	Aerospace Research Centre Director General	Overall summary of facility renewal requirements and timeline - December 2020 Facility sustainment priorities - June 2021
6. The Aerospace Research Centre should identify and implement actions to reduce stress and overwork for employees.	High	Response: Accepted Although traditional indicators of overwork (increase in overtime and/or general labour expenditure on projects) show contradictory results, the evaluation findings corroborate the results of the 2018 employee engagement survey and the direct observations of the Aerospace Research Centre management team. We believe that actions already taken are having a beneficial effect (see PSES 2019 results), but it is recognized that more work is needed in this area. Action 1: Establish an Aerospace human resources committee in order to track, and drive progress on HR initiatives and priorities Action 2: In consultation with HR, empower employees to undertake a thorough root cause analysis of staff stress and workload, informed by the results of the past three Public Service Employee Surveys Action 3: Develop and implement an action plan to address issues identified in the root cause analysis.	Action 1: Aerospace Human Resources Committee in operation. Action 2: Employee-led root cause analysis completed and documented. Action 3: Action plans developed and tracked (expected improvement on PSES results).	Aerospace Research Centre Director General	December 2020     June 2021
7. The Aerospace Research Centre should explore different means or metrics to better capture the impact of its work, especially with regards to environmental benefits.	low	Response: Accepted Measuring impact is a challenge that we share with all other research centres, due to the nature of the NRC’s mandate. We will support corporate initiatives to improve in this area. Action 1: Develop mechanisms for tracking projects with environmental impact across all Aerospace Research Centre labs and programs to enable independent, external assessment of outcomes achieved.	Action 1: Tracking mechanism developed and implemented.	Aerospace Research Centre Director General	June 2021

Appendices

Appendix A – Methodology

Document review
Internal and external documents were reviewed to provide context and to complement other lines of evidence in assessing relevance and performance. Internal documents included program business plans, research centre operational and strategic plans, presentations, progress reports, etc. External documents included industry studies, market assessments, as well as documents related to government priorities. Also included in the document review were facility review self-assessments of the Gas Turbines Laboratory, the Flight Research Laboratory, and the Aerodynamics Laboratory. These reviews were completed as part of an NRC-wide facilities review exercise, and were used to inform questions related to capabilities.

Key informant interviews
Interviews were conducted with a total of 34 stakeholders. This included 20 external clients and partners, and 14 research centre senior staff comprised of directors of research and development and program leads. This information was used to complement other lines of evidence and to contextualize quantitative information.

Client survey
A total of 247 external clients and collaborators were invited to complete an emailed client survey which included questions related to the relevance of the research centre activities, satisfaction with the availability of competencies and facilities, and the achievement of outcomes. 63 responded for a response rate of 26%. The breakdown of survey respondents was 83% from industry and 17% from government departments.

Bibliometric study
NRC’s Library and Information Management Services conducted a bibliometric analysis of peer-reviewed publications affiliated with the research centre and indexed in Scopus for the period 2013-2018. This analysis was used to answer questions of scientific excellence.

Data review
Research centre and program administrative and performance data for 2012-13 to 2018-19 were reviewed to provide information on program inputs (i.e., resources), outputs, and client reach. This included financial data, human resource data, project data and intellectual property data.

Peer review
A peer review committee was convened to assess the research centre along four dimensions: relevance, stakeholder engagement, capabilities and performance. The committee was composed of six members with expertise in each of the research centre’s laboratories. Members included national and international representatives from academia, research organizations and industry. To ensure objectivity and avoid conflicts of interest, peer review committee members signed a confidentiality and conflict of interest agreements.

The process included:

1. Reviewing background material produced by the research centre and by the NRC evaluation team.
2. Participating in a pre-site visit teleconference to discuss the committee’s initial assessment of the research centre, information gaps and questions.
3. Participating in a two day site visit to the NRC.
4. Writing a peer review report.

Limitations and Mitigation Strategies

Availability of interviewees
Despite attempts by the evaluation team to reach certain key external stakeholders, some individuals were not available for interviews within the timelines allowed for the evaluation, and thus could not be interviewed.

Mitigation
Where possible, alternate stakeholders from key client organizations and associations were identified and included in interviews.

Challenges identifying comparable organizations
No comparable organization was identified for the aerospace research centre. Other global research and technology organizations conducting similar research are much larger in size and funding, making a comprehensive comparative analysis ineffective.

Mitigation
Where applicable, organizations working in the same areas, despite differences in size, were included to benchmark performance (or to introduce some degree of comparison). For instance, the bibliometric assessment included comparisons with NRC overall and other international organizations in the field. Some administrative and performance data compared Aerospace Research Centre results to other NRC research centres.

Use of publications to measure scientific excellence
The challenge with bibliometric analysis is that there is a time lag prior to citation of published work. As a result, the actual use of more recent publications is likely underestimated in the current study. Also, not all research is published in academic peer-reviewed journals.

Mitigation
To mitigate these limitations, other lines of evidence were used to assess the excellence and scientific impact of the Aerospace Research Centre, including the use of an expert peer review.

Client survey response bias
The NRC client relationship management system did not include client contact information for all research centre clients and, for some clients, available contact information was for individuals in the organization’s finance or accounts payable units, who would have been unable to answer the survey questions. An email distribution list of clients, maintained by the research centre’s quality management team, was therefore used to reach more appropriate individuals within the client firms. The extent to which this list included all research centre clients over the evaluation period is unknown.

In addition, as with any survey, the fact that only a subset of the entire population responded to the survey introduces possible error into the study, since the respondent population may not entirely reflect the larger total population.

Mitigation
To mitigate this limitation, no findings were based solely on survey results. Survey findings were examined in conjunction with findings from other lines of evidence.

Challenges associated with peer review committee
In order to provide an objective, independent assessment of the research centre, arms-length experts were sought to participate in the peer review process. Given the need to be independent and objective, experts selected for peer review committees may not have in-depth knowledge of the research centre and its activities, or the NRC. As well, each member comes to the peer review exercise with their own experience, expertise and associated biases. Finally, one enlisted peer review committee member was, in the end, unable to attend the site visit, and did not participate in the peer review process.

Mitigation
In order to ensure the needed expertise on the committee, and to select the best possible committee composition, the project team consulted with the research centre’s senior management and advisory board, and with the Knowledge, Information and Technology Services branch, to identify and invite the peer review chair and members. All peer review committee members were vetted and approved by the research centre VP and DG.

To mitigate any bias, in constituting peer review committees, the evaluation team tries to ensure the inclusion of experts from various areas of expertise, including different genders, geographic locations, and representatives from various types of organizations (industry, academia and other governmental organizations). In addition, at least one Canadian representative is sought in order to bring an understanding of the Canadian context. In the case of the Aerospace peer review committee, member expertise covered all five Aerospace Research Centre laboratories, in spite of the loss of the one member prior to the visit, and included representatives from academic institutions, industry and other research-type organizations.

Peer Review Committee members

Dimitri Mavris, Chair
S. P. Langley Distinguished Professor, National Institute of Aerospace; Regents Professor, Boeing Chair in Advanced Aerospace Systems Analysis; Director, Aerospace Systems Design Laboratory, Executive Director, Professional Master’s in Applied Systems Engineering, Georgia Institute of Technology

Yolanda Hicks
Research Engineer, Combustion Branch, NASA Glenn Research Centre

Rakesh Kapania
Mitchell Professor of Aerospace and Ocean Engineering, Virginia Polytechnic Institute and State University

David Rancourt
Assistant Professor, Director of Aerospace, l’Université de Sherbrooke

Jeffrey Stith
Manager, Research Aviation Facility and Senior Scientist, National Centre for Atmospheric Research Earth Observing Laboratory

Alexander Velicki
Structural Design Engineer, Boeing Research and Technology Group

Appendix B – Aerospace research centre logic model

 

 

6. Ultimate (10+ years)	Economic impacts 6.1 NRC fosters a healthy, vibrant and competitive Canadian aerospace industry			Social impacts 6.2 Improved quality of life for Canadians (e.g., improved in-flight comfort, connectivity and safety)		Environmental impacts 6.3 Improved Canadian environment position (e.g., environmentally friendly aircrafts)
5. Intermediate (6-10 years)	Advancement of knowledge 5.1 NRC contributes to the future direction of the Aerospace industry			Business innovation 5.2 Creation & growth of Canadian firms (increased competitiveness & productivity)		Advancement of government policies and priorities 5.3 Regulation of aviation in Canada benefits from NRC’s expert contribution
4. Immediate (1-5 years)	4.1 NRC is recognized as a national leader and hub for aerospace research & technical excellence			4.2 New scientific or engineering discoveries advance aerospace R&TD (aerodynamic, flight research, gas turbines, structures & materials, manufacturing) 4.3 Extended customer base and reach within the Canadian aerospace industry (increased engagement with SME clients)		4.4 NRC is integrated with other government entities (federal, provincial) in matters relating to air transport and the environment 4.5 Scientific / technical advances contribute to effective stewardship of DND’s airborne assets
3. Outputs	Enhanced / maintained capabilities 3.1 Enhanced / maintained competencies 3.2 Enhanced / maintained world-class R&DT infrastructure			Scientific & technical achievements 3.3 Scientific & technical knowledge 3.4 Intellectual property		Collaborations 3.5 Strategic relationships / partnerships
2. Activities	Facilitate scientific, technical & organizational capability 2.1 HR management (expertise / workforce development, staffing & strategic succession planning) 2.2 Facility management (development & management, health and safety) 2.3 Strategic planning (foresight, market intelligence) 2.4 Program & portfolio management / coordination & project management			Build scientific & technical capability 2.5 Self-funded R&TD projects		Deploy scientific & technical capability 2.6 Collaborative R&TD projects 2.7 Technical services 2.8 Participation in scientific / technical committees
1. Inputs	1.1 NRC investment	1.2 Earned revenues	1.3 Contributions from other NRC portfolios & collaborators / partners (in-kind & cash contributions)	1.4 NRC common services & SSC	1.5 People (HQP, specialized expertise & experience)	1.6 Facilities (specialized facilities & equipment)