Status: | Active |
---|---|
Effective date: | December 21, 2020 |
Prepared by: | Office of Audit and Evaluation, National Research Council Canada |
Approval: | NRC's President |
Related document: | Summary - Evaluation of NRC’s Design and Fabrication Services Branch |
ISBN: | 978-0-660-37502-1 |
Alternate format: | Evaluation of NRC’s Design and Fabrication Services Branch (PDF, 1 MB) |
Acronyms
NRC research centres
- ACRD
- Aquatic and Crop Resource Development
- AEP
- Advanced Electronics and Photonics
- AERO
- Aerospace Research
- AST
- Automotive and Surface Transportation
- CONST
- Construction
- DT
- Digital Technologies
- EME
- Energy, Mining and Environment
- HAA
- Herzberg Astronomy and Astrophysics
- HHT
- Human Health Therapeutics
- IRAP
- Industrial Research Assistance Program
- MD
- Medical Devices
- METRO
- Metrology
- NANO
- Nanotechnology
- OCRE
- Ocean, Coastal and River Engineering
- SDT
- Security and Disruptive Technologies
Other acronyms
- CAD
- Computer-Aided Design
- DFS
- Design and Fabrication Services
- NRC
- National Research Council
- RTO
- Research and Technology Organization
- SME
- Small to Medium Enterprise
Executive Summary
The National Research Council of Canada (NRC)’s Design and Fabrication Services (DFS) Branch supports NRC research centres, programs, and facilities by providing design, engineering and fabrication services of precise mechanical prototype equipment and apparatus. DFS is a centralized function with 13 workshops located across the country. The evaluation of DFS covers 2012-13 to 2018-19 inclusively, and drew on a cross-NRC poll, data review, document review, key stakeholder interviews (DFS management and NRC’s 14 research centres), and an international comparative study of similar functions in other research and technology organizations. This is the first time DFS has been evaluated. Findings are presented by evaluation question.
- 1. To what extent is there a demonstrable need for DFS’ services within the NRC?
NRC research centres rely on DFS to varying degrees, and there is a vast range of understanding about the service both within and across research centres. However, DFS’ services are directly aligned to the core business needs of certain research centres in particular (Herzberg Astronomy and Astrophysics [HAA]; Ocean, Coastal and River Engineering [OCRE]; Aerospace [AERO]; Energy, Mining and Environment [EME] in Vancouver; and Automotive and Surface Transportation [AST] in Boucherville) and they rely heavily on DFS.
- a) What factors have influenced NRC research centres’ use (or non-use) of DFS services?
Enablers for the use of DFS
Most of DFS’ strengths are a direct result of its integration within the research centres. As an internal NRC service that is co-located within several research centres, DFS has developed important expertise and understanding tied to the research centres. This allows DFS to provide responsive and specialized advice and support in a range of scientific areas, from conception to fabrication. DFS has demonstrated an overall quality of service, high level precision and specific expertise that supports research centres in making scientific and technological advances, and in innovating with industry and other government partners.Barriers to the use of DFS
Barriers that prevent the use of DFS include variable and inconsistent understanding of the service, misalignment of expectations between research centre timelines and DFS responsiveness, and conflicting perspectives about the cost of DFS (especially as it relates to NRC external pricing and project management practices). Although DFS has engaged with research centres to increase awareness of its services and further explain its costing model, barriers persist. This is because the barriers are a result of multiple interacting factors, both within DFS’ control (e.g., time estimates, project management), and outside of it (e.g., NRC costing rates, pricing guidelines, time coding practices). Additionally, DFS and the research centres experience these barriers differently because of their distinct functions within the NRC. As such, an integrated approach that meets the needs of both DFS and the research centres, and addresses issues at the root cause within the organization, is required.Recommendation
DFS should work with relevant stakeholders (i.e., NRC Finance and Procurement Branch, Business Management Services, and Finitiative) to clarify DFS’ costing model and separate it from research centre pricing practices.
b) Are any changes needed to DFS services?DFS’s centralized service delivery model with co-located workshops maximizes utilization of DFS staff and equipment. In particular, it allows for the distribution of work based on capacity, specialized capabilities and equipment, and standardization of safety and quality protocols across 13 different workshops. This model is similar to that of other international research and technology organizations (RTOs).
Recommendation
DFS should maintain its current centralized service model, with co-located workshops, to ensure the continued provision of integrated high quality support to NRC facilities and research, development, and technology activities.
There are nevertheless drawbacks to the centralized model that require attention. In particular, as a centralized enabling service, DFS will always have to balance between adapting to the varied needs of the research centres, and applying consistent processes.
Additionally, centralization adds complexities to the relationship between DFS and the research centres in which it has co-located shops. While the research centres highly value their local DFS teams and treat them as their own, they are also wary of the impact central decision making has had, and will continue to have, on their independence and control over local operations. This creates tensions around roles and responsibilities, accountability, project planning and costs. DFS’ contribution to the NRC is at its best when its expertise and knowledge are attuned to the nuances of the research centre’s needs. To ensure continued support of the highest quality to the research centres that rely on their co-located shops, DFS must lead in frequent and transparent engagement with the research centres, who must also actively participate, so that they can achieve their mutual objectives.
Recommendation
DFS should engage in intentional collaboration with the research centres it serves, especially those where it has co-located shops, to ensure common understanding and achievement of goals.
- a) What factors have influenced NRC research centres’ use (or non-use) of DFS services?
2. Given its objectives, to what extent does DFS have the appropriate expertise, critical mass and facilities?Expertise and critical mass
DFS staff are well-respected and have accrued years of expertise in the design and fabrication of NRC’s various research areas that would be difficult, if not impossible, to replicate with external service providers. As one of DFS’ greatest strengths is the tailoring of its expertise to the needs of the research centre, it is important for DFS to ensure knowledge is retained and research centre needs continue to be met as it heads into a wave of retirements. This will continue to require careful consideration by DFS, as vacated positions have been reallocated within the branch in recent years, to meet growing demand within the NRC for digitization and advanced manufacturing capabilities. Current capacity is adequate. However, should DFS face a further increase in demand for new services or capabilities, without increased resources, DFS’s would have to review its current service offering, level of quality, or timeliness to meet the demand.Facilities
DFS’ work is highly reliant on its equipment and computer and network infrastructure. Although DFS has engaged in regular repair, replacement and purchase of new equipment, current budgetary restrictions and costly ageing equipment are a challenge. DFS has prioritized renewal and replacement of equipment, and has been agile in extending its lifecycle. Nevertheless, breakdown of DFS equipment has affected research centre projects through time and cost overruns. As DFS increases collaboration and communication with research centres, it will be important to consider ways that co-investment in DFS equipment and facilities can benefit both DFS and the research centres.
3. How has DFS enabled the NRC to achieve its objectives?The evaluation found that NRC research centres rely on DFS to support their innovative research and development work, provide technical services to industry and government clients, customize their scientific equipment and support their facilities. Several examples are provided in the report that demonstrate that DFS is an important internal service. Without it, research centres would have to fill their design and fabrication service gaps externally, which would often be more costly, less timely, and of lower precision and quality. DFS supports research centres in providing turnkey solutions to NRC clients and collaborators and enhances their contributions to scientific excellence and global influence.
1. Introduction
An evaluation of the Design and Fabrication Services Branch (DFS) was conducted by the NRC’s Office of Audit and Evaluation to assess DFS’ relevance, performance, and efficiency. The evaluation covers the period of 2012-13 to 2018-19, with some information for 2019-20. It was carried out in accordance with the NRC’s approved evaluation plan, Treasury Board policies, and in support of NRC senior executives’ information needs. This is the first time DFS has been evaluated.
This report begins with a profile of DFS, followed by the evaluation findings, conclusion, and recommendations.
Evaluation approach
Methods
Mixed methods were used to maximize the generation of valid and relevant evaluation findings. This approach also allowed for convergence of results across methods and contributed to a better understanding of complex issues by exploring different facets. Methods include:
- Document review
- Data review (financial, administrative, and performance)
- Key informant interviews (staff from 14 NRC research centres, N=59, and DFS leadership, N=10)
- Cross-NRC poll (N=345 respondents)
- Case studies (N=5)
- International comparison study (N=7)
For more detailed information on the methods, including challenges and limitations, refer to Appendix A.
Evaluation questions
The evaluation questions were developed based on consultations and a review of key documents. The questions were:
- To what extent is there a demonstrable need for DFS’ services within the NRC?
- What factors have influenced NRC research centres’ use (or non-use) of DFS services?
- Are any changes needed to DFS services?
- Given its objectives, to what extent does DFS have the appropriate expertise, critical mass and facilities?
- How has DFS enabled the NRC to achieve its objectives?
2. DFS profile
DFS is an enabling service established within the NRC in 1995. Throughout the years, design and fabrication services had been under the control of research centres and different corporate divisions. However, since the establishment of NRC common services in 2012, DFS has been a centralized NRC service whose mandate is to directly support research centres with their design, engineering and fabrication needs.
DFS is led by a director general within the Business and Professional Services Division, reporting to the Vice-President of Business and Professional Services. The branch’s management team includes a director of fabrication, a performance and planning manager, a quality manager, and a senior design engineer.
Services
DFS services include:
- Design and engineering
- consultation and development of initial or final designs (for parts, research equipment, devices, facilities, etc.)
- generation of manufacturing information to assist in the fabrication process
- production of engineering drawings and graphics
- Advanced fabrication
- machined and 3D printed parts; mechanical, welded and woodworking assembly
- complex mechanical assembly and precision adjustment of research instruments
- prototype installation in research facilities
- Reverse engineering and rapid prototyping
- 3D scanning, additive manufacturing
- virtual or augmented reality technologies
- Quality control
- verification of all parts and assemblies for precision, tolerance and finish of components
Each of these services, including advisory services, can be accessed independently or as a turnkey solution, from start to finish within a project. To this end, DFS provides support across the NRC for research and development projects, technical services, and facilities maintenance or development (See Appendix B for DFS’ logic model). DFS shops offer further specialized services based on the needs of research centres where its workshops are co-located. For example, DFS engages in ship model making in St-John’s for the Ocean, Coastal and River Engineering Research Centre (OCRE), aircraft modifications at the NRC’s Uplands campus in Ottawa for the Aerospace Research Centre (AERO), and telescope components in Victoria for the Herzberg Astronomy and Astrophysics Research Centre (HAA).
DFS worked on an average of 723 projects per year during the evaluation period (2012-13 to 2018-19). These varied between:
- Tasks (68%)
- → simple, defined work requests that typically take less than 75 hours to complete and can be completed by a single DFS team
- Service agreements (7%)
- → pre-approved amount of labour established at the beginning of a fiscal year, usually to support a client facility (hours vary based on the agreement)
- Projects (25%)
- → complex or un-defined work requests that may require more than one DFS team to complete and/or special security requirements
- → most DFS projects (78%) in this category took between 75 and 249 hours to complete
Human resources
In 2018-19, DFS employed 99 employees across Canada, with 80% located in Ottawa. The majority of DFS employees are male (93%), technical officers (88%) with permanent positions (96%). As indicated in Figure 1, there are few employees newly starting their career in DFS, with 80% of staff between the ages of 36 and 55 years.
When compared to workforce availability, women are under-represented, especially in technical roles. According to employment equity representation data as of March 31, 2020, if DFS hired four more women, it would be on par with workforce availability. Aboriginal people, persons with disabilities, and other visible minorities each represent less than 5% of DFS’ workforce and therefore workforce availability data is unavailable for these groups.
Figure 1. In 2018-19, the majority of DFS’ workforce was 36 to 55 years old.
Figure 1 ‑ Text version
Figure 1 is a horizontal bar graph of the age distribution of the DFS workforce in fiscal year 2018-19. In 2018-19, the majority of the DFS workforce was between the ages of 36 to 55 years old. Specifically, the distribution is as follows:
Under 25 years old to 35 years old |
Between 36 and 45 years old |
Between 46 and 55 years old |
Over 56 years old |
---|---|---|---|
6% | 36% | 44% | 13% |
Facilities
DFS operates 13 workshops in 5 cities (with a 14th coming in 2021 in Winnipeg). One central shop is DFS’ Ottawa headquarters, serving all research centres, while the 12 other shops are co-located within different research centre buildings across Canada.
While all shops provide on-site fabrication services, only three in Ottawa have on-site designers, with the main design office located in DFS’ headquarters. Research centres outside of Ottawa requiring design, advanced technologies (e.g., 3D scanning, reverse engineering), and quality control that cannot be performed by toolmakers in their DFS shop can request the services. Depending on project requirements, the work is either completed in Ottawa and shipped to the research centre, or a DFS team is deployed to their location.
Figure 2. DFS shops are co-located in NRC research centres across Canada.
Figure 2 ‑ Text version
Figure 2 is a map of the different locations of the DFS shops. There are a total of 13 DFS shops (with a 14th coming in 2021) and they are co-located in NRC research centres across 5 cities in Canada. The DFS shops are distributed as follows:
St-John’s, Newfoundland: |
1 |
---|---|
Boucherville, Quebec: |
1 |
Ottawa, Ontario: |
6 shops on the Montreal Road Campus (M2, M4, M7, M10, M14, M36), and 3 shops on the Uplands Campus (U61, U 62, U66) |
Winnipeg, Manitoba: |
1 (coming in 2021) |
Vancouver, British Columbia: |
1 |
Victoria, British Columbia: |
1 |
The Victoria shop is embedded in the Herzberg Astronomy and Astrophysics Research Centre (HAA), while the Vancouver shop operates within the Energy, Mining and Environment Research Centre (EME). In Ottawa, one shop is co-located within the Metrology Research Centre (METRO) and the other seven co-located shops are within the Aerospace Research Centre (AERO). The Boucherville shop is co-located with the Automotive and Surface Transportation (AST), Medical Devices (MD), and Energy, Mining and Environment (EME) research centres, while the St-John’s shop is embedded within the Ocean, Coastal and River Engineering Research Centre (OCRE).
Revenues and expenses
As an internal NRC service, DFS is mainly focused on serving NRC research centres, but 5% of its projects are for external clients. These projects allow DFS to meet its annual revenue target of $500,000 (see Figure 3).
The majority of DFS revenues (97%) are earned through services performed for other government departments. The remaining revenues are earned from services for external industry clients (3%).
Figure 3a In recent years, DFS has exceeded its revenue target.
Figure 3a ‑ Text version
Figure 3A is a bar graph illustrating the revenue of DFS between fiscal years 2021-2013 to 2018-2019. The revenue target across all years is $500,000. With the exception of 2012-13 and 2014-15, DFS has exceeded its revenue target. The total earned revenue for the years observed is as follows:
2012-13 | $316,000 |
---|---|
2013-14 | $705,000 |
2014-15 | $477,000 |
2015-16 | $852,000 |
2016-17 | $840,000 |
2017-18 | $622,000 |
2018-19 | $752,000 |
Figure 3b. Yearly expenses averaged $6.7 million during the evaluation period.
Figure 3b ‑ Text version
Figure 3B is a horizontal bar graph depicting the yearly expenses between fiscal years 2012-13 to 2018-19. Across all 7 fiscal years, it averaged at $6.7 million. Specifically:
2012-13 | $8.3M |
---|---|
2013-14 | $6.5M |
2014-15 | $7.0M |
2015-16 | $7.1M |
2016-17 | $5.6M |
2017-18 | $5.2M |
2018-19 | $6.7M |
Clients and collaborators
While DFS provides services to all NRC research centres, AERO is its largest client, accounting for 37% of its project work over the evaluation period. Although HAA is not a top user of DFS according to number of project, it has a small number of multi-year projects that fully occupy the DFS Victoria shop’s capacity.
Figure 4. Five research centres accounted for 78% of DFS’ work between 2012-13 and 2018-19.
Figure 4 ‑ Text version
Figure 4 has two figures. There is a donut chart that revealed that 78% of DFS’ work between 2012-13 and 2018-19 were accounted by five specific research centres (i.e., EME, OCRE, METRO, AST, AERO), whereas the remaining 22% were accounted by other research centres.
Pointing to the donut chart is a horizontal bar graph that shows the breakdown of the proportion of DFS work that each of the five research centres account for. Specifically:
EME | 6% |
---|---|
OCRE | 6% |
METRO | 10% |
AST | 18% |
AERO | 37% |
DFS work for all research centres fluctuates yearly due to various factors including: different research centre budgets and priorities, and cyclical or variable external industry demands. DFS manages these fluctuations, and matches equipment and staff capabilities to research centre needs, by distributing work across its various workshops (see Figure 5).
Figure 5. In addition to serving their main co-located client(s), all DFS shops have taken on work for other research centres when needed, with the exception of Victoria, being fully occupied with work for HAA.
Figure 5 ‑ Text version
Figure 5 shows that in addition to serving their main co-located client(s), all DFS shops have taken on work for other research centres when needed, with the exception of the DFS shop in Victoria, British Columbia, which is fully occupied with work for HAA. There is a map that indicates the main research centre which the DFS shop serves, and other research centres that DFS completed at least 50 hours of work for during the evaluation period. Specifically:
St-John’s, Newfoundland | Main client is OCRE, but it has taken on work for the following research centres as well: AERO, METRO, HAA, SDT, and EME. |
---|---|
Boucherville, Quebec | Main clients are AST, EME, HHT, and MD, but it has taken on work for AERO, SDT, ACRD, and OCRE. |
Ottawa, Ontario |
While the headquarters serves all of NRC, the main client at the Uplands Satellites campus is AERO and the main client at the Montreal Road Satellites campus is AERO and METRO. Uplands Satellites have taken on additional work for the following research centres: AST, METRO, SDT, CONST, HAA, EME, and DT. The Montreal Satellites have taken on additional work for the following research centres: AST, EME, SDT, CONST, DT, OCRE, AEP, and HAA. |
Vancouver, British Columbia | The DFS shop in Vancouver mainly serves EME, but it has taken on work from HAA and AST as well. |
Victoria, British Columbia | Fully occupied with HAA work. |
Among its external revenue clients between 2012-13 and 2018-19, DFS mainly supported Defence Research and Development Canada (50% of external client projects), Communications Research Centre Canada (28%), and the Department of National Defence (10%). External industry clients with whom DFS works directly are mostly small to medium enterprises (SMEs), with a few larger companies that design and manufacture specialized components in various fields.
3. Relevance of NRC’s Design and Fabrication Services
Across the NRC, there is an inconsistent level of awareness and understanding about DFS services, including the nature of its work and cost. Use of DFS also varies within and between research centres. Although many turn to DFS for their design and fabrication needs, certain research centres in particular are highly dependent on DFS for their core business and operations (i.e., AERO, EME, HAA, and OCRE). The research centres who rely on DFS do so because of the high quality and precision delivered. DFS’ proximity, expertise, and ability to understand the unique needs of researchers and to work iteratively with them over the course of a project with them is valued. The most common barriers research centres face in their interactions with DFS are related to cost, pricing, and timelines. These barriers are created and sustained by multiple interacting factors, some of which are out of DFS’ immediate control. Research centres with a heavy reliance on DFS services face additional challenges related to communication and sense of control over design and fabrication, human resources, and equipment. Overall, DFS’ centralized model, with co-location of some shops, is consistent with other research and technology organizations (RTOs). The benefits far outweigh the drawbacks, which can be addressed through increased communication and intentional collaboration.
Main supporting evidence: Cross-NRC poll, interviews, document review, data review, international comparison study.
a) Factors that have influenced NRC research centres’ use or non-use of DFS services
Multiple factors influence the use of DFS services. While some have led to increased need and use of the service (i.e., iteration and customization needs, alignment between DFS and research centre’s core business, proximity, and high quality and precision requirements), others have contributed to decreased or mixed use of DFS (i.e., varying levels of awareness and understanding, timeliness/responsiveness, and cost/price).
Factors contributing to increased use of DFS
Several factors related to DFS’ service model and the needs of research centres contribute to a greater reliance on DFS and use of its services. These include:
Iteration and customization needs: Research centres seek DFS’ expertise and ability to work on one-off prototypes, or customized solutions. In fact, DFS’ ability to collaborate closely with research centre staff and work iteratively over the course of a project is a key element of the branch’s value proposition to the NRC. Research centres that rarely use DFS services (e.g., ACRD and HHT) tend to not require customized equipment and instruments or physical prototypes and products.
Alignment between DFS and research centres’ core business: Research and operations of five research centres in particular (i.e., HAA, OCRE, AERO, EME in Vancouver, and AST in Boucherville) are highly dependent on DFS. For example, HAA depends on DFS to support the design and fabrication of the majority of its instruments for large and small astronomy projects, while OCRE in St-John’s relies exclusively on DFS to build or add instrumentation to the marine models tested in its tanks. In Vancouver, EME turns to DFS for setting up and customizing much of its lab equipment and its mining consortium relies heavily on DFS support. In Ottawa, AERO staff are dependent on DFS to build wind tunnel models and support the modification, repair and instrumentation of aircraft.
"They’re an integral part. We can’t work without them"
It is important to note that while these research centres value their integrated DFS shops, they are uncomfortable with their lack of control over the key aspect of their core business that is dependent on design and fabrication resources. In particular, there is a loss of control over resources at the project level (i.e., DFS capacity/availability to meet the research centre’s deadlines) which results in a perceived loss of control over cost/budget, and limited input in DFS decisions on HR and equipment/facility decisions in their co-located shop.
Proximity facilitates collaboration: Proximity of DFS services, whether in co-located shops or not, was identified as an important benefit of working with DFS (43% of poll respondentsFootnote 1 and the majority of research centre clients interviewed). In particular, the proximity of DFS staff allows for greater interaction as research centre staff iterate with DFS for their design and fabrication needs. Proximity was seen as a facilitator for collaboration and higher quality services, as DFS staff build knowledge and expertise by working closely with the research centres, and become more attuned to their specific needs. For some, proximity also means faster and more responsive timelines (a benefit of DFS identified by 23% of cross-NRC poll respondents and a large proportion of interviewees), but this is not a consistent belief across the NRC as will be presented in the following section.
“Integration of the DFS team within the research centre is a definite advantage. Because of it, we have access to a service that is more personalized, of quality, and a greater control over delivery time (…) This allows for a closer and more efficient collaboration.”
High quality and precision requirements: Overall, DFS was recognized across the NRC for its high quality and precise workmanship (34% of poll respondents indicated they rely on DFS because they require high precision). There was consensus among stakeholders that DFS provides higher quality and more precise work than external service providers, which is important to many DFS clients. For example, HAA requires precision from DFS at the micron level, sometimes up to 1 micron, which represents 1/50th of the width of a human hair.
Despite a general appreciation for DFS precision and quality, staff in some research centres identified instances of inconsistent quality (N= 5 research centres) or over-production (N=3 research centres). This typically appears to be a result of poor communication and comprehension between both parties, and is subject to the level of familiarity DFS staff have with specific research centres’ needs. For example, it can be difficult for DFS staff accustomed to working on highly precise fabrication for AERO (due to aircraft safety requirements) to shift their mindset to fabricating something simple for a facility or a low-complexity testing apparatus. Similarly, when work is distributed between different DFS shops, staff that are not co-located with the research centre requesting the work, are not as familiar with the type of work done within the research centre, or the level of precision and quality typically required.
To ensure consistent quality and meet the varied needs of the NRC’s 14 research centres, DFS implemented a quality management system in 2014. It also developed a quality manual and standard operating procedure for all DFS staff in 2015. DFS has also since conducted 3 internal audits of its quality and project management processes, and continues to do so at regular intervals.
Factors contributing to decreased or mixed use of DFS
When polled, 38% of respondents from the 14 NRC research centres indicated they never use DFS for their design and fabrication needs. A majority of these respondents work in research centres that do not frequently rely on DFS (e.g., HHT, MD, and ACRD) and acknowledge having low awareness of their services. However, 49% of CONST respondents indicated that they never rely on DFS for their design or fabrication needs, despite the majority (78%) being aware of the service. According to the poll, many CONST respondents (43%) opt to do the work themselves or contract it out (32%).
As use of DFS varies among and within research centres, there are pockets of users within some lower-use research centres who regularly rely on DFS and have taken on the role of advocates for the service. DFS and these advocates often try to correct information or increase knowledge about the service. However, as several factors contribute to decreased or mixed use of DFS across the NRC, it is difficult to counter misinformation, past negative experiences (e.g., 25% of AERO poll respondents), or address factors individually when many interact and compound each other. These factors include:
Varying levels of awareness, understanding, and access: Although the majority of the NRC is aware of DFS, there is great variability within research groups, and among different research centres, when it comes to understanding how DFS operates and what services it offers. Individuals with greater awareness and knowledge of DFS services tend to belong to research centres where DFS has co-located workshops or are regular users of the service. Knowledge about how to access DFS, and ways in which staff within the NRC access these services, also varies greatly. As indicated in Figure 6, certain NRC research centres in particular are not aware of DFS and its services. Additionally, in interview, regular or occasional DFS users also admitted to not knowing the breadth of DFS services available to them, or were not aware that they could access services in other DFS shops. Finally, 14% of poll respondentsFootnote 2 indicated they do not use DFS because they do not know how to request DFS services. Again, those with co-located shops benefit from ease of access while others face less obvious ways to access DFS, especially if they do not have existing ties or past collaborations with DFS staff.
In recent years, DFS has actively engaged in promotional activities across the NRC. In particular, DFS is involved in orientation sessions for new NRC employees as a means of introducing them to DFS services and capabilities. DFS has also recently engaged in relationship-building with research centres that do not typically use its services (e.g., NANO and DT) and organized presentations in the regions (e.g., EME, HAA, OCRE) to share more information about its service offering and costing model. While these communication efforts may increase awareness, addressing other barriers described below, or long ingrained beliefs based on past experiences will take more time and communication at multiple levels of the NRC.
Figure 6. Research centres who are not regular users of DFS have low awareness of its services.
Figure 6 ‑ Text version
Figure 6 has two figures. A donut chart was used to illustrate the results of a cross-NRC poll for the item “I know that I can use DFS for all my design and/or fabrication needs. 78% of the respondents indicated that they were aware, whereas the remaining 22% were unaware.
The second figure indicates that the majority of respondents from these research centres were unaware of DFS: MD (71%), ACRD (64%), HHT (50%), and DT (50%).
Misaligned expectations regarding timelines and responsiveness: Staff in some research centres experience a misalignment between their project timelines and DFS’ ability to respond in a timely way. For example, 30%Footnote 3 of respondents in the cross-NRC poll indicated that “My timelines do not align with DFS availability/capability and I cannot wait.” This was a particular issue for AERO (65%) and CONST respondents (46%). Different examples of challenges in aligning DFS deadlines to external client timelines, or long lead times for DFS to complete work were also mentioned by more than a quarter of research centre interviewees. Factors contributing to these experiences are discussed below.
Research centres in both the Engineering and the Transportation and Manufacturing divisions work with a large number of external clients on projects which typically involve tight timelines and high revenue. Misalignment between DFS timelines and research centre needs within these divisions is therefore heightened by these external pressures. Staff in these divisions are more apt to say that going through DFS is slower than going through an external contractor, and are more frustrated by the misalignment. DFS capacity is discussed in Section 4 and is not the main cause of this issue. Instead, multiple factors interact to create frustrations with these groups including:
- Communication surrounding project objectives and scope: When client expectations are not known or not clearly identified, DFS tends to estimate its time at the higher end of the spectrum to accommodate for unknowns.
- Timing of DFS involvement in projects: Whether DFS is an afterthought to some, or perceived as too costly to involve in the earlier stages of project planning, later involvement tends to contribute to a mismatch between DFS capacity and research centres’ timeline expectations.
- Project prioritization: Project planning at the research centre level does not account for availability of DFS resources. At times, this leads to competition for DFS’s limited resource pool among research groups in a research centre.
Although alignment of timelines is a challenge, only 13% of poll respondents indicated that “past projects with DFS have not been delivered on time.”
As indicated in Figure 7, DFS has made positive strides in reducing its rate of late delivery in recent years. However, having a consistently high percentage of early delivery is likely an indication of the challenges faced by DFS and the research centres in scoping projects accurately.Footnote 4
Figure 7. Overall variability of DFS timeliness over the years has mainly seen projects delivered late (in the past) or early (recently).
Figure 7 ‑ Text version
Figure 7 is a line graph illustrating DFS’ project delivery trends over the evaluation period. Overall, it has been observed that projects were delivered late in the past, or early in recent years. Specifically, the proportion of projects delivered early, on-time, and late for each fiscal year is as follows:
FY12 | Early: 63%, On time: 24%, Late: 13% |
---|---|
FY13 | Early: 51%, On time: 31%, Late: 17% |
FY14 | Early: 53%, On time: 37%, Late: 10% |
FY15 | Early: 33%, On time: 27%, Late: 39% |
FY16 | Early: 32%, On time: 24%, Late: 44% |
FY17 | Early: 34%, On time: 12%, Late: 54% |
FY18 | Early: 51%, On time: 28%, Late: 21% |
FY19 | Early: 59%, On time: 26%, Late:16% |
Conflicting perspectives about the cost of DFS are detrimental to its value-add to the NRC: Cost is the most controversial aspect of DFS services, drawing polarized responses across the NRC and demonstrating varied levels of understanding.
Those who think DFS is cost-effective say it is a result of many of the enabling factors listed previously. In their view, engaging DFS is more cost-effective than requesting the same service from an external provider. DFS better understands and responds to their needs than external providers who often deliver a product of poor quality and low precision that is ultimately unusable. NRC research centre staff who believe DFS to be cost-effective can often be described by one or all of the following characteristics. They are often individuals who:
- Understand how certain elements of their request impact DFS labour and cost, and are able to plan their project budgets accordingly. Examples of these elements include: level of specificity of their instructions and timelines, requirements for high precision/complex parts, and continuous iteration.
- Are not typically project leads, and therefore can work with DFS on a project without having to deal with cost, pricing, and project budget considerations.
- Work on internal research and development projects, as opposed to external client projects, and therefore do not need to consider how they will include the cost of DFS in their external client pricing.
In particular, staff from HAA and METRO were most likely to see the cost of DFS as a non-issue, both in interviews and in the cross-NRC poll.
Cost was nonetheless identified as one of the biggest barriers to use of DFS. Thirty percent of the cross-NRC poll respondents indicated that “DFS is too expensive.” Respondents from OCRE (67% of the research centre’s respondents), EME (58%), and AERO (48%) were more apt to express this opinion both in the poll and in interviews.
At the root of this barrier is whether the cost of DFS is a true cost or not to NRC research centres. DFS asserts that its labour is not a true cost to research centres because DFS labour costs are assumed by DFS’ salary budget. As such, DFS stresses that its labour estimates are simply a reflection of value and that it is the prerogative of the research centres/divisions whether or not to include the full value of DFS labour in their project pricing for external clients and collaborators. From DFS’ perspective, material costs above $1,000 are the true and only cash cost research centres must assume with their operations budgets.
It should be noted that the international comparison study of DFS-type functions in other RTOs revealed that other organizations also typically demonstrate the dollar value of their work to research units, or directly charge them for the cost of their services. According to one of the organizations, it is important for research units to see the cost of their service because “If no costs are allocated, they do not appreciate the value of the service.”
DFS has recently engaged in outreach with some research centres to clarify this notion of cost vs. value. However, many in the research centres object to this characterization of the cost of DFS, especially when they are project leads who manage a budget or when it comes to working on external client projects. From the perspective of research centres who believe DFS to be too expensive, the following elements combine to make DFS appear too costly:
- Research centre staff working with external clients include DFS labour costs (including DFS’ facilities cost-recovery rate of $28.92/hr regardless of type of equipment used) in their client invoices. According to some, this automatically monetizes the cost of DFS labour and nullifies DFS’ assertion that its labour is not a true cost.
- Due to NRC costing practices, all DFS labour, regardless of the complexity of the work involved, is charged at the same rate (e.g., $123.19/hour for a TO4Footnote 5 ). When DFS staff is engaged in complex work supporting world-class research and development or innovative practices, this rate is appropriate. However, when research centres require production of multiple components or simple work, the research centres indicate that external service providers are less expensive than DFS when comparing hourly wages.
- Lack of clarity in understanding pricing and costing guidelines across the NRC, and consistency in applying them, make it confusing and difficult for research centres to figure out how to price their work externally, apply various discounts (e.g., IRAP certificate program, SME and academia rates, collaborative research project model), and meet revenue targets. In the case of research centres like AST, where external contracts are tied to time and materials invested by the NRC, it is difficult to justify not billing clients for DFS labour. In other research centres like OCRE, staff and management have indicated that if they include actual DFS labour costs in their pricing and the DFS labour estimates are high (due to labour costs of model-making), that it can cost them the job. As such, disregarding or reducing the cost of DFS labour in research centre pricing is easier said than done.
- Although it is true that DFS labour costs are not cash costs that draw on a research centre’s operational budget, they do appear as a cost in project-level budgets because DFS charges its labour directly to research centre project time codes. As a result, any unplanned increases in DFS labour on a project appear as deviations from project budgets. This triggers a response from the project management offices within research centres as they work with project leads to stay on time and on budget in their projectsFootnote 6.
- Additionally, some research centres believe that DFS estimates are overinflated and they do not see or understand how DFS time is allocated to their projects until it is too late because DFS can code time directly to their project. In some cases, there is a perspective that DFS is padding its estimates to meet utilization rate key performance indicators.
“The cost of the service is a surprise box. [It is] difficult to plan a project that requires [DFS], [we have] no control over what is done and we have to pay for the services rendered, even if it exceeds the initial estimate.”
Iteration, novel methods, and evolving scope affect DFS labour estimates. The more a project idea is undefined or requires novel or iterative processes, the more time is required by DFS to do the work, and the more difficult it is to predict time and cost early on. When this is not communicated and understood by all parties, especially when external client estimates are involved, project budgets are already fixed (and do not factor in DFS effort), this can lead to frustration and conflict. However, as indicated previously, research centres rely on DFS to collaborate closely with them in the iterative process of their unique designs and builds, to understand and adapt to their specific scientific and industry needs, and to deliver high quality and high precision. This is part of the value of DFS that cannot be easily costed/priced and that is often missing from research centre comparisons with external contractors, or that is confused with price comparisons for simple fabrication projects. Despite the tendency of some to indiscriminately compare the cost of DFS to external service providers, interviewees across the NRC readily acknowledge that external contractors require fully flushed-out designs, are hit and miss on quality and precision, and that once the contract is signed, any deviations from the agreed upon statement of work come at additional costFootnote 7.
This increases DFS labour costs and can make it more difficult for certain research centres to compete with international competitors on price in trying to secure an external client contract. For example, approximately 600 marine models have been made in St-John’s with DFS/OCRE in the last thirty or more years, whereas some competitors are said to produce more than 100 models per year, making them faster and cheaper due to economies of scale. The same applies to the design and fabrication of AERO wind tunnel models. According to AERO, DFS cannot compete on time and price with international companies whose core focus is wind tunnel model design and manufacturing. These competitors can provide quick and accurate quotes (+/- 5%) on the cost of a model because they have been doing it daily for more than 20 years while DFS has only been making models sporadically for fewer years. As a result, although AERO would like to offer a turnkey service to its clients, encompassing all their needs from model-making to testing in their wind tunnels, AERO has decided not to bid on some external client projects, knowing that DFS is not able to design and fabricate a model in time to get it tested in a tunnel within the timelines required by the clientFootnote 8.
Research centres adopt different methods to compensate for DFS labour costs (whether initial costs or cost-overruns) in their project budgets. Some research centre staff reported cutting out earlier consultations with DFS, especially design services, to save on cost. This sometimes has the opposite effect of increasing project costs and timelines, as re-work and re-scoping of projects due to safety, specificity, or quality of designs is more likely when DFS is unable to provide early input into designs, materials, and fabrication methods.
Others reduce their project scope and deliverables. As DFS and research centre labour are lumped together in proposals that go to external clients, this can lead to clients asking why they are charging so much and committing to so little. Therefore, projects proposals become a juggling act of reducing and adjusting level of effort to meet client price expectations. This is particularly frustrating in the regions where, prior to centralization, research centres used to have direct control over DFS resources and therefore had more flexibility in managing its impact on project proposals.
Some research centres opt to bypass DFS entirely and contract their design and fabrication needs externally to save on costs (23% of the cross-NRC poll respondents indicated they did so for simple projects to save on time and money). While total labour costs for simple projects are at times lower than DFS labour costs (even if they are true cash costs), research centres that choose to contract externally for larger projects may neglect to factor in the administrative costs of the procurement process for the NRC. In fact, external service contracts cost NRC Procurement Services approximately $500 in labour to establish sole source contracts, and between $8,000 and $11,000 for higher value projects that require calls for tender, standing offers or subsequent contract to a supply arrangementFootnote 9.
Finally, while some research centres hope that clients will forgive and pay for any cost overruns on DFS labour once projects are underway, others account for DFS labour in other research centre overhead or facilities time codes, thereby making it difficult to accurately assess the true labour costs for projects
In sum, several barriers interfere with the use of DFS across the NRC. Although DFS has engaged in efforts to address awareness and cost in particular, their current approach will not be successful if the following is not considered:
- Multiple barriers interact in various combinations, and are compounded by the variability that is inherent to each research centre. Unless these barriers, especially those related to the cost of DFS, are addressed collectively, and communicated at various levels (e.g., director general, director, project manager, researcher, and facilities manager) within the NRC, they will persist. Additionally, if DFS fails to acknowledge that research centres are affected by DFS labour costs, regardless of whether this has cash or accounting implications, and regardless of corporate and research division responsibilities with regards to pricing (which DFS does not control), research centres will not be open to hearing about solutions to DFS’ costing model.
- Research centres must be willing to work with DFS to resolve issues and collaborate in a transparent and productive way.
Recent work undertaken by a corporate initiative (Finitiative) to simplify NRC project management processes presents an opportunity for DFS and NRC research centres to address some of these barriers together. In fact, upcoming Finitiative changes to be implemented across the NRC in the fall of 2020 may benefit DFS and alleviate some of these issues and resulting tensions. For example, clarifying the distinction between projects and activities so that they can be managed differently (especially with regards to budgets and schedules), and a reduction of time coding entry requirements may diminish the scope of issues between DFS and research centres with regards to labour costs and the effect of time coding on project budgets from a project management perspective. As an enabling service that works with the NRC’s 14 research centres, DFS should stay abreast of Finitiative working group activities as they continue to implement measures that simplify and align practices across the NRC, and actively contribute to proposed solutions.
b) DFS’ service model
No major changes are required to the structure or offer of DFS services. DFS capabilities are non-duplicative of the research centres’ and DFS has an appropriate approach to its service offering. DFS’ centralized service model, with co-located services in certain research centres, is consistent with that of other RTOs. Although centralization has led to tension or difficulties between DFS and certain research centres, these can be addressed through increased clarity, communication and collaboration.
Duplication of services is not an issue:
Fabrication – There is generally no overlap between DFS fabrication equipment and capabilities, and those of the research centres. While some research centres maintain small workshops with basic tools that are appropriate to the needs of the research centres, none are of the caliber of DFS’ equipment. For example, CONST staff have a relevant skillset and field of research that allows for some fabrication, while the SDT and AST research centres have their own 3D printers, but these are of lower quality than DFS’ and are mainly used for materials testing, not fabrication purposes. Finally, where some research centres work with virtual reality, there may be a small amount of duplication of capabilities with DFS’s advanced technologies group. DFS’ launch of a new NRC virtual reality community of practice may help research centres familiarize themselves with DFS competencies in this area, promote collaboration, and reduce duplication.
“Time is money. It’s better to engage DFS so our researchers can focus on their research rather than manufacturing.”
Design – While staff within the research centres often draft their own designs, they are not typically designers by trade and their capabilities are not duplicative of those in DFS’ design team. As indicated previously, there is value in involving DFS design early on in the conception of fabrication projects to ensure the fabrication team receives precise requirements and designs that are safe to build and use. DFS’ plan to lead a NRC-wide Computer-Aided Design (CAD) user group in 2020-21 has the potential to strengthen ties between DFS’ design team and the research centres. Additionally, as DFS shares its expertise and supports the development of CAD capabilities across the NRC, it is further highlighting its value-add.
To ensure a continued lack of duplication between DFS and the research centres, clear understanding of DFS’ scope of work and expertise, and ongoing dialogue about research centre needs and equipment is important. This includes the exploration of opportunities for co-investment in new facilities or equipment.
Divestment of services is not required: The evaluation did not identify a specific service area of DFS that should be divested. DFS activities are relevant to its mandate and current DFS processes of contracting out work on an as-needed basis are appropriate. DFS should continue the current practice of contracting out work for the following reasons: production of multiple units, very simple jobs when they can be produced faster or at a lower cost by a local fabrication shop, certain specialized or higher risk capabilities not frequently required or worth the investment, and to supplement DFS capacity when project timelines and requirements make it necessary.
A little more than 25% of the poll respondents (including regular DFS users) indicated that they do not exclusively use DFS for their design and fabrication services because they contract out some or all of it. NRC employees who understand DFS services and have built a positive relationship with DFS automatically go to them to discuss their project needs, and identify together whether DFS or external contractors are more appropriate for the job. This decision is not always obvious as it is subject to a variety of project-specific variables.
Whether design and fabrication work is contracted out or not, DFS’ work for the research centres is specialized and does not take away from private industry, which is more focused on low-risk, large productions rather than riskier one-of-a-kind design and fabrication projects, and which does not cover the breadth of expertise that DFS offers.
Therefore, the current system that allows research centres the freedom to rely on DFS or not for their design and fabrication work should be maintained. Forcing the more reluctant users into using DFS will only aggravate existing tensions. Through increased understanding and relationship-building between DFS and research centres, the current practice to sometimes bypass DFS for unnecessary and potentially cost-prohibitive and inappropriate external contracting may therefore be eliminated. At the very least, it would be in the research centres’ best interest to get comfortable consulting DFS for advice on their design and fabrication projects, as DFS is well-equipped to provide the guidance they need in making informed decisions.
Centralized delivery model (with co-located services) is appropriate despite some drawbacks that need to be addressed: An international comparison study of DFS-type functions in other research and technology organizations (RTOs) revealed that DFS’s centralized model is consistent with the approach adopted by other RTOs like the Commonwealth Scientific and Industrial Research Organization (CSIRO), Defence Science and Technology (DST), and German Aerospace Centre (DLR) (see Appendix C). Four other RTOs included in the study adopted decentralized models where control of the DFS functions was assumed by each research group, rather than a central corporate function. However, co-location of DFS-type services was present in all RTOs regardless of model.
Several benefits of having co-located shops like DFS’ were identified by all stakeholders (DFS management team, research centre clients, and other RTOs). These benefits include:
- facilitated collaboration process
- reduction of duplication in equipment
- timely communication and advice
- responsiveness
- increased DFS expertise
The benefits of DFS’s centralized model as identified by DFS’ management team and some research centre clients include:
- standardization of safety protocols, training, and quality control
- dedicated budget for the purchase of design and fabrication equipment (as opposed to competing with other research centre budget priorities)
- ability to distribute work across DFS’ 13 shops to match capacity/demand, specific capabilities, and specialized equipmentFootnote 10
The drawbacks of DFS’s centralized model include:
- Individualized knowledge and client-centered expertise that co-located shops offer research centres is diluted when work is shipped to a different location. Research centre staff build trust and understanding with DFS over time as they work closely together. Therefore, the proximity that facilitates communication and iterative work with DFS can be lost. In fact, some researchers equate other DFS shops to external service providers, or just a step above.
- Certain DFS processes (e.g., administrative forms and resource planning) sometimes frustrate research centres and DFS staff. As an enabling service that works with 14 different research centres, each with their own project management processes or requirements, DFS is unlikely to find a process that meets everyone’s needs. DFS currently juggles having to account for its labour and materials costs, and effectively planning, tracking, and reporting on its projects, all the while trying to be timely and efficientFootnote 11.
- Uncertainty and perceived loss of control felt by research centres, especially those with a high reliance on DFS for their core research and operations, those in regions outside of central Canada, and those who had their own design and fabrication services prior to the centralization of DFS.
- Perceived barrier of us vs. them between DFS and certain research center staff, especially with regards to control over time coding/allocations and costing.
These drawbacks currently have an impact on DFS’s relationship and work with four research centres in particular on the following issues:
- DFS’ costing/model-making methods (time required) and their impact on OCRE St-John’s bids for external contracts.
- DFS’ Aircraft Maintenance Engineers working on aircraft modifications to ensure AERO’s executive accountable to Transport Canada for airworthiness is meeting requirements related to control over resources under the Flight Research Lab’s experimental flight licenses.
- DFS’ costing and time coding and their impact on EME project planning in Vancouver, especially for external clients.
- DFS’ maintenance of HAA telescopes (a highly specialized process for which DFS staff are trained and experienced), and consultation/collaboration regarding the purchase of DFS equipment and resource decisions in Victoria.
In all instances, DFS management has reached out to research centre management (either the director general, directors of research and development, or directors of operations) and made efforts to identify solutions. However, due to turnover in both the DFS and research centre management teams over the years, no formal agreements have been identified or documented. It is in the best interest of both DFS and the research centres to work together to find a way forward that meets everyone’s needs.
4. Efficiency of NRC’s Design and Fabrication Services
Research centres recognize and appreciate the skills, expertise, and professionalism of DFS staff. DFS is aware of the areas where it needs to build up skills and experience, and is fairly agile in shifting work among its 13 shops. DFS is planning ahead for a wave of retirements. However, experience is a valuable commodity that DFS has not always been able to maintain as staff with many years of service retire. While DFS capacity is generally sufficient to meet research centre needs, there is a disconnect between DFS and the research centres regarding work scope, timing and project prioritization, which causes some pressure and tension around resourcing. Additionally, any increase in demand would be difficult to meet without additional resources.
DFS generally has the facilities and equipment required to meet its objectives. Like the rest of the NRC, DFS faces difficult decisions with regards to upgrades and replacement of ageing equipment, and working within older facilities that do not provide the best or safest layout. DFS is heavily reliant on a computer and network infrastructure, whereby outages or disruptions pose a risk to the majority of its operations. Several pieces of DFS equipment are critical to meeting the needs of research centres, and upgrades or replacement of these often falls outside of minor capital purchases managed by DFS. To date, few opportunities to co-invest in the acquisition of equipment or new facilities have been explored, though the increased need for advanced manufacturing capabilities across the NRC may provide new opportunities for DFS.
Main supporting evidence: Cross-NRC poll, interviews, document review, data review.
Human Resources
While DFS is starting to feel stretched, overall it has the required human resources to meet its current objectives. However, any increase in demand would be difficult to manage as DFS currently offers a broader range of services than it did five years ago, but with a smaller workforce. DFS staff are recognized by NRC research centres for their abilities and expertise, which DFS continues to adjust to meet the needs of research centres. This expertise is important to maintain as long-serving DFS staff approach retirement. DFS’ capacity and ability to meet NRC needs can be improved through increased communication about project objective and scope, early involvement of DFS in projects, and project prioritization at the research centre level.
Expertise
DFS staff are highly skilled: DFS staff across the NRC are generally perceived as professional, highly capable and skillful. Research centres appreciate the ability of DFS staff to understand their requirements, and deliver quality products and services in a broad range of fields. DFS management is aware of employees who require further training to meet client needs, and has invested in training to stay abreast of continuing developments in design and fabrication (e.g., advanced fabrication processes, 3D scanning, CAD).
“DFS has a long history of working with our research centre. They have knowledge of [our research area] that extends well beyond normal design/fabrication contractors. This knowledge leads to the DFS group being part of the team rather than an external contributor.”
New capability needs are being addressed: As the need for digitization and advanced manufacturing continue to grow across the NRC, DFS has responded appropriately (e.g., growth of its advanced technologies group, launch of an NRC virtual reality community of practice, increased number of 3D printers, and associated training.)
Capability gap in model-making for Aerospace wind tunnel testing is being rebuilt: DFS capabilities in wind tunnel model-making have progressively been lost over time. To offer a true turnkey solution to its clients, AERO’s Aerodynamics Lab requires DFS support with this competency. DFS is aware of the gap and is trying to shift resources to accommodate AERO, but this is a specialization that will take time to build back up, and that must be maintained by regular use.
Retirements and knowledge transfer
DFS is planning ahead for a wave of retirements: A large portion of DFS employees have been at the NRC for over 20 years (30% in FY 2018-19). These are all technical officers, mostly at the TO4 and TO5 levels, and many are senior leads or supervisors. Additionally, 57% of DFS employees are over the age of 46, and 29% will be eligible for retirement by 2023-24. DFS is planning ahead by identifying areas where resources can be shifted to allow for new capabilities, or where increased capacity is needed. With the support of the Human Resources Branch, DFS is also identifying employees who have the potential to move into supervisory or management roles as upcoming retirements will also impact DFS’ management team.
Figure 8. New hires are increasing but 30% of DFS employees have more than twenty years of service.
Figure 8 ‑ Text version
Figure 8 is a horizontal bar graph depicting the proportion of DFS employees with more than 20 years of service and the proportion of DFS employees with 0 to 5 years of service, for each fiscal year of the evaluation period (2012-13 to 2018-19). While the proportion of new hires is increasing each year, results indicated that in 2018-19, 30% of DFS staff have over 20 years of service.
2012-13 | The proportion of DFS staff with over 20 years of service is 25%, and the proportion of DFS staff with 0 to 5 years of service is 12%. |
---|---|
2013-14 | The proportion of DFS staff with over 20 years of service is 24%, and the proportion of DFS staff with 0 to 5 years of service is 12%. |
2014-15 | The proportion of DFS staff with over 20 years of service is 25%, and the proportion of DFS staff with 0 to 5 years of service is 13%. |
2015-16 | The proportion of DFS staff with over 20 years of service is 21%, and the proportion of DFS staff with 0 to 5 years of service is 19%. |
2016-17 | The proportion of DFS staff with over 20 years of service is 23%, and the proportion of DFS staff with 0 to 5 years of service is 21%. |
2017-18 | The proportion of DFS staff with over 20 years of service is 31%, and the proportion of DFS staff with 0 to 5 years of service is 19%. |
2018-19 | The proportion of DFS staff with over 20 years of service is 30%, and the proportion of DFS staff with 0 to 5 years of service is 22%. |
Knowledge transfer can be challenging when long-time employees retire: Succession planning does not equal retention of all knowledge and expertise, especially when positions are reallocated within the branch to a different function, as DFS has been doing to align its resources to meet demand. Long-standing DFS employees have often accumulated a large quantity of knowledge during their careers. Not only do they have specific knowledge and experience in certain design and fabrication processes, but also related to certain DFS equipment and facilities (operation and maintenance), specialized materials characteristics, and unique client needs. As such, both DFS and research centres have been affected by the loss of long-standing DFS employees in the past, despite efforts to transition information. Research centres and DFS staff in regional satellite shops are particularly affected by retirements because they have a smaller pool of resources and rely more heavily on DFS for core elements of their work. In particular, when there are resource changes within DFS, research centres sometimes want to know if:
- a replacement will come in at the same classification or lower
- there will be an opportunity to shadow the retiree
- there is an opportunity to bring in someone with a different skillset to meet their needs
- the position will be reallocated to another DFS shop
Consequently, when they are not informed or involved in discussions about what happens to the position, the research centres fear that specialized DFS knowledge regarding their research and technology areas can be lost. Increased communication with the research centres about retirements and knowledge transfer may facilitate the transition.
Capacity
Capacity is appropriate to the current workload as DFS is able to shift work among its 13 shops:
In general, DFS appears to have sufficient capacity to meet current workload across the NRC despite some unstaffed positions and variability of workload across different sites. DFS supervisors are typically well integrated into research centre project teams in satellite shops and work closely with them to anticipate workload, although this is not always predictable. DFS’ centralized model is effective in managing that unpredictability by distributing the variable workload across its 13 shops. One area where an additional resource may be needed is in Boucherville where there are currently four employees, but there have been five or six in the past. In consultation, the AST staff consistently expressed a need for greater support from DFS in their feedback, especially at end of fiscal year. DFS plans to add an additional resource in Boucherville following an upcoming retirement in another shop.
Current capacity cannot meet an increase in demand without affecting quality or timeliness: The number of DFS staff has consistently dropped over the years from 113 in 2012-13 to 99 in 2018-19. As a result, DFS staff are relying on fewer resources to meet the diverse needs of the NRC’s 14 different research centres. Additionally, certain vacant fabrication positions have been reallocated within DFS to meet increased demand across the NRC for advanced technologies (e.g., 3D printing and scanning, and virtual reality). DFS’ advanced technologies group was created in 2017 and has since quickly doubled from 5 to 10 positions. Additional positions within DFS were also shifted into its project management office to support consistent project management as the group’s numbers fluctuated between 6 and 10 positions between 2013 and 2020. Should the demand for DFS services increase across the NRC, and should DFS be asked to further expand its capabilities into new areas of expertise, DFS will not be able to meet the need if current quality and responsiveness levels are to be maintained.
Facilities
DFS facilities generally meet the needs of the NRC. DFS is fully aware of equipment and facilities in need of upgrades or replacement and has addressed these on an ongoing basis within the limits of its budget. NRC IT infrastructure and networks are important to DFS’ daily operations – any disruptions directly affect productivity. Similarly, failures of key DFS equipment have a significant impact on the operations of the research centres that rely on DFS. As scientific research infrastructure is reviewed both within the NRC and the federal public service, it will be important for DFS to find ways to match needs and budgets, and investigate opportunities for co-investment.
DFS has maintained and improved equipment/facilities but these require continued investment: DFS actively monitors, maintains and upgrades its facilities, which are housed in older buildings where the layout is not always practical and space is not always sufficient. Safety upgrades have been implemented in recent years (e.g., machine guarding) and newly purchased equipment always includes increased features for user safety. DFS supervisors have current lists of equipment in need of repair or replacement for its 13 shops and are attuned to risks and challenges related to these. The average age of DFS fabrication equipment is 20 years. Smaller shops have one to two pieces of equipment in need of upgrades or replacement while larger shops have 3 to 5.
Like the rest of the NRC, DFS faces difficult decisions about where to prioritize its investments with a limited budget and ageing equipment that can require anywhere from hundreds of thousands to millions of dollars to replace, especially for the more unique, complex or precise machinery. DFS equipment is generally expensive to purchase, upgrade or replace. As a result, DFS can only purchase 1 or 2 key pieces of equipment per year with its reduced minor capital budget (see Figure 9), leaving most shops in the cue for another year. This makes it difficult to keep up with technological changes and maintain relevant equipment, especially with regards to DFS’ advanced technologies group.
Figure 9. DFS’ minor capital budget has been reduced by a little over a third in the last two years.
Figure 9 ‑ Text version
Figure 9 is a line graph depicting DFS’ minor capital budget and actuals between fiscal years 2013-14 to 2018-19. Overall, the line graph indicates that DFS’ minor capital budget has been reduced by a little over a third in the last two years. Specifically:
2013-14 | Minor capital budget: $920,000; Actuals: $691,424 |
---|---|
2014-15 | Minor capital budget: $919,335; Actuals: $1,040,365 |
2015-16 | Minor capital budget: $915,000; Actuals: $728,244 |
2016-17 | Minor capital budget: $973,000; Actuals: $854,634 |
2017-18 | Minor capital budget: $580,992; Actuals: $581,745 |
2018-19 | Minor capital budget: $588,493; Actuals: $675,120 |
While DFS received additional major capital funding for facility modernization in 2015, the funding was canceled in 2017 after $5.57 million of the initially approved $9.39 million was spent due to NRC-wide budgetary cuts. In 2017, a proposal for a second phase major capital investment in the modernization of DFS facilities was unsuccessful. When full replacement of a machine is not affordable, DFS works to upgrade key parts of the equipment to extend its lifecycle.
DFS equipment failures affect research centre projects and exposes them to cost and time overruns: As many upgrades or replacements are for specialized equipment requiring capital funding, which DFS cannot readily access, research centre projects have been exposed to significant delays or cost overruns when certain DFS equipment has broken-down (e.g., breakdown of SLS 3D printer can lead to project delays or sub-contracting costs in the tens of thousands of dollars). At times, the equipment has been repaired quickly enough to avoid delays or expensive external contracting to meet timelines. However, certain essential pieces of DFS equipment in need of repair or upgrades, continue to pose a risk to research centre projects (e.g., Liné milling machine in St-John’s, SLS 3D printer in M4, waterjet cutters in Boucherville and Ottawa).
DFS decisions regarding where to prioritize minor capital funds and when to seek major capital funds have a significant impact on the operations of certain research centres, especially those that rely heavily on DFS. For example, the Okuma CNC machine in Victoria needs to be replaced, having already caused project downtime for HAA. Although DFS is not able to afford a new machine, it is maximizing available funds by replacing a key component of the Okuma, and thereby extending its lifecycle by another 10 to 15 years.
DFS operations are dependent upon a reliable IT and network system: Through the years, fabrication has evolved from a manually intensive trade to a highly computer-reliant profession. Ninety-five percent of DFS’ fabrication machines are connected to a computer and network, and its design and advanced technology groups are fully reliant on these. As a result, regular upgrades or replacements of fabrication equipment older than 10 years becomes important to ensure proper functioning. Additionally, any network outages significantly impact DFS’ ability to operate, and jeopardizes research centre timelines and project costs. Due to their distance from network servers and unpredictable weather, DFS shops in Newfoundland and British Columbia must contend more frequently with unstable and unreliable network connections, thereby affecting their support to research centres.
Opportunities for co-investment in DFS equipment and facilities: As DFS continues to anticipate the needs of the NRC, opportunities for co-investments and collaboration in purchasing equipment should be explored as a way to access additional capital funds for equipment renewal, replacement and purchase. In addition to reducing the risk of failure of equipment that is important to research centres, areas of common interest across the NRC such as 3D printing may also be most promising for co-investments. To this end, DFS is currently developing a proposal for an NRC 3D printing facility.
Ongoing planning and discussions for Laboratories CanadaFootnote 12 may also present opportunities for DFS to further support NRC research centres and other government partners. As Laboratories Canada works to identify the future needs of Canada’s federal science infrastructure, DFS’ cross-sector capabilities and equipment could be seen as a valuable component of future state-of-the-art science hubs that support collaboration across departments.
5. Performance of NRC’s Design and Fabrication Services
DFS works collaboratively across the NRC on a wide range of projects, supporting design and fabrication needs of researchers and equipment/facilities. Over time, DFS has built a broad expertise and knowledge base tailored to the research centres’ unique needs. It has thereby contributed to expanded research capabilities, turnkey service offerings for NRC clients/collaborators, innovation, global influence, and scientific impact of the NRC.
Main supporting evidence: Cross-NRC poll, interviews, document review, data review, case studies.
Supporting NRC facilities and scientific equipment: DFS supports research centre facilities and equipment in various ways. DFS has engaged in the design and layout planning of new facilities (e.g., AERO’s new world class Centre for Air Travel Research, NRC’s new advanced materials research facility in Mississauga, and the new Advanced Manufacturing Collaboration Centre in Winnipeg scheduled to open in 2021).
DFS workshops across the country support researchers when their equipment or facilities are in need of repair or break during testing and need to quickly resume their operations. For example, when a fan blade in the M17 Altitude Icing Wind Tunnel malfunctioned, spinning at a speed of 3,000 RPM, it caused a lot of damage. The original drawings for the wind tunnel were from 1959 and not necessarily accurate. DFS was able to scan and reverse engineer the fan blades and fabricate approximately 30 different parts so that AERO could get the busy wind tunnel up and running again within a month.
Finally, DFS has helped research centres build custom equipment that sets them apart and provides them with greater influence in their fields. For example, DFS designed, fabricated, stress-tested and supported installation of a large-scale dynamic wind rain rig system for AERO. This new equipment enables AERO to perform testing that it did not previously have the capability to perform for clients. AERO is now currently the only laboratory in America with the capability to provide wind tunnel testing on inclined cables under dry-wind and rain-wind conditions for 1:1 scale model at high wind speed.
In fact, without DFS, certain research centres would have to abandon their projects for acquiring customized high precision equipment due to the prohibitive cost of doing so without DFS (e.g., METRO stakeholders indicated that several projects would not have been undertaken due to a value of more than $1 million without DFS support). For example, DFS designed and fabricated a new cryostat that operates with an acoustic gas thermometer for METRO. Acoustic gas thermometer cryostats are rare, as only countries or institutions with significant resources can build their own. This equipment will allow METRO to further innovate, influence, and keep up with other developments in this area. In fact, with this new piece of equipment and associated research capabilities, Canada will join and collaborate with a handful of countries that significantly influence international policy and research.
“It’s rare and only the big players have these. By having one, it puts us in a very small club of countries that can call the shots.”
“If we had to go external for everything, it would cost much much more, and we probably wouldn’t have been able to get it done with our budget. Without DFS, this would probably be a no-go from the start.”
Text version
There are three images, from clockwise left to right:
- AERO wind rain rig
- DFS model of AERO’s Centre for Air Travel Research
- Water jet tension tester for EME
Meeting NRC objectives: DFS’ work has supported research centres in meeting the NRC’s three main objectives: scientific and technological advances, innovative businesses grow, and evidence-based solutions inform decisions in government priority areas.
Scientific and technological knowledge advances: Though DFS assumes a support role and does not directly work towards scientific and technological advances, a large portion of its work enables research centres to make these advances. Research centres that are highly dependent on DFS emphasize that the service is critical to their work. For example, HAA relied on DFS support to fabricate a very challenging and unique cryogenic spectrograph, Canada’s contribution to the SPIRou project, which is one of the most precise infrared velocimeters in the world. Not only does the instrument provide Canadian and international astronomers with the opportunity to make important discoveries, but it has also helped raise HAA’s profile in the international astronomy community. As such, HAA was recently awarded a contract for building a similar instrument in collaboration with the European Southern Observatory. This instrument has the potential to be more advanced than SPIRou, leading to further innovation.
DFS supported the RCMP in a novel project in 2019 when they were asked to create 3D reproductions of the skulls of unidentified male remains. It was DFS’ first time scanning and printing biological objects and DFS had to adapt its processes to accommodate for the challenge presented. The 15 skull models created by DFS were then shipped to the New York School of Arts for facial reconstruction. Publicity about the unique project led to the identification of a missing person and important closure for a family.
Text version
There are two images depicting DFS’s recent work for the RCMP.
The first picture is an image of a DFS employee using a 3D printer, and the second picture is the final product for the RCMP project, four 3D printed skull models (of the total of 15 printed).
Although the Nanotechnology Research Centre (NANO) has not been a regular user of DFS services, recent collaboration with DFS on a joint project with HAA, and exposure to DFS capabilities, has opened the door to new projects. In particular, NANO is counting on DFS to help it manufacture things that have never been built before – like a special open source microscope whose blueprints will be made publicly available.
Innovative businesses grow: Research centres that work closely with industry rely on DFS to provide turnkey and innovative solutions to their clients that cover the range from product conception, to design, fabrication and into testing and data acquisition and analysis. For example, AST’s cold-spray additive manufacturing team in Boucherville regularly relies on DFS’ on-site advice and support for urgent modifications and fabrication when their external clients are on-site. In St-John’s, OCRE benefited from support when DFS staff developed a novel design solution to mount a camera on the underwater carriage in OCRE’s testing tanks. This allowed OCRE’s client, the US Coast Guard, to observe the interactions between their polar icebreaker models and the ice in the tank.
An advantage of DFS support to clients is its ability to work with controlled goods. In 2019, DFS supported AERO client Bombardier’s Aerospace Experimental Group in making modifications to its Global 6000 wind tunnel model. DFS deployed a team to Montréal to support on-site inspection and was able to introduce the Bombardier group to novel inspection methods.
Evidence-based solutions inform decisions in Government priority areas: DFS provides support to other government departments indirectly through research centre projects. For example, DFS helped AERO install sensors on one of its aircraft in support of a joint project between AERO, Environment and Climate Change Canada, and university partners for oil-sands monitoring. DFS also directly supports other government departments through its own revenue-generating work. Notably, DFS has engaged in regular support of Defence Research and Development Canada through the years in the design and fabrication of multiple solutions for military equipment, vehicles and facilities. In particular, DFS has worked closely with the Canadian Navy in designing a support system for an antenna and radio frequency distribution unit to be installed on Canadian Forces Marine Coastal Defence Vessels without modification to the vessel, and with the ability to withstand moderate to very rough waves.
6. Conclusion
Relevance
DFS provides important support to NRC research centres. In particular, AERO, HAA, OCRE, EME in Vancouver, and AST in Boucherville depend on DFS for a large part of their operations, and research and development activities. Factors that have enabled the use of DFS across the NRC include DFS’ ability to iterate and collaborate closely with research centres, understand and respond to their specific and unique scientific and technical needs, and provide high precision and quality support.
Awareness of DFS and attitudes towards the service vary widely across the NRC. Some are unaware of DFS’s existence, or how to access services, while others are dedicated clients who value and rely on DFS. Another group of DFS users are frustrated by multiple barriers, and either reduce or avoid use of DFS as a means of circumventing barriers. The interaction between DFS costing and research centre pricing (including time coding as it relates to project management), and misalignment of expectations about timelines and responsiveness, are the main barriers that affect collaboration between DFS and the NRC’s research centres. In research centres where DFS has co-located shops, there is some tension around roles and responsibilities, and accountability. These feed into existing frustrations around project planning, costing and pricing. To address all barriers, DFS must initiate and lead intentional collaboration and communication with the research centres, who must also engage in good faith.
DFS provides an important and valuable service to the NRC. Its centralized co-located model is comparable to that of other global research and technology organizations, and allows for close collaboration with NRC research centres. DFS’ model should be maintained as it also promotes the standardization of safety and quality protocols, and a better matching of capabilities, capacity, and specialized equipment to research centre needs. By engaging in the current corporate initiative (Finitiative) aimed at the simplification of NRC project management processes, DFS has an opportunity to further clarify its value to the research centres and separate the cost of its services from research centre pricing and project management issues.
Efficiency
Over the years, DFS has shifted work and resources across sites and built up capabilities to support new areas of need across the NRC (e.g., digitization and advanced manufacturing) all while experiencing a reduction in workforce. Increased demand would have an impact on DFS’ current quality and timeliness standards. With regards to facilities, DFS equipment is meeting current needs, though DFS’ diminishing budget means that DFS and the research centres must contend with the impact of equipment failures and breakdown on their project costs and timelines. New ways to co-fund DFS equipment and facilities could provide relief and ensure DFS’ support to NRC operations and mandate is maintained or even enhanced.
Performance
DFS supports a broad range of projects across the NRC, in addition to supporting other government departments and industry. DFS staff actively engage in the work of research centres and offer services from conception to fabrication, whether it is related to a ship model, modifications to an aircraft, the design or repair of an NRC facility, specialized equipment and parts for metrology, mining, or astronomy. Much of this work is unique and specialized, and supports research centres in being recognized for their expertise nationally and internationally.
DFS is an important support service within the NRC. Without it, research centres would either have to look externally to meet their design and fabrication needs or forfeit the development of custom specialized scientific equipment, or participation in certain projects, due to the high cost, specialized expertise, and effort associated to these.
7. Recommendations
Recommendation 1
DFS should work with relevant stakeholders (i.e., NRC Finance and Procurement Branch, Business Management Services, and Finitiative) to clarify DFS’ costing model and separate it from research centre pricing practices.
In particular, this may include the following considerations:
- A way for DFS to track and show the value of its services separately from research centre pricing and time coding
- Best practices in estimating level of effort and costing for exploratory research and development
- Different labour rates for exploratory research and development vs. straightforward design and fabrication services
- Verifying the appropriateness of DFS’ current facilities recovery rate for all equipment
Once clarity is obtained with regards to DFS costing in relation to research centre project management and pricing practices, it should be documented and communicated at all levels of the NRC.
Rationale
There is much confusion, mixed messaging, and inconsistent practices across the NRC’s 14 research centres in applying the cost of design and fabrication services to external pricing and internal project management processes. This has created a barrier between some research centres and DFS, while causing frustration and avoidance of the service in some cases. Although DFS has recently engaged in activities that promote greater awareness of its costing model, it is not enough to counter ingrained beliefs in the research centres about the cost of DFS, and the many interacting factors that affect costing and pricing in the research centres.
Greater clarity across the NRC is required. Evaluation findings tied to DFS cost/price barriers are consistent with observations from the Finitiative project management simplification exercise. As Finitiative continues to identify and implement various changes to NRC project management practices, there is an opportunity for DFS to actively contribute to the process and ensure that changes lead to improvements in DFS’ work with the research centres, especially with regards to costing, pricing and time coding. As DFS does not establish NRC costing and pricing practices, accountable branches for costing (Finance and Procurement Branch) and pricing (Business Management Services), should be involved in activities that aim to clarify and communicate appropriate processes.
Recommendation 2
DFS should maintain its current centralized service model, with co-located workshops, to ensure the continued provision of integrated high quality support to NRC facilities and research, development, and technology activities.
Rationale
DFS’ operational model is similar to that of other global research and technology organizations. As a centralized enabling function within the NRC, DFS is able to maximize its service offering and adapt to a wide range of demands from the research centres by shifting work across its 13 workshops. Co-located workshops within the research centres are important to DFS’ delivery of responsive, specialized and high quality work in support of the NRC’s mandate, and should be maintained.
Recommendation 3
DFS should engage in intentional collaboration with the research centres it serves, especially those where it has co-located shops, to ensure common understanding and achievement of goals.
In particular, DFS should consider addressing:
- variable levels of understanding of DFS services (including awareness that DFS can support research centres in deciding whether some projects are better suited to DFS or an external service provider)
- unclear points of access to DFS for non-co-located research centres
- lack of documented processes with regards to clarifying specific responsibilities and accountabilities between DFS and certain research centres (e.g., HAA for telescope maintenance and AERO for DFS aircraft maintenance engineers)
- need for regular consultation and collaboration with co-located research centres on:
- identifying research centre needs and changes to on-site DFS resources (i.e., staffing and equipment)
- integrated planning of all research centre projects with regards to DFS resources and timelines
Rationale
DFS has the mandate to support NRC research centres in their design and fabrication needs. The breadth of expertise and areas of research and development that DFS is asked to support across the 14 different research centres is substantial and as such, prone to much variability in terms of needs, expectations, and processes. While DFS has actively engaged in outreach and relationship building with individual research centres, it often either takes place management to management (with little trickle down), or at the working level between DFS staff and research centre personnel in an informal manner. As different elements (e.g., cost, price, time coding, project management processes, resource allocation, equipment failures or replacements) combine and influence how research centres perceive DFS and interact with the service, a more structured approach to engagement and communication may be beneficial to all. This requires active engagement and willingness to collaborate from both DFS and the research centres, but must be led by DFS as the service provider.
8. Management Response and Action Plan
Recommendation | Risk‑level Associated with not Addressing Recommendation | Management Response | Measure of Achievements | Proposed Person(s) Responsible | Expected Date of Completion |
---|---|---|---|---|---|
1. DFS should work with relevant stakeholders (i.e., NRC Finance and Procurement Branch, Business Management Services, and Finitiative) to clarify DFS’ costing model and separate it from research centre pricing practices. | Medium | Response: Accepted | Director General, DFS | April 2021 | |
Action 1: DFS will engage with key CBIs to review the existing costing model (including upcoming changes from Finitiative’s contracting-in and project management practices) to clarify and align with research centres’ approaches. |
|
||||
Action 2: Implement a communication strategy to clarify DFS costing model in response to the work noted in Action 1. |
|
Director General, DFS | September 2021 | ||
2. DFS should maintain its current centralized service model, with co-located workshops, to ensure the continued provision of integrated high quality support to NRC facilities and research, development, and technology activities. | N/A |
Response: Accepted DFS will maintain its current centralized service model with co-located workshops |
No action required | N/A | N/A |
3. DFS should engage in intentional collaboration with the research centres it serves, especially those where it has co-located shops, to ensure common understanding and achievement of goals. | Low | Response: Accepted | Director General, DFS | February 2021 | |
Action 1: DFS will establish MOUs with AERO’s Flight Research Lab and HAA to clarify specific responsibilities and accountability between research centres and DFS. |
|
||||
Action 2: DFS will create a formal process to guide its collaboration with co-located research centres (AERO, AST, EME, OCRE, HAA, METRO). |
|
Director General, DFS | September 2021 |
Appendix A – Methodology
Document review
Internal and external documents were reviewed to provide context and to complement other lines of evidence in assessing relevance, performance, and efficiency. Internal documents included, but were not limited to, operational and strategic plans, major capital investment documentation, internal DFS quality audit reports, publications, media coverage, and various DFS presentations.
Data review
DFS administrative and performance data for 2012-13 to 2018-19 were reviewed to provide information on inputs (i.e., resources), outputs, and outcomes. This included financial data, human resources data, project data, and labour sharing data.
Key informant interviews
A total of 69 stakeholders were interviewed as part of this evaluation. This included three focus groups with staff from three different research centres (n=16), interviews with DFS management (n=10), telephone interviews with low-use research centre management teams (n=15), case study interviews (n=6), and interviews with research centre clients (n=12). Written interview responses were also obtained from 10 research centre clients. This information was used to complement other lines of evidence and to contextualize quantitative information.
Cross-NRC poll
An NRC-wide poll was conducted to better understand why (and how) NRC research centres use or do not use DFS for their design and fabrication services. An email invitation was sent to NRC staff and publicized through the Echo newsletter. A link was also made available on the NRC’s intranet for 14 days (January 14 to 28, 2020). A reminder email was sent midway through the poll period. A total of 345 NRC staff completed the poll. There were respondents from all 14 research centres (min. n=7, max. n=65), and 4 from other corporate services. Poll respondents were well-distributed between regular/high users of DFS (32%), those who sometimes use the service (30%), and those who never use DFS (38%).
Case studies
Five case studies were conducted – each profiling a specific project for DFS’ five largest research centre clients. The case studies focused on the extent to which DFS met the needs of the research centres and their external clients, provided quality service and value, and contributed to positive impacts and outcomes. The case studies included interviews with key research centre staff for each project, as well as a review of supporting documentation (i.e., publications, news articles, web sites).
International comparison study
Twenty two research organizations were identified and approached by email to inquire about their design and fabrication services. Feasibility telephone interviews were conducted with 10 organizations and 8 agreed to participate in semi-structured telephone interviews. Interviews were typically conducted with the head of design and fabrication services or in a few cases, with an international liaison (see Appendix C for more information on the results of the study).
Limitations and mitigation strategies
Data quality
There were limitations associated with the quality of the project data. For example, the data required considerable recoding of projects to research centres and work groups. Additionally, approximately 40% of projects were missing project cost data.
Mitigation
DFS was consulted during recoding to ensure appropriate attribution of projects. To mitigate the gaps in project data, other lines of evidence were examined (e.g., interviews, cross-NRC poll, data from NRC Finance and Procurement Branch).
Availability of interviewees
5 focus groups (scheduled for March and April 2020) with key user groups within METRO, AERO and AST were cancelled due to the COVID-19 pandemic. Interviews scheduled with external paying DFS clients were also canceled due to the pandemic.
Mitigation
A sample of the internal focus group participants were invited by email to submit their written response to the focus group questions, or schedule a telephone interview as an alternate means of participation. A few additional research centre staff were approached in some cases when no response was received, to ensure appropriate representation of each research centre.
External client interviews were not rescheduled or moved to a written format as they represent a small portion of DFS projects (5%) and the evaluation team did not wish to burden clients during a pandemic. Instead, news clippings, NRC Communications Branch communications, and publications were used to supplement information provided by internal DFS interviews with regards to DFS impacts for external clients.
Potential poll response bias
As with any survey, there were limitations associated with the generalization of results to the larger NRC population. Over half of the respondents (54%) were from 4 research centres (AERO, AST, CONS, and EME). For this reason, the respondent population may not entirely reflect the larger total population.
Mitigation
To mitigate this limitation, no findings were based solely on poll results. Poll results were examined in conjunction with other lines of evidence.
International comparative study
Identification of comparable design and fabrication services
As design and fabrication type services are typically an internal support service or integrated within research group, they are rarely profiled on an organization’s website, making them difficult to identify. Different nomenclature (engineering, fabrication, machine shop, manufacturing, etc.) among organizations also complicates identification of comparable services.
Mitigation
NRC’s international branch shared contact information of colleagues who are part of an international research and technology organization benchmarking exercise with the NRC-Evaluation team. Emails asking for help in identifying suitable contacts within each organization were sent, along with a reminder email. Additionally, NRC’s liaison officer in Germany provided assistance in connecting the evaluation team to NRC researchers with relationships with German researchers to broker introductions. NRC-Evaluation also approached other NRC researchers to broker introductions to international colleagues as applicable when identified through data collection processes. Unfortunately, despite these efforts, it was not possible to interview someone from Brookhaven National Laboratory (although some information was provided by email), and Natural Resources Canada as interviews fell through during the COVID-19 pandemic.
Comparative analysis
Not all organizations provided complete information or information that can easily be compared to others.
Mitigation
Efforts were made to follow-up after interviews to obtain clarifications as needed. Additionally, individual profiles were sent to all participating organizations for validation before finalizing. However, as this was done during the COVID-19 pandemic, 2 organizations did not respond to requests to validate.
Appendix B – DFS logic model
Logic model
Logic model ‑ Text version
Inputs |
|
---|---|
Activities |
In a collaborative, interactive, and iterative/agile manner, DFS engages in:
|
Outputs |
DFS aims to provide high quality and safe turnkey solutions on time and on budget through its:
|
Immediate Outcomes |
|
Intermediate Outcomes |
|
Ultimate Outcomes |
DFS contributes to the research centres’ goals and achievements.
|
Appendix C – International comparison study
An international comparison study of design and fabrication services in other research and technology organizations (RTOs) identified three service delivery models:
Centralized |
Design and fabrication facilities may be geographically dispersed but the management of services is centralized, usually in the executive branch of the research organization with centralized budget management. The NRC's Design and Fabrication Branch is classified as a centralized service model. |
---|---|
Decentralized |
A decentralized model sees design and fabrication services located in each research institute within an organization. There may be some sharing of services (formal or informal) among different reseach institutes, but services are managed and budgeted at the institute level. |
External service delivery |
In an external delivery service model, some small workshops may be available on-site within an organization's research institutes, but the bulk of design and fabrication services are commissioned from outside the organization. |
Centralized
- CSIRO (Australia)
- DST (Australia)
- DLR (Germany)
- NRC (Canada)
Decentralized
- AFRL (USA)
- CSIR (South Africa)
- Fraunhofer (Germany)
- ITRI (Taiwan)
External service delivery
- Brookhaven National
Laboratory (USA) - NIMS (Japan) –
Participation declined
Key findings
- Most research and technology organizations have an internal design and fabrication service
- Participating organizations were evenly split between centralized and decentralized models
- Co-location of services is present in all models
- Several design and fabrication services charge back their cost to research groups
- Cost of services compared to external providers is difficult to assess
- Some services are increasing their use of contractors to adjust to variable demand and reduced budgets