Evaluation of NRC’s Energy, Mining and Environment Research Centre

Alternative format

PDF version (1.1 MB)

Prepared by: Office of Audit and Evaluation, National Research Council Canada

Approval: This report was approved by the NRC's President on July 22nd, 2018

Acronyms and abbreviations

AST Automotive and Surface Transportation Research Centre
BE Bioenergy Systems for Viable Stationary Application Program
EAM Environmental Advances in Mining
FY Fiscal year
GDP Gross domestic product
HEM High Efficiency Mining Program
EAM Environmental Advances in Mining Program
EME Energy, Mining and Environment Research Centre
ES Energy Storage for Grid Security and Modernization Program
IP Intellectual Property
MoU Memorandum of Understanding
NSERC Natural Sciences and Engineering Research Council
NRC National Research Council
NRCan Natural Resources Canada
LIBS Laser Induced Breakdown Spectroscopy
OAE Office of Audit and Evaluation
OGD Other Government Departments
PRC Peer Review Committee
R&D Research & Development
TRL Technology Readiness Level

Executive summary

Program description

Created in 2012, the Energy, Mining and Environment (EME) Research Centre delivers advanced technology solutions to Canada's resource and utility sectors to increase industrial productivity and competitiveness and reduce environmental risk. There are currently four programs within the EME Research Centre: Bioenergy Systems for Viable Stationary Applications (BE), Energy Storage for Grid Security and Modernization (ES), High Efficiency Mining (HEM) and Environmental Advances in Mining (EAM).

To fulfill its objectives, EME targets stakeholders across the entire supply and value chain in the mining and energy sectors, including industry, other government departments (OGDs) and academic institutions. EME also provides a large proportion of its labour to support other NRC programs (40%).

Resources

Between 2012‑13 and 2016‑17, EME's expenditures totaled $124M and it generated $33M in revenue. EME had an average of 177 staff per year, located in Ottawa, the Greater Montreal area and Vancouver.

Long description follows.
 
Long description of revenues

This figure is a horizontal bar chart comparing the revenues (in millions) earned from Technical Services as opposed to strategic research over the course of fiscal years 2012-13 to 2016-17.

Revenue in millions $
Fiscal year Revenue from Technical Services (in millions) Revenue from Strategic Research (in millions)
2016-17 1.5 4.5
2015-16 1.9 4.2
2014-15 2.1 3.7
2013-14 3.2 4
2012-13 2.2 5.7

Scope and methodology

The evaluation of the EME Research Centre and its four programs covered the period from fiscal year 2012‑13 to 2016‑17 inclusive. The evaluation was carried out in accordance with the NRC's approved evaluation plan and TBS policies. The Research Centre and its programs had not been previously evaluated.

Data was collected by NRC's independent evaluation team. The evaluation employed both qualitative and quantitative research methods, including a document and data review, interviews (n = 36), market assessment, client survey (n = 25) and two peer reviews.

Overall evaluation findings

Relevance

EME is focused on areas that are important to the Canadian economy and that are addressing the needs of stakeholders in the energy and mining sectors. EME can enhance the relevance of its work by better understanding stakeholder needs and the R&D ecosystem within which it operates.

There is a need for public R&D in the energy and mining sector as there is minimal private sector investment in R&D. While there are other public and not‑for‑profit organizations that conduct similar R&D in the area of energy storage, bioenergy and mining, the NRC facilities and expertise make it unique in Canada.

EME is aligned with federal government priorities related to the environment, evidenced from its new vision focused on clean technology and clean resources. The activities of EME programs are aligned with EME's new vision. However, clean technology and clean resources are not explicit in the objectives of EME's mining programs.

Appropriateness of capabilities

EME's capabilities support both EME and non‑EME programs. EME embodies a matrix management structure, providing a large proportion of its labour to support non‑EME programs. To this effect, EME has put in place processes to identify the capabilities needed by its programs, as well as some other NRC programs. EME competencies meet the needs of non‑EME programs, however, there were some gaps for EME programs. EME's mining programs would benefit from broad expertise in the mining application domain and hydrology while EME's energy programs would benefit from expertise in energy storage systems, grid operations, biomass conversion technologies, and feedstock supply and logistics.

In addition to gaps in some competency areas, EME programs faced challenges with critical mass of human resources. This had a notable effect on the progress by EME's energy programs toward their objectives. The availability of resources for EME's programs was affected by two factors – the absence of a planned increase in resources within EME for the programs to access and a significant proportion (40%) of EME resources being directed toward non‑EME programs (in support of the matrix management structure at the NRC).

EME's facilities meet the needs of non‑EME programs, and the majority of EME programs. The exception was the ES program, which did not have access to important facilities that would facilitate its achievement of objectives ((e.g., full‑scale installations, Smart Grid Lab). The extent to which EME's facilities were used could not be assessed because facility use was not consistently tracked by EME over the evaluation time period.

Scientific excellence

EME has notable strengths in several areas, including binder development, hydrothermal technologies, anaerobic digestion, battery materials, electrochemistry, laser‑induced breakdown spectroscopy (LIBS), ultrasonics, wear and corrosion, bio‑remediation, bio‑mining, and environmental sensing and monitoring. Overall, EME scientific and technical staff produced research that had a high scientific impact, that is of high scientific quality, and that is recognized internally and externally at the national‑ and international‑level.

Stakeholder engagement

EME worked with many clients and collaborators. Aside from the clients that EME worked with, there was limited awareness of the Research Centre, and more generally of the NRC. EME programs' stakeholder engagement plans highlight the importance of engaging stakeholders from across the value chain in order to achieve program objectives. EME programs have made some progress on reaching key stakeholders along the value chain, however, there are important gaps that need to be addressed. This includes enhanced engagements with utilities, municipalities, remote communities, relevant regulatory bodies, and material and component companies. Over the evaluation time period, EME worked with few universities. Increased collaborations with universities is important to leverage existing expertise, develop networks, keep abreast of developments in the field and to facilitate the movement of low Technology Readiness Levels (TRL) technology to higher TRL levels of commercial scale.

Performance

EME programs completed many projects and for the most part generated revenue that met targets. The mining programs are on track to achieve their objectives, with successes in the areas of LIBS, ultrasonics, bio‑remediation, bio‑mining, sensing and monitoring. The energy programs had notable achievements in the areas of battery electrode materials, small and portable waste conversion systems and pellet binders. Despite these accomplishments, progress made by the energy programs are not sufficient to ensure the realization of their objectives by the program end date. Contributing to this is the breadth of projects conducted by the two energy programs and insufficient resources.

EME had a positive impact on its clients and collaborators, which included the development or commercialization of new technologies or products, growth, increased productivity and decreased costs. One of EME's programs has contributed to initiatives that will directly impact policy and regulations while the other three programs are conducting research that has the potential to do so in the future. Insufficient time has lapsed to clearly see the impact of EME, and its programs, on the value chain. It will be important for EME to address the identified gaps in stakeholder engagement if it wants to have an impact on the value chain in the future.

Recommendations and management response

Recommendation 1:

EME should continue to increase its understanding of the mining and energy sector, stakeholder needs and the R&D ecosystems within which its programs operate.

Management response: Accepted

EME, working with its four programs, will identify ways in which it can increase its understanding of stakeholder needs and the R&D ecosystem within which the programs work. This will include updating and executing each of the four programs' Stakeholder Engagement Plans with a planned approach.

Recommendation 2:

EME should consistently track the use of EME facilities and equipment.

Management response: Accepted

EME will design and implement a consistent method of facility/equipment tracking.

Recommendation 3:

EME should increase awareness of its research capabilities and its programs within the mining and energy sector.

Management response: Accepted

EME will develop and execute a communication/ engagement plan to communicate its capabilities and programs in the mining and energy sectors. The plan will identify different ways EME will pursue to increase its awareness, as well as indicators of success.

Recommendation 4:

EME should ensure the BE program engages a greater number of utility companies, municipalities, provincial agencies and remote communities.

Management response: Accepted

EME will work with the BE program to increase its engagement with the identified stakeholder groups. This will include updating the program's Stakeholder Engagement Plan to include strategies to increase engagement with the identified stakeholders.

Recommendation 5:

EME should ensure the ES program engages a greater number of material and component companies, Canadian utility companies and government bodies that plan and regulate provincial electricity systems.

Management response: Accepted

EME will work with the ES program to increase its engagement with the identified stakeholder groups. This will include updating the program's Stakeholder Engagement Plan to include strategies to increase engagement with the identified stakeholders.

Recommendation 6:

EME should ensure the EAM program engages the regulatory community to a greater degree.

Management response: Accepted

EME will work with the EAM program to increase its engagement with the identified stakeholder groups. This will include updating the program's Stakeholder Engagement Plan to include strategies to increase engagement with the identified stakeholders.

Recommendation 7:

EME should increase its collaborations with academic institutions.

Management response: Accepted

In order to increase engagement with universities, EME will prepare and implement a plan to increase its collaborations with academic institutions. As part of this, a proposal will be developed for a collaborative research centre between EME and an academic institution.

Recommendation 8:

EME should narrow the scope of its energy programs and accordingly make any necessary adjustments to its activities and value propositions.

Management response: Accepted

EME will review the current scope of the two energy programs during FY2019, as outlined in the FY2019 operational plan. Consideration will be given to the capabilities and human resources that the programs have access to.

Recommendation 9:

EME should modify the value proposition of its two mining programs to ensure:

  1. clear alignment with EME's focus on clean resources and clean technology, and
  2. they are attainable and measurable

Management response: Accepted

The value propositions of EME's two mining programs will be revised as part of the program renewal process, to be completed through FY2019.

 

1. Introduction

The National Research Council (NRC) Energy, Mining and Environment (EME) Research Centre delivers advanced technology solutions to Canada's resource and utility sectors. Its unique technology risk management capabilities seek to help companies develop and maintain a globally competitive position, and support quality improvements and cost reductions for vital products and services delivered to Canadians.

In 2017-2018, an evaluation of EME was conducted by the NRC Office of Audit and Evaluation. This project included reviews of EME's four programs:

  • Energy Storage for Grid Security and Modernization (ES), which helps the energy industry overcome obstacles to adopting and using energy storage technologies, including durability, cost and risk associated with the development and implementation of energy storage in Canada.
  • Bioenergy Systems for Viable Stationary Applications (BE), which helps Canadian companies overcome the technical and cost barriers involved in the integration of locally-sourced biomass into stationary energy (heat and power) systems.
  • High Efficiency Mining (HEM), which seeks to improve, develop and commercialize practical technologies —diagnostics, separation and materials —and demonstrate the value of process integration specifically applied to gold, nickel, copper and oil sands.
  • Environmental Advances in Mining (EAM), which targets mining sustainability challenges by developing, demonstrating and validating solutions to problems that drive costs and liabilities of hard rock mining.

Between 2012-13 and 2016-17, the time period covered by the evaluation, EME's expenditures totaled $124 million (average of $24.8M per year). Over this period, EME generated $33M in revenue by working with 183 unique clients and collaborators, the majority of which were from industry. In addition to working with external clients and collaborators, EME staff worked with non-EME programs at the NRC. EME has facilities and approximately 177 staff in three locations – Vancouver, Ottawa and the Greater Montreal Area. A more detailed profile of EME and its four programs is in Appendix A.

EME was selected for evaluation based on consultations with the NRC Senior Management and the work was carried out in accordance with the NRC's approved evaluation plan. The evaluation assessed the relevance and performance of EME through the lens of its four programs. EME's role in the environment was considered in the evaluation in the context of its energy and mining programs. The evaluation used the following methods:

  • Document and literature review
  • Data review
  • Market assessment
  • Key informant interviews (internal staff/management n = 36; external partners/stakeholders n = 10 )
  • Industry client/collaborator survey (n = 25)
  • Peer review (n = 2)

One of the challenges faced in this evaluation was the diverse areas that EME and its programs addressed. In some instances, findings and recommendations were more appropriately presented at the program level than at the Research Centre level. A detailed description of the evaluation methodology and its limitations is provided in Appendix C.

The evaluation report is organized as follows:

  • Section 2 to 6 presents the evaluation findings organized by theme
  • Section 7 presents the overall conclusion
  • Section 8 describes the management response to the recommendations and the actions that will result

2. Relevance

2.1. EME's programs are focused in areas that are important to the Canadian economy.

The energy and mining sectors are significant contributors to the Canadian economy. The energy sector's annual contribution to the Canadian Gross Domestic Product (GDP) between 2013 and 2016 was, on average, 9%Footnote1. The mining sector's annual contribution to the Canadian GDP between 2013 and 2016 was, on average, 8%Footnote2. Within these two sectors, EME programs target several industries, all of which are contributing to the Canadian economy (see table 2.1). In addition, EME's focus on bioenergy has the potential to impact Canadian industries outside of the energy sector. For instance, the manufacturing of bioproducts offers the forestry industry additional markets for its commodities, and it strengthens and diversifies the agricultural sector by adding value to waste and by the development of new cropsFootnote3.

Table 2.1: EME programs target industries that influence the Canadian economy
EME Program Industry Contribution
Energy storage Other Electrical Equipment and Component Manufacturing Accounted for 4% of GDP in 2016
Bioenergy Electric Power Generation, Transmission and Distribution Accounted for 2% of GDP 2016
High Efficiency Mining Oil Sands, Gold, Nickel and Copper Represents 40% of total goods and services produced in the Mining Sector, which accounted for 3% of GDP in 2016
Environmental Advances in Mining Metal Ore Mining Industry Accounted for 1 % of GDP 2016

Source: Statistics Canada

As part of the evaluation, a market assessment was conducted to validate the original prospective economic impact assessments prepared by each program at their inception. Findings from the updated prospective economic assessments suggest that EME programs are focused in areas that have the potential to positively affect the Canadian economy. The updated economic impact estimates, which were calculated for the time period used in each program's original prospective economic assessments, are:

  • $1,330.26M between 2013 and 2019 for the Energy Storage for Grid Security and Modernization (ES) program
  • $200M between 2013 and 2019 for the Bioenergy Systems for Viable Stationary Applications (BE) program
  • $4.70M in 2015-16 for the High Efficiency Mining (HEM) program
  • $1,309.61M between 2016-17 and 2026-27 for the Environmental Advances in Mining (EAM) program

2.2. The objectives of EME programs are aligned with the needs of stakeholders. EME must enhance its awareness of stakeholder needs and their environments to ensure its continued relevance.

The alignment of EME objectives with stakeholder needs is examined on a program by program basis given the diverse areas of EME's programs. This assessment drew in large part on findings from the two peer review committees (PRCs). The two PRCs based their assessment on expert knowledge of the industries, discussions with EME and its programs, a review of program documents and evidence gathered by the evaluation team.

Energy Storage program

The energy PRC confirmed that the focus of the ES program on strengthening the Canadian energy storage industry and on reducing the cost of energy storage systems in utility applications aligns with the needs of stakeholders. The NRC's strong research expertise in battery materials and electrochemistry is well aligned to support the needs of the Canadian energy storage sector to secure a greater share of the global battery materials market. The global energy storage sector is expected to have significant growth in the next eight to ten yearsFootnote4. The large potential economic impact of the ES program presented earlier (i.e., $1,330.26M) is predominately driven by the growth of the global battery manufacturing industry and the role that the NRC can play in supporting the Canadian energy storage sector to enter it. The projected economic impact from the ES program on Canadian utility companies only accounted for $1.26M of the $1,330.26M estimated for the ES program. This is due to the low uptake of energy storage technologies by Canadian utility companies.

In terms of energy storage systems for utility application, the energy PRC was of the opinion that there is interest from utility companies if the cost of available systems and technologies justify it. Literature provides evidence of the need for reduced costs of energy storage systems, as focused on by the program. Lowey (2017), for example, referenced a Canada-wide study that highlighted high installation costs as one factor that contributed to the slower than expected growth of the Canadian energy storage market. Despite this, significant growth is still projected for CanadaFootnote5.

In addition to the cost of energy storage systems, other barriers to the uptake of energy storage systems by utility companies include an existing large amount of hydroelectric capacity, which reduces the need for energy storage in many jurisdictions. The energy PRC highlighted this as well as low demand growth for electricity and low electricity prices as reasons why utilities may not be interested in energy storage for balancing renewable energy resources (e.g., wind, solar) and / or investment deferral for transmission and distribution upgrades. Other documents, however, point towards energy storage as an option to improve grid-stabilization and buffer peak electricity demands, which could in-turn, support a larger share of renewables in the electricity gridFootnote6. This may differ depending on the province as they have different sources of electricity.

Some provinces have started focusing on energy storage in their provincial energy strategies (e.g., Ontario, Alberta, Quebec, Maritime provinces). Therefore, despite limited market incentive for utilities to adopt energy storage systems, there are government / policy drivers. As the program has largely worked with utilities and regulatory agencies in Alberta and Ontario (e.g., through the Energy Storage Roadmap), there is an opportunity for the ES program to continue examining the role of energy storage with a broader range of utility companies and regulatory bodies across Canada. This is important to ensure continued relevance of the ES program.

Bioenergy program

The energy PRC confirmed that the BE program's focus on reducing the production cost of biofuel and biopower aligns with stakeholders needs. Documents and literature indicate that with the increased interest in, and use of, renewable energy sources there is a need to reduce the cost of bioenergy systems to make them competitive with fossil fuel systemsFootnote7 Footnote8 Footnote9. The BE program's emphasis on bioenergy within remote communities is appropriate as these locations have significant potential to use biomass and are interested in biomass-based energy systems due to the high cost of fossil fuels. For example, the Standing Senate Committee on Energy, Environment and Natural Resources estimates that biomass heating can reduce costs by 30-50% compared to oil-fired heating in the Northwest TerritoriesFootnote10.

The energy PRC did caution that to continue to be relevant, the program needs to keep abreast of developments in the bioenergy field and incorporate these into its work. For example, the program focuses on conventional fast pyrolysis oil production when other technologies have produced oils with better properties. Thus, the energy PRC recommended that the program seek to better understand the R&D ecosystem within which it operates to ensure the technologies they are developing align with recent advances in the field. The energy PRC also noted that the target identified in the BE program's value proposition for biofuel production cost reduction are not aligned with currently lower market costs. While there is still a need to reduce the production costs of biofuel, particularly for non-pellet based forms, the targets in the program's value proposition need to be revised.

Environmental Advances in Mining program

The EAM program's focus on reducing environmental costs in the mining sector was highlighted by the mining PRC as one of the significant challenges faced by the sector. The EAM program is therefore addressing an important need. The scale of environmental liability issues in the Canadian mining sector is substantial, and requires sustained and holistic efforts that can be offered by a federal program. The document and literature review provided evidence that environmental liability costs are a significant burden to the Canadian mining sector, and in turn, the Canadian economy. Since 2012, the environmental liability cost has increased rapidly in the three major Canadian mining jurisdictions: Ontario, British Columbia and Quebec, and represents an overall liability cost of greater than $9.1BFootnote11. Further, there is a need for improved tailings management, as Canada has more mine tailing spills than most of the worldFootnote12, as well as a need to deal with acid rock drainage, which has notable environmental impactsFootnote13. These are areas addressed by the EAM program. The mining PRC did highlight that the program can increase its relevance by building linkages with regulatory bodies that deal with environmental issues, at both the provincial and federal levels. Examples of these regulatory bodies include: provincial governments, territorial governments, Environment Canada (Canadian Environmental Assessment Act), Fisheries and Oceans Canada (Fisheries Act) and Canadian Nuclear Safety Commission (for Uranium Mines).

High Efficiency Mining program

The mining PRC confirmed that the HEM program's focus on the enhancement and optimization of process technologies and equipment durability is aligned with the needs of stakeholders in the mining sector. The document and literature review indicated a strong need for R&D in these areas, particularly because mining companies need to reduce costs to remain profitable given lower commodity prices (which decreased in the 2009 global economic downturn and, despite having rebounded, have not risen to those pre-2009)Footnote14 Footnote15 Footnote16 Footnote17. Lower grade ores, challenging ores (i.e., refractory ore) and smaller deposits are also making it necessary for mining companies to adapt innovative technologies to help reduce production costs, improve productivity and extend the life of a mine by making currently uneconomic resources economicFootnote18.

The mining PRC did, however, identify opportunities for the program to enhance its relevance and value to the mining sector by:

  • Complementing current activities with a parallel set of initiatives focused on longer-term, more transformative innovation that can have synergistic impacts across the mining cycle
  • Enhancing its understanding of the R&D ecosystems by conducting an environmental scan of other research labs / organizations
  • Building upon existing methods to identify industry needs and engaging with a broader range of industry organizations (e.g., Mining Suppliers Trade Association and Global Mining Standards and Guidelines Group) to expand its understanding of sector needs. Existing methods used by EME to identify stakeholder needs have included hiring new staff from industry, attending industry conferences and events, and being involved with the Canadian Mining Innovation Council and the Centre for Excellence in Mining.
  • Establishing a program advisory board that includes members from across the value chain.
Recommendation:

1. EME should continue to increase its understanding of the mining and energy sector, stakeholder needs and the R&D ecosystems within which its programs operate.

2.3. There is a need for public R&D in energy and mining.

There is a need for public sector R&D in both the area of energy storage and bioenergy due to low private sector investment in R&D. The document and literature review indicated low private sector investment in bioenergy R&D particularly when compared to investments in fossil fuel R&D. In 2015, Canadian private sector R&D in bioenergy was estimated to be $45M whereas R&D on fossil fuels was approximately $948MFootnote19 Footnote20. Similarly, the private sector has not focused on energy storage R&D. The limited private sector investment in this area is influenced by the current state of the technologies, which are typically low TRL or emerging in natureFootnote21.

Public sector R&D is also necessary in the mining sector due to low investment in R&D by the private sector. Despite the relative importance of the mining sector to the Canadian economy, Canadian mining companies spend a relatively smaller amount on mining R&D compared to other major mining countries such as Australia. In 2013 (the most recent year for which data are available), Canadian mining companies invested Cdn$677M on R&DFootnote22 whereas in 2015‑16, Australian mining companies spent more than twice this amount (i.e., approximately Cdn$1.75B) on R&DFootnote23. The mining PRC also reflected that the private sector's investment in mining-related R&D is heavily influenced by economic fluctuations – as commodity prices decrease so do investments in R&D – creating a need for sustained R&D efforts that can only be provided by a public entity such as the NRC and EME

2.4. The NRC has some unique facilities and expertise compared to public or not-for-profit organizations working in similar areas.

There appears to be some overlap between EME mining and energy programs and Natural Resources Canada (NRCan) CanmetEnergy and CanmetMining programs. However, EME staff pointed out that there are differences in mandate between the NRC and NRCan, making the programs complementary. NRCan's mandate is to support the resource sectors whereas the NRC's mandate is to support industry within the sectors. EME signed a Memorandum of Understanding (MoU) with CanmetMining in October, 2015 to facilitate collaboration and leverage capabilities within the two organizations. EME staff highlighted that there is an opportunity for the Research Centre to further collaborate and leverage existing capabilities with CanmetEnergy. Work is underway to establish a MoU with CanmetEnergy, however, a formal arrangement has not yet been established.

In addition to NRCan, EME's two energy programs appear to be complementary to other national organizations focused on energy storage and bioenergy R&D, as identified through the document and literature review (e.g., InnotechAlberta, Electric Power Research Institute, Centre for Energy Advancement through Technological Innovation). The energy PRC concluded that the NRC has a unique role to play in supporting the Canadian energy storage sector because of its expertise in battery materials and electrochemistry, and its ability to support the early and low TRL research needs of the materials industry. The NRC also has a unique role to play in the area of bioenergy as there is limited infrastructure in Canada for research at TRLs four to seven (i.e., laboratory validation through to prototype demonstration in an operational environment).

The energy PRC, however, did not see a role for EME energy programs in conducting techno-economic analyses. These activities do not draw on expertise unique to the NRC and are commonly offered by the private sector. EME staff, however, felt that the focus of EME's techno-economic analyses is different than those offered by the private sector. According to EME staff, the private sector assesses commercial technologies whereas EME develops tools for analyzing the techno-economics of cutting edge technologies to better understand how research activities can effectively increase the market impact of these technologies.

Much like was the case for EME energy programs, the evaluation identified other public organizations that conduct R&D in areas similar to the EME mining programs (e.g., RIME UQAT-Polytechnique, Centre for Excellence in Mining Innovation, COREM, and Saskatchewan Research Council). However, the mining PRC concluded that there are unique attributes in both EME mining programs. For instance, the HEM program's focus on LIBS and ultrasonic monitoring in particular is one of a kind. The EAM program's focus on a broad spectrum of environmental issues in mining make it unique. As well, the EAM program's focus on the environment is an area where there is a particular need for federal support because it is viewed as a public good.

2.5. EME's vision has evolved to align with current federal priorities. Objectives of some of EME's programs now require adjustment to align with the "clean" focus of EME's new vision.

EME's original vision was "to be the most valued provider of practical and imaginative solutions to critical technology challenges in the energy, mining and environment sectors, enhancing their innovation through the value chain"Footnote24. This vision aligned with the priorities of the federal government at that time (i.e., 2012-13 to 2015‑16) as well as the associated mandate given to the NRC to fill the gap between early stage R&D and commercialization, and to develop and deploy solutions that help improve the innovation capacity of Canadian industryFootnote25. The objectives of the four EME programs, which focus on solving industry challenges and resulting in economic benefits for industry, were developed to align with EME's original vision.

EME has since changed its vision (2017-18) to align with the priorities of the current federal government related to clean technology and clean resources (e.g., as evidenced in the Federal Sustainable Development Strategy, 2016, which promotes clean technology, clean energy and reduced greenhouse gas emissions for effective action on climate change). EME's new vision is "to be the preferred research partner to the Canadian clean technology and clean resource sectors, bridging science and application"Footnote26. The activities of EME's four programs contribute to the new EME mission, however, the way in which the objectives of the mining programs are formulated doesn't make the alignment with clean technology and clean resources obvious. This is because the more prominent focus of the EAM and HEM program objectives and value propositions are the economic benefits to industry. Clean technology and clean resources do not take a central position. Modifying the objectives of the mining programs will be particularly important for the HEM program, as principles of clean resource extraction are not necessarily required to achieve the program's current objective of improved process efficiency and equipment durability. For the EAM program, the majority of the program's activities are built around environmentally sustainable mining practices, however, these are not strongly conveyed in its current value proposition.

3. Appropriateness of capabilities

3.1. EME provides a large proportion of its labour to other NRC programs. EME has put in place processes to identify the capabilities needed by NRC programs.

The NRC uses a matrix management approach whereby programs draw on resources from any research centre. EME is one of the largest providers of resources to programs outside of its research centre. This was influenced by the restructuring of the NRC in 2012 whereby institutes were re-organized into research centres that were aligned with industry sectors. As a result of this restructuring, some ongoing projects were misaligned with EME and were therefore moved to relevant programs in other research centres. Some researchers working on these projects, however, remained in EME due to their area of expertise and the way in which the competencies were organized. For example, as a result of legacy projects, the Automotive and Surface Transportation (AST) Research Centre's Vehicle Propulsion Technology (VPT) program draws on much of the fuel cell expertise located in EME.

The creation of EME also resulted in some researchers having expertise that was no longer required or not required to the same extent by EME. Therefore, productivity of these individuals could be maximised through labour sharing and working on non-EME program projects for which their expertise was needed.

These circumstances have resulted in EME covering significant labour costs for work done in other research centres. Between 2013-14 and 2016-17, on average each year, 19% of EME's labour costs were spent on staff working on projects for other research centres. Aside from EME programs, the largest beneficiaries of EME labour were AST VPT, the Aquatic and Crop Resource Development (ACRD) Research Centre Algal Carbon Conversion (ACC) program and the Ocean, Coastal and River Engineering (OCRE) Research Centre Arctic and Marine Vehicles (MV) programs (see table 3.1 below). In contrast, 2% of EME's annual labour costs were recovered by other research centres.

Table 3.1: Almost two thirds of EME's annual labour costs were spent externally for AST and ACRD
Tier 1 program Average percent of annual labour costs paid by: Average percent of total EME labour costs
EME Host Research Centres Other Research Centres
AST: Vehicle Propulsion Technologies 49 48 2 7
ACRD: Algal Carbon Conversion Flagship 29 70 1 5
OCRE: Arctic 15 72 13 1.7
OCRE: Marine Vehicles 9 88 2 1

Note: EME's total labour costs from 2013-14 to 2016-17 was $ 62.2 M.
Source: Financial data

Programs shown in table 3.1 (above) are classified by EME as Tier 1. Tier 1 programs include the four EME programs as well as four other NRC programs that are most likely to benefit from the capabilities housed within EME, and for which EME will consider the needs of as part of its capability development. The document review found that EME has processes in place to manage its capabilities, including identifying gaps and strategies to meet needs. Overall, capability management is integrated into EME's operational plan and links to program plans. A key tool used by EME to manage capabilities is the Multi-criteria decision analysis (MCDA) which uses a cost-benefit process to consider needs and decide what capabilities to invest in. Interviewed EME staff noted that this process has been effective for the Research Centre.

3.2. EME competencies meet the needs of non-EME programs, however, some gaps were identified for EME programs.

Tier 1 Program Leaders interviewed for the evaluation had very positive feedback about the EME resources who contribute to their programs.  Overall, their needs are met and the expertise required has usually been available. These collaborations have also resulted in positive outcomes that benefited the Tier 1 programs. For example, in collaboration with EME researchers, the ACC program developed a novel piece of equipment related to algae harvesting and processing that will be made available to clients on a fee-for-service basis. For the MV program, leveraging the expertise of EME researchers for a project led to more opportunities benefitting the program.

Findings from multiple lines of evidence indicate that EME lacks certain competencies needed by its own programs to achieve their objectives. Interviews with EME staff identified competency gaps in systems engineering as well as general expertise in mining. The mining PRC also concluded that EME lacked expertise in mining and recommended adding personnel with broad understanding of the mining sector to further enhance EME's mining program's achievements. Limited expertise in hydrology, important for the EAM program, was also highlighted by the mining PRC.

The energy PRC raised concerns about the lack of a critical mass of expertise in energy storage system and grid operations to support the mandate of the ES program. Noteworthy gaps on the bioenergy side, which may affect the program's ability to achieve its objectives, include limited expertise in biomass conversion technologies (i.e., advanced gasification technologies), and feedstock supply and logistics.

While labour sharing benefited many programs external to EME, EME programs have not been able to leverage the matrix management approach to fill competency gaps to the same extent. Reasons for this, as noted by EME staff, include:

  • Required expertise does not exist in other research centres,
  • Competencies needed are already used at full capacity by programs within other research centres (e.g., digital technology and systems engineering skills, available within the Digital Technology Research Centre and the Herzberg Astronomy and Astrophysics Research Centre are fully used by their own programs), and
  • Challenges of working across geographic locations.

EME staff did note that EME programs have, on limited occasion, successfully used the competencies of other research centres (e.g., Digital Technologies, Automotive and Surface Transportation, Construction Research, and Human Health and Therapeutics). This, however, was not the norm. Administrative data supports the finding that EME used limited labour from other research centres. As noted previously, other research centres contributed 2% of the labour costs for EME and its programs. Half of the labour provided to EME was from Aerospace (1%).

Despite challenges in accessing the required competencies, EME programs have been able to fill gaps to a certain extent through collaborations with external partners and academia. For example, the ES program leveraged the grid integration expertise of a private sector company, PowerTech, on a collaborative project the program conducted with the Alberta Electric System Operator.

EME staff noted that even with the correct scientific or technical knowledge, applying these skills to new areas such as energy and mining takes time. They also noted that with the previous focus on generating revenue, researchers had less time to spend developing competencies in these new areas. Interview results suggest a movement to mitigate these challenges within EME through strategic resource allocation. For instance, decisions about the resources allocated to a project are heavily influenced by the Team Leaders, who are expected to develop a vision and direction for their teams as well as limit "ad hoc" resource allocation. This approach is meant to reduce the number of programs and projects that EME researchers support by prioritizing work. This increases researchers' focus and further develops their expertise in strategic areas.

3.3. EME programs have not had access to the number of human resources planned in their business plans. This has affected the progress made by some of EME's programs toward their objectives.

Along with gaps in expertise, EME programs have not had access to their planned number of human resources (see figure 3.1, below). In 2015, three of EME's four programs went through recalibration, where the program life cycle was prolonged to mitigate the limited resourcesFootnote27. While the mining PRC concluded that the resources for the HEM program were sufficient to achieve objectives, they did note that in 2017-18 the EAM program had not accessed the resources it had planned for that year (i.e., 17 FTEs versus the planned 33.4 FTEs). While the impact of the limited critical mass has been minimal to date given the early lifespan of the program, the PRC cautioned that a continued lack of critical mass will affect the program's ability to achieve its objectives. On the energy side, the PRC concluded that the energy programs' insufficient progress toward achieving their objectives was in part due to this lack of critical mass.

Figure 3.1: Over the evaluation time period, EME programs accessed fewer FTEs than planned

Figure 3.1. Long description follows.
 

*BE average FTE for 2013-14 to 2016-17, revised plans reflects 2014-15 to 2016-17 average
**ES average FTE for 2014-15 to 2016-17
***HEM average FTE for 2013-14 to 2016-17, revised plans reflects 2016-17
****EAM FTE 2016-17

Source: Administrative data

Long description of figure 3.1.

This figure is a vertical bar chart comparing the number of originally planned full time equivalents (FTEs), revised planned FTEs and actual FTEs for each of the Energy, Mining and Environment (EME) programs. This figure shows that EME accessed fewer FTEs than planned.

Program Original Plans Revised Plans Actuals
BE 30.25 16.33 14.25
ES 21.67 12.33 12
HEM 47.75 27 20.5
EAM 16.6   15

Despite the impact of insufficient critical mass on the progress made by EME energy programs, the impact on EME clients overall was minimal. Only 5% of client survey respondents said that there was insufficient human resources to fulfill their project objective. That said, EME staff did highlight some instances where insufficient critical mass led to projects being delayed, being delivered at a lower level of quality or being abandoned. A few EME interviewees also mentioned that insufficient critical mass had an impact on researchers and technical officers, some of which are working on over 10 projects at a time. A few EME interviewees felt that in some instances the large workload led to inefficiencies, as well as lower job satisfaction and negative health effects.

The lack of resources for EME programs is largely due to the absence of a planned increase in resources within EME, but also due to the significant proportion of EME labour being shared with non-EME programs, as previously discussed. As a result, many EME staff noted that EME programs must compete with EME and non-EME programs for resources. The challenge faced by EME programs to acquire the resources necessary is amplified when some EME research teams prefer to work under a non-EME program because it better aligns with their expertise and preferences.

3.4. EME facilities met the needs of other NRC programs and three of EME's four programs. Facility use was not consistently tracked.

EME facilities met the needs of other NRC programs, particularly VPT, ACC, Artic, MV (i.e., Tier 1 programs). According to internal interviewees, the facilities that are most used by Tier 1 programs include facilities for battery testing, fuel cells, wet labs, anaerobic digestion and hydro-thermal liquefaction.

For the most part, EME facilities also meet the needs of EME programs and its clients. They were sufficient to meet client needs and to allow programs to progress toward their objectives. EME staff and the two PRC's were of the opinion that the BE, HEM and EAM programs had access to the facilities needed. The mining PRC also noted that facilities were world class. The energy PRC was also impressed with the quality of the facilities they toured as part of the peer review site visits.

50% of client survey respondents reported to a significant extent that facilities were available to fulfill their project objectives on time, budget and within scope.

Where facilities are not available within EME, appropriate partnerships had been built. For example, the HEM program engaged in partnerships with NORCAT and NRCan CanmetMining to access mining specific equipment and mine sites. EME programs, and in particular, BE, also used facilities in other NRC Research Centres, such as Aerospace's combustion centres and rigs. EME staff did, however, note that having to use other organizations' facilities can result in projects taking longer and being more expensive.

68% of client survey respondents reported to a significant extent that the type of facilities and equipment needed to fulfil their project objectives were available.

While having access to some important facilities, the evaluation found some gaps in facilities for the ES program. The energy PRC concluded that the absence of access to full-scale installations will affect the program's ability to achieve its objectives as they relate to energy storage in utility applications. Similarly, interviews with EME staff indicate that the absence of a Smart Grid Lab, which was never funded despite being identified as necessary for the program, will affect its ability to achieve its objectives. The ES program had also planned on having access to an Energy Storage Technology Performance and Abuse Testing Laboratory. The construction of this facility was delayed from a planned opening date of December 2016 to July 2018. As a result, EME staff noted that important projects were delayed.

While gaps in facilities seem to be well understood by EME, the use of available facilities is difficult to assess. The challenge is the inconsistent manner in which facility use is recorded across EME. Simply put, some hours are recorded, some are not and some are tracked differently. This concern was often discussed during interviews and other consultations with EME employees. Incomplete facility use data has affected EME's ability to make informed and strategic decisions about replacement of capital investments, such as closing underused facilities, as well as limited effective project planning and management.

Recommendation:

2. EME should consistently track the use of EME facilities and equipment.

4. Scientific excellence

4.1. EME has notable strengths in several areas of expertise, has conducted research that has had a high scientific impact and that is of high scientific quality.

Internal and external interviewees, as well the mining and energy PRCs found that EME had demonstrated strengths in certain research areas. These include the following:

  • Binder development
  • Hydrothermal technologies
  • Anaerobic digestion
  • Battery materials
  • Electrochemistry
  • LIBS
  • Ultrasonics
  • Wear and corrosion
  • Bio-remediation
  • Bio-mining
  • Environmental sensing and monitoring

The excellence of the research is due to the work of high calibre researchers. The competence of EME researchers is demonstrated through the findings of a bibliometric study that assessed the scientific impact and scientific quality of the work produced by EME researchers. The study found that publications from EME researchers had the greatest scientific impact and were of the highest scientific quality of all Research Centres in NRC's Engineering Division and of NRC overall. Further, their publications received about twice as many citations as the global average (average of relative citations [ARC] = 2.0) and were published in journals about 60% more visible than the global average (average of relative impact factors [ARIF] = 1.6).

Figure 4.1: Publications by EME researchers had a greater scientific impact and were of greater scientific quality compared to NRC Engineering Division Research Centres and for NRC overall

Figure 4.1. Long description follows.
 

Source: Science-Metrix NRC bibliometric study (2011-2015)

Long description of figure 4.1.

This figure is a horizontal bar chart comparing the scientific impact and quality between the National Research Council (NRC) Engineering Division Research Centres and the NRC overall. This figure shows that EME researchers had a greater scientific impact and were of greater scientific quality compared to the NRC Engineering Division Research Centres and for the NRC overall.

Name Scientific impact (ARC) Scientific quality (ARIF)
NRC 1.5 1.3
EME 2 1.6
RC X 0.5 0.6
RC Y 1.5 1.2

This bibliometric study also found that areas within EME with the greatest scientific impact and highest scientific quality were energy and microbiology. This finding supports results from interviews and the two PRCs on research areas of particular excellence.

4.2. EME researchers are recognized internally, as well as nationally and internationally, for their research excellence.

EME researchers are recognized by their EME and NRC colleagues (from Tier 1 programs) as high calibre and leaders at the international level. One program leader, whose program draws on EME researchers, noted that these researchers are strong ambassadors, drawing international interest to their programs. Other program leaders echoed this notion, recognizing the benefit of EME researchers working on their programs and their important role in the prorgams' successes.

Clients have also recognized the expertise of EME employees. In the survey conducted as part of this evaluation, all respondents noted that EME provided the expertise needed to fulfill their project objectives; and 76% said this was to a significant extent.

The excellence of EME researchers were also recognized outside of the NRC, through prestigious awards such as:

  • Royal Society of Chemistry Fellow
  • Electrochemical Society Fellow
  • Queen Elizabeth II's Diamond Jubilee Medal in recognition of internationally renowned scientific work carried out at NRC

Many EME researchers were also invited to speak at international conferences such as:

  • International Workshop on Laser-Ultrasound for Metals
  • Annual Meeting of the American Crystallographic Association
  • Arctic and Marine Oil Spill Program Seminar

5. Stakeholder engagement

5.1. Three quarters of EME clients and collaborators were from industry, and approximately half of its revenues were from government clients and collaborators.

EME stakeholder engagement occurs at the program level, with the exception of a few strategic accounts (e.g., CanmetMining, Canadian Mining Innovation Council), which are coordinated at the research centre level. Stakeholder engagement largely occurs through individual meetings with stakeholders and attendance of staff at conferences and other industry events. These efforts have resulted in projects with many clients and collaborators. A review of administrative data indicates that EME had revenue projects with approximately 183 clients and collaborators between 2012-13 and 2016-17, with the majority being from industry (see figure 5.1). This is consistent with the NRC's focus during this time period.

Figure 5.1: Three quarters of EME's clients and collaborators were from industry (2012-13 to 2016-17)

Figure 5.1. Long descriptions follows.
 

Source: Administrative data

Long description of figure 5.1.

This figure is a horizontal bar chart showing the source of EME clients and collaborators. This figure shows that three quarters of EME's clients and collaborators were from industry.

Client type Number
Industry 137
University/Academia 20
Federal/National Government 15
NGO/NFP 3
Industry Association/Consortium 4
Provincial Government 2
RTO 2

Similarly, when looking at the clients and collaborators at the program level, all programs worked with more clients and collaborators from industry than government or academia. However, with the exception of the HEM program, a larger proportion of revenue was generated from work with OGDs (see figure 5.2). The energy PRC in particular raised concerns over the large proportion of revenue from OGDs, while still acknowledging some of the benefits of working with government departments, such as facilitating the exploitation of program results.

Figure 5.2: Over the evaluation time period, a large proportion of EME's program revenues came from OGDs, with the exception of the HEM program*

Figure 5.2. Long description follows.
 

*HEM, ES, and BE programs' data corresponds to the period 2013-14 to 2016-17. EAM program data corresponds to the period 2015‑16 to 2016-17.

Source: Administrative data

Long description of figure 5.2.

This figure is a horizontal stacked bar chart comparing the source of revenues for each EME program. This figure shows that over the evaluation period, a large proportion of EME's program revenues came from Other Government Departments (OGDs) with the exception of the High Efficiency Mining (HEM) program.

Program Name OGD Industry University/Academia Other
EAM $2,851,073.00 $235,475.00   $173,274.00
BE $3,818,805.00 $455,946.00   $173,274.00
ES $2,831,804.00 $815,502.00 $383,978.00 $582,308.00
HEM $1,600,431.00 $7,446,380.00 $331,329.00 $504,146.00

Going forward, the proportion of program revenue from government departments may decrease somewhat for the ES program and the EAM program in particular. This is because some program activities not directly related to program objectives will be moved out of the program and will become research centre activities (EME FY 2018-19 Operational Plan). For the ES program this includes work with Defence Research and Development Canada (DRDC) related to soldier battery packs, which accounted for $1.33M or 47% of OGD revenues during the program time period. For the EAM program this includes some of the technical services provided to the Department of National Defence for site closures.

5.2. EME will need to continue its efforts to raise awareness of the Research Centre and the NRC within the energy and mining sector as they are not well known.

Prior to EME's establishment in 2012, the NRC had not focused on the mining and energy sectors in an organized manner. EME staff indicated that as a result of the NRC entering two new sectors there was a significant level of effort required to raise awareness of the NRC and its capabilities, which was greater than originally anticipated. Findings from the peer reviews and interviews with external stakeholders suggest that EME has more work to do in order to establish the NRC brand and reputation in the mining and energy sectors. For instance, members from both the energy and mining PRC were largely unaware of the NRC prior to the peer reviews. Interviews with stakeholders from the mining and energy sector suggested confusion about the difference between NRCan and the NRC, and that there is a need for EME to demonstrate its unique role.

5.3. While EME programs engaged stakeholders from along the value chain, there are opportunities to enhance engagement of key stakeholders.

The stakeholder engagement plans for all four of EME's programs highlight the importance of engaging clients from across the value chain in order to achieve program objectives.

Figure 5.3: EME targets stakeholders across the value chain in the energy and mining sectors

Figure 5.3. Long description follows.
 

Source: EME Strategic Plan (2016)

Long description of figure 5.3.

This figure shows the different stakeholders across the value chain. This figure shows that EME targets key stakeholders across the value chain in the energy and mining sectors such as: materials, components, integrators, end users, regulators and enablers.

A review of administrative data and findings from the PRC provide evidence that EME programs have made progress in reaching stakeholders along the energy and mining value chains, however, there are opportunities for additional engagement for three programs in particular (i.e., EAM, ES and BE). Specifically:

  • The EAM program should more fully engage the regulatory community (at the federal, provincial and territorial levels), which has been identified as important given the relationship between the program's focus on sustainable mining and the regulatory environment.
  • The BE program should more fully engage utility companies, municipalities, provincial agencies, and remote communities. The limited engagement of remote communities was of particular concern to the energy PRC given the program's specific focus on bioenergy in these remote locations.
  • The ES program should more fully engage utility companies, and government bodies that plan and regulate provincial electricity systems. These stakeholder groups are important to the program's success given that it is focused on energy storage for utility application and given that energy is provincially planned and regulated. The energy PRC concluded that the program would also benefit from greater engagement of materials and components companies as they provide the underpinnings of energy storage systems.

While not related to improving the reach to stakeholders along the value chain, the mining PRC recommended that EME mining programs consider engagement with the mining sector in the United States. The closure of the US Bureau of Mines, two decades ago, has resulted in a significant research organization vacuum in the US mining sector. As a result, the US is a potentially significant market for the HEM program. The Australians (Mining3) and other research organizations are now starting to explore this market.

EME programs' abilities to engage stakeholders were not without challenges. EME staff interviewed as part of the evaluation were of the opinion that the budgets available for stakeholder engagement were insufficient. It was noted that meaningful engagements were limited because of the remote locations of stakeholders and to build relationships, particularly with mining companies who are reluctant participants in R&D, in-person visits are much more effective. Some EME staff also highlighted the breadth of programs, with a wide range of stakeholders, as contributing to challenges in stakeholder reach. EME's current Operational Plan (2018-19) also highlights gaps in business development needs with only one Client Relationship Leader per program.

5.4. EME engaged a limited number of universities and would benefit from greater collaborations.

EME engaged few academic institutions over the evaluation time period. Administrative data indicates that 10% of EME's clients and collaborators were universities, half of which were foreign institutions who had purchased coin cells from EME. The limited number of collaborations with universities is attributed to the approach of the NRC, at that time of EME's creation, where the primary focus was to provide support to industrial partners. Both the mining and the energy PRCs commented on the importance of engaging universities to leverage expertise, develop networks, keep abreast of developments in the field and to facilitate the movement of low TRL technology (traditionally found within a university) to higher TRL levels of commercial scale. Additional collaborations with Canadian universities would also enable expanded, cost-effective outreach and stakeholder engagement, as well as enhance the generation of Highly Qualified Personnel (HQP).

The NRC Dialogue exercise, which was conducted in 2016-17 and internally assessed the state of the NRC along four lines (innovation support, engagement, governance and management) also revealed opportunities for improved engagement with academic institutions across NRC. Efforts are now underway to increase collaborations with universities (e.g., pursuit of collaboration centres, post-doctoral employment program; NRC Action Plan 2017-2021). In addition to the NRC-wide actions, there is an opportunity for EME and its programs to collaborate with universities to a greater degree.

Recommendations:

3. EME should increase awareness of its research capabilities and its programs within the mining and energy sectors.

4. EME should ensure the BE program engages a greater number of utility companies, municipalities, provincial agencies and remote communities.

5. EME should ensure the ES program engages a greater number of material and component companies, Canadian utility companies and government bodies that plan and regulate provincial electricity systems.

6. EME should ensure the EAM program engages the regulatory community to a greater degree.

7. EME should increase its collaborations with academic institutions.

6. Performance and impact

6.1. EME and its programs met recalibrated revenue targets through over 500 projects.

Over the evaluation time period, EME resources worked on close to 530 projects within EME and its four programs. The breakdown of projects completed by program is presented in figure 6.1 below.

Figure 6.1: EME programs completed many projects over the evaluation time period*

Figure 6.1. Long description follows.
 

*HEM, ES, and BE programs' data corresponds to the period 2013-14 to 2016-17. EAM program data corresponds to the period 2015‑16 to 2016-17. Data presented in the figure does not include projects that were conducted within EME, however, before the programs were formally launched.

Source: Administrative Data.

Long description of figure 6.1.

Figure 6.1 is a horizontal bar chart showing the number of projects completed by each EME program over the evaluation time period.

Program Number of projects completed
HEM 170
BE 79
EAM 69
ES 56

These projects contributed to each program's ability to meet their revised revenue targets. Revenue targets were adjusted as part of the 2015 recalibration initiative and made more realistic in response to the limited resources the programs had access to. When assessed against the recalibrated targets, EME's programs performed well, with the exception of the EAM and the HEM programs in 2016-17.

Table 6.1: With minor exception, EME programs met their recalibrated revenue targets
  2014-15 2015-16 2016-17
BE Revenue $1,107,940
$1,096,594
$1,138,622
Target $1,050,000 $930,000 $850,000
ES Revenue $1,064,644
$978,554
$1,346,710
Target $1,100,000 $900,000 $1,200,000
EAM Revenue NA NA $1,554,214
X - did not meet
Target $1,700,000
HEM Revenue $2,049,912
$2,209,251
$1,924,192
X - did not meet
Target $2,050,000 $2,210,000 $2,210,000

Note: Table only presents years in which the recalibrated targets were in effect. As a result, the table excludes 2012-13 and 2013-14.

Source: Administrative data

EME staff noted external challenges to meeting revenue targets such as economic health and commodity prices. However the greatest challenge continues to be access to the necessary resources for projects. Despite recalibration, programs indicated that meeting targets would become increasingly challenging because resources are not growing as planned, due to resource constraints within EME and within the NRC as a whole.

6.2. EME programs have had some successes related to their value proposition. Narrowing the focus of EME's energy programs would assist in the further realization of their value propositions.

Mining programs

The mining PRC was unable to assess progress against the quantitative targets in the mining programs' value propositions because they felt these were neither measurable nor achievable. As approved by SEC in 2012-2013, the value proposition suggests that EME's two mining programs will have a positive impact on the whole Canadian mining industry. According to the mining PRC, this is not realistic nor measurable. To this effect, the mining PRC recommended that the value propositions be revised to focus on impacts for clients and collaborators rather than for the Canadian mining sector.

In spite of the fact that they could not assess progress against quantitative targets, the mining PRC concluded that sufficient progress toward achievement of the value proposition had been made by the EAM and the HEM programs, as measured by the achievement of master projects' milestones. For the HEM program, achievements in the area of laser-induced breakdown spectroscopy (LIBS) and ultrasonics was viewed as supporting the value proposition of increased recovery of ore and improved equipment durability. Further evidence of the progress made by the HEM program is the seven patent applications it filed for LIBS related technologies, and another two for rock-bolt sensors and nano-structured ceramic coatings on a substrate. These are all closely aligned with the program's value proposition. For the EAM program, accomplishments in the areas of bio-remediation, bio-mining, sensing and monitoring were viewed as supporting the program's value proposition of decreased capital and operational expenditures, and environmental liability costs.

Energy programs

The energy PRC highlighted successes related to each of the programs' value propositions. For example, the ES program's work in the area of battery electrode materials was exemplary, and contributes to its value proposition of improving the cost and durability of energy storage technologies. The BE program had some projects with noteworthy achievements in the area of small and portable waste conversion systems and pellet binders, which are related to its value proposition of reducing the production cost of biofuel and biopower.

A review of program documents (e.g., Quarterly Reports, material provided to the PRC) indicates that the energy programs made progress toward achieving the milestones identified for their master projects. Despite this, the progress made by the energy programs may not be sufficient to enable the achievement of their value propositions by the end of the program. The energy PRC was of the opinion that the programs did not make sufficient progress toward their value propositions because the master projects were not sufficiently aligned with it, and therefore diluted the programs' efforts and achievements. These projects did, however, respond to industry needs, which is consistent with the NRC's strategic direction from 2012-13 to 2016-17 where the emphasis was on supporting industry. The work the PRC felt did not directly contribute to the programs' value propositions included:

  • ES and BE programs' techno-economic analysis,
  • ES program's work to demonstrate and validate component and system integration, and
  • BE program's work to address technology gaps within conventional power plant systems to facilitate biofuel use.

When the limited resources are considered, the energy programs would benefit from a more focused set of activities directed towards one targeted objective. Some EME staff also expressed concerns over the breadth of EME's programs, and of spreading themselves too thin to achieve objectives. While there are different ways of narrowing the focus of the programs, the PRC highlighted several options to consider. This included focusing on fewer, more significant projects that are all directly aligned with the value proposition and narrowing the programs to existing areas of expertise. For the ES program, this is the program's existing expertise related to battery materials and electrochemistry. For the BE program, it is focusing on technologies at the TRL four to seven level in the areas of densification of biomass, utilization of digestate from anaerobic digestion and hydrothermal liquefaction.

6.3. EME had positive impacts for its industrial clients and collaborators.

Despite the limited progress made by some of EME's programs toward their formal value propositions, all programs conducted many projects with clients and collaborators from which there were positive benefits. A survey of EME's industrial clients and collaborators provides evidence of these benefits (see figure 6.3).

Figure 6.3: Industrial clients and collaborators reported positive impacts attributable to their work with EME

Figure 6.3. Long description follows.
 

Source: Survey of EME industry clients and collaborators, n = 25

Long description of figure 6.3.

Figure 6.3 is a horizontal bar chart showing the proportion of clients who reported positive impacts attributable to their work with EME.

Impact Category Impact Percentage of clients
Developed or Commercialized Technologies New or improved technologies 48%
Commercialized new products 40%
Growth Increased funding or investments 28%
Increased annual revenue 24%
Increased employees 20%
Productivity Increased productivity 20%
Decreased Costs Decreased operational costs 16%
Decreased capital costs 8%

More survey respondents noted impacts around developed or commercialized technologies as a result of work with EME compared to other impacts. This is expected as these are direct and immediate outcomes of the work. Growth, productivity and decreased costs are often indirect impacts of EME's work, as well as outcomes that occur in the intermediate or longer term. This is important considering that a large proportion of EME clients have been working with the Research Centre and its programs for five years at most (following EME's creation).

The energy and mining PRCs also concluded that EME programs had a positive impact on clients and collaborators and highlighted examples of this in their peer review reports. For example:

  • The BE program worked on the development of binders for pellets to help reduce the cost of the solid fuel for a client.
  • The ES program had benefits for its clients and collaborators in the area of safety testing of cells and battery packs, and material development, particularly for lithium materials.
  • The HEM program worked with clients and collaborators to develop new technologies, such as a portable LIBS system to evaluate minerals onsite and ultrasonic sensors for real-time monitoring of rock bolts.
  • The HEM program also developed partnerships with a rock bolt manufacturer and a vendor of software solutions for rock mechanics monitoring (i.e., DSI Underground and ESG Solutions), which has significant potential for successful commercialization of rock bolt sensors.
  • The EAM program had collaborative projects with several stakeholders to improve existing technologies (e.g., Teck, Canadian Malartic, Agnico Eagle, Tata Steel) and shows great promise for continuing to do so on a significant scale.

6.4. Insufficient time has lapsed to clearly see the impacts of EME, and its programs, on the value chains in the natural resource and utility sectors.

EME seeks to strengthen its value chain by improving the technological ecosystem as well as the connections between parts of the chain. Findings from interviews with EME staff as well as the conclusions drawn by the mining and energy PRCs indicate that insufficient time has elapsed to clearly see the impact of the programs on the value chains in the natural resource and utility sectors. EME programs' engagement of stakeholders from along the value chain will assist in its ability to strengthen the value chains in the natural resource sector. To this effect, it is important for EME to continue enhancing its engagement of key stakeholder groups from along the value chain, as discussed earlier. EME staff reflected that the ability to positively influence the value chain is challenging considering the size of the industry and the make-up of stakeholders (i.e., small-and-medium-sized enterprises), particularly in the energy storage and bioenergy industries.

6.5. One of EME's energy programs contributed to initiatives that will impact policy and regulations while the other programs are conducting research that has the potential to do so in the future.

Findings from the evaluation indicate that the ES program's work with Transport Canada to examine the safe transportation of lithium-ion batteries is currently informing regulations around the shipment of this type of batteries. Both the energy and the mining peer review committees felt that EME's other three programs, despite not having a direct impact yet on policy and regulations, have the potential to do so in the future. Examples include:

  • The BE program's proposed projects on the safety of bio-fueled energy systems operations have the potential to inform safety related policy and regulations.
  • The EAM program's work on tailings, acid mine drainage, and selenium contamination and treatment can influence sustainable mining policy and regulations.
  • The HEM program's work could inform policy and regulation related to underground health and safety (e.g. with respect to monitoring requirements and/or re-entry protocols).

Recommendations:

8. EME should narrow the scope of its energy programs and accordingly make any necessary adjustments to its activities and value propositions.

9. EME should modify the value proposition of its two mining programs to ensure:

  1. clear alignment with EME's focus on clean resources and clean technology, and
  2. they are attainable and measurable.

7. Conclusion

EME seeks to have a positive impact on the energy and mining sectors, and has made some progress in this. EME mining programs have notable successes in the areas of LIBS and ultrasonics, bio-remediation, wear and corrosion, bio-mining, and environmental sensing and monitoring. The energy programs had achievements in battery electrode materials, small and portable waste conversion systems and pellet binders. EME's work in these areas is aligned with stakeholders needs in the mining and energy sector. EME's many projects have resulted in positive impacts for its clients and collaborators, including the development or commercialization of new technologies or products, growth, productivity and/or decreased costs.

Despite these successes, there is an opportunity for EME to enhance its effectiveness. EME can further improve its relevance by seeking opportunities to better understand stakeholder needs and the R&D ecosystem within which its programs operate. The objectives of EME mining programs need to be revisited to ensure these programs are contributing to EME's vision, and to government priorities, related to clean resources and clean technology.

In order to facilitate the achievement of its programs' objectives, EME should narrow the focus of its energy programs. Existing capabilities and critical mass will need to be considered as part of this exercise. Expanding engagement of key stakeholders (i.e., utilities, municipalities, remote communities, and relevant regulatory bodies) will also be important for EME programs to achieve their objectives and to raise awareness of EME in the mining and energy sectors. Enhancing engagement of these stakeholders will facilitate EME's ability to strengthen the value chains in the natural resource and utility sectors in the future. Increasing collaborations with universities is one way that can enable expanded, cost-effective outreach and stakeholder engagement, as well as the ability to leverage existing expertise.

EME has fully embraced the NRC's matrix management model, and has capabilities that generally meet the needs of EME and non-EME programs. Addressing the gaps in expertise and facilities to support EME's programs will be important for their achievement of objectives. In order to continue supporting the facility needs of EME and non-EME programs, a consistent method for tracking facility use is needed.

Management has accepted the evaluation recommendations. Their response and planned actions are found in the next section.

8. Management response

Recommendation Response and planned action(s) Timelines Proposed person(s) responsible Measure(s) of achievement
1. EME should continue to increase its understanding of the mining and energy sector, stakeholder needs and the R&D ecosystems within which its programs operate. Recommendation accepted.

EME, working with its four programs, will identify ways in which it can increase its understanding of stakeholder needs and the R&D ecosystem within which the programs work. This will include updating and executing each of the four programs 'Stakeholder Engagement Plans with a planned approach.

June 2019 EME Director General, in consultation with the Program Leads (EAM, HEM, ES, BE) Program Stakeholder Engagement Plans are updated and executed.
2. EME should consistently track the use of EME facilities and equipment. Recommendation accepted.

EME will design and implement a consistent method of facility/equipment tracking.

May 2019 EME Director of Operations Tracking method for facility/equipment use designed and implemented.
3. EME should increase awareness of its research capabilities and its programs within the mining and energy sector. Recommendation accepted.

EME will develop and execute a communication/ engagement plan to communicate its capabilities and programs in the mining and energy sectors.

The plan will identify different ways EME will pursue to increase its awareness, as well as indicators of success.

June 2019 EME Director General Communication/engagement plan developed and implemented.
4. EME should ensure the BE program engages a greater number of utility companies, municipalities, provincial agencies and remote communities. Recommendation accepted.

EME will work with the BE program to increase its engagement with the identified stakeholder groups. This will include updating the program's Stakeholder Engagement Plan to include strategies to increase engagement with the identified stakeholders.

September 2019 EME Director General, in consultation with Program Lead (BE) Program Stakeholder Engagement Plan is updated and implemented.
5. EME should ensure the ES program engages a greater number of material and component companies, Canadian utility companies and government bodies that plan and regulate provincial electricity systems. Recommendation accepted.

EME will work with the ES program to increase its engagement with the identified stakeholder groups. This will include updating the program's Stakeholder Engagement Plan to include strategies to increase engagement with the identified stakeholders.

September 2019 EME Director General, in consultation with Program Lead (ES) Program Stakeholder Engagement Plan is updated and implemented
6. EME should ensure the EAM program engages the regulatory community to a greater degree. Recommendation accepted.

EME will work with the EAM program to increase its engagement with the identified stakeholder groups. This will include updating the program's Stakeholder Engagement Plan to include strategies to increase engagement with the identified stakeholders.

September 2019 EME Director General, in consultation with Program Lead (EAM) Program Stakeholder Engagement Plan is updated and implemented.
7. EME should increase its collaborations with academic institutions. In order to increase engagement with universities, EME will prepare and implement a plan to increase its collaborations with academic institutions. As part of this, a proposal will be developed for a collaborative research centre between EME and an academic institution. September 2019 EME Director General Increased collaborative research projects with universities.

Collaborative research centre proposal submitted.

8. EME should narrow the scope of its energy programs and accordingly make any necessary adjustments to its activities and value propositions. Recommendation accepted.

EME will review the current scope of the two energy programs during FY2019, as outlined in the FY2019 operational plan. Consideration will be given to the capabilities and human resources that the programs have access to.

June 2019 EME Director General, in consultation with Program Leads (ES and BE) The two energy program plans are revised with a more focused scope.
9. EME should modify the value proposition of its two mining programs to ensure:
  1. clear alignment with EME's focus on clean resources and clean technology, and
  2. they are attainable and measurable
Recommendation accepted.

The value propositions of EME's two mining programs will be revised as part of the program renewal process, to be completed through FY2019.

June 2019 EME Director General, EME Program Leads (EAM and HEM) The two mining programs' value propositions are revised.
 

Appendix A - EME profile

The EME Research Centre was created in April 2012 and seeks to have a positive impact on the energy, mining and utility sectors in terms of increased industrial productivity and competitiveness and reduced environmental risk (see Appendix B for EME's results chain/logic model)Footnote28. There are currently four programs within the EME Research Centre. These are summarized in table A1 below.

Table A1: Summary of the current EME programs
Bioenergy systems for viable stationary applications, 2013-14 to 2022-23
Focus Integrating bioenergy into target markets within Canada's energy supply mix (for remote communities/industry and from municipal solid waste) by producing economic biofuels and adapting conventional industrial power and heating equipment (gas turbines, diesel generators, and boilers) to be biofuel flexible.
Value proposition To make bioenergy technology platforms economically viable in Canadian energy markets by strengthening the Canadian value chain and reducing the production cost of biofuel to $13/GJ (solid) and $10.5/GJ (gas) and biopower to 13¢/kWh (solid) and 8¢/kWh (gas) by 2023.
Energy storage for grid security and modernization, 2013-14 to 2021-22
Focus Reducing the risks of energy storage adoption by utilities and other end users, and strengthening the Canadian energy storage value chain by developing energy storage materials, components, systems and services.
Value proposition To demonstrate at Technology Readiness Level (TRL) seven, an installed cost reduction from the current energy rating of ~$1000/kWh to under $500/kWh and from the current power rating ~$2500/kW to less than $1250/kW, while increasing the operating lifetime to >15 years from today's five to seven years and strengthening the Canadian Energy Storage supply chain.
High efficiency mining, 2013-14 to 2022-23
Focus Reducing costs and increasing the efficiency of mining and mineral recovery and processing, including low grade ore and through reduced wear and corrosion of rock handling equipment.
Value proposition Improving, developing and commercializing practical technologies - diagnostics, separation and materials, and demonstrating the value of process integration specifically applied to gold, nickel, copper and oil sands that: increase economic recovery from Canadian lower grade ores by 10% and improve equipment cost-of-ownership by 6%. Thereby, by 2023, enabling increased savings and cost reductions estimated to be at least $225M and an increase in economically recoverable ore valued at more than $1.8B.
Environmental advances in mining, 2015-16 to 2021-22
Focus Reducing environmental costs and liabilities using a multidisciplinary approach to improve and accelerate remediation, and de-risk technology development and deployment through testing and optimization, to enable greater market capture by the Canadian mining supply chain.
Value proposition Working with the mining supply chain to promote innovative solutions that will: decrease capital and operating expenditures as well as liability costs for Canadian mine operators and other government departments (OGDs) by 10%, and developing and optimizing technologies that will lead to increased international market capture by Canadian Mining Equipment, Technology and Service providers by 5%.

Financials

During the evaluation period, EME's expenditures totaled $124 million.

Table A2: EME expenditures in $M (2012-13 to 2016-17)
Type of expense Fiscal year Total
2012-13 2013-14 2014-15 2015-16 2016-17
Direct operations* 6.3 7.4 7.6 7.2 9.6 38.1
Indirect operations** 21.5 16.6 17.0 14.8 14.9 84.8
Total 27.8 24.0 24.6 22.0 24.5 122.9

* Direct expenditures are costs which are traceable to a project deliverable.
**Indirect expenditures are costs that are not physically traced to a project deliverable.
Source: Administrative data

Clients and collaborators

EME aims to engage stakeholders across the entire supply and value chain in the mining and energy sectors. During the evaluation period, EME worked with 183 unique clients and collaborators. Clients used the NRC's capabilities on a fee-for-service basis (285 projects) whereas collaborators engaged in strategic R&D (237 projects). The majority of EME clients and collaborators were from industry (75%), however these only generated 38% of its revenue. Projects with OGDs generated over half of EME's revenue during the evaluation period.

In addition to working with EME's external clients and collaborators, EME staff supported non-EME programs at the NRC. On average each year, approximately 40% of EME resources (in terms of full time equivalents) worked on non-EME programs.

Table A3: EME external clients and collaborators, projects and revenue (2012-13 to 2016-17)
Description OGD Industry University Other* Total
Number of clients 17 137 20 9 183
Number of projects 119 249 11 151 530
Revenue $17.6 M $12.4 M $1.8 M $1.2 M $33.00 M
% of revenue 53% 38% 5% 4% 100%

*Other includes not-for-profits, foreign governments, industry associations/consortiums, research organizations and all projects where no client is specified.
Source: Administrative data

Human resources

Within the NRC's research centre/program structure, a matrix management approach is used; the research centres are responsible for managing the resources (i.e., facilities/equipment and competencies/expertise), and programs draw on these resources to fulfill their objectives. Programs can access resources from within their own research centres or from other research centres.

Between 2012-13 and 2016-17, EME had an average of 177 staff per year. The majority of EME's human resources were scientific and technical staff, who were supported by Research Centre administration and management. Overall, the number of scientific and technical staff increased during the evaluation period, with the administrative and management staff remaining relatively constant (see table A4).

Table A4: EME human resources (2012-13 to 2016-17)
Type of staff 2012-13 2013-14 2014-15 2015-16 2016-17 Average
Administrative and management 15 13 11 12 15 15
Scientific and technical staff 152 158 161 168 179 164
Total 167 171 172 180 194 177

Source: Administrative data

Facilities

EME maintains and operates R&D facilities and equipment in three locations: Vancouver, Ottawa and Greater Montréal. The main facilities used by each program are presented below.

Energy Storage for Grid Security and Modernization program

Storage evaluation facilities

  • Energy storage component cyclers
  • Thermal chambers
  • Fire and containment facilities

Storage materials fabrication and characterization facilities

  • Dry room
  • Electrode fabrication facilities
  • Materials synthesis labs
  • A wide range of characterization equipment (e.g. XRD, TEM, SEM, XPS, TGA, XRF, NMR)

Systems facilities

  • Energy storage systems lab
  • Testing facilities
  • Cluster computing resources

Source: EME website

Bioenergy Systems for Viable Stationary Applications program

  • Bioengineering and fuel cell laboratories
  • Biorefining pilot plants
  • Laboratory and pilot scale autoclaves
  • Combustion, engine and gas turbine research laboratories and test cells
  • Advanced sensors and diagnostic equipment
  • Modeling hardware and software
  • Material and gas characterization and analysis

Source: EME website

High Efficiency Mining program

  • Laser induced Breakdown Spectroscopy (LIBS) and laser ultrasonic labs
  • Powerful materials characterization and microscopy centres, including Scanning Electron Microscope (SEM), Transmission Electron Microscopy (TEM), and X ray diffraction (XRD)
  • Separation facilities
  • Specialized modeling hardware and software
  • Wear, corrosion and environmental testers and simulators

Source: EME website

Environmental Advances in Mining program

  • LIBS system- General facility
  • X-ray methods facility
  • TEM facility
  • SEM facility
  • Materials Analysis facility
  • Hydrodynamic facility
  • Separation equipment facility
  • Bioengineering facility
  • Biomonitoring facility
  • Remediation facility
  • Medium scale facility
  • Electrochemical testing facility
  • Energy Beam Analysis facility

Source: Correspondence with the EAM program

 

Appendix B - EME logic model

Long description of image follows.
 
Long description of EME logic model.

This logic model present the inputs, activities, outputs and outcomes of the EME portfolio.

EME Portfolio & Programs

  1. 1. Inputs
    • 1.1 NRC investment
    • 1.2 Revenues
    • 1.3 NRC common services support and Shared Services Canada
    • 1.4 People: Highly Qualified Personnel (HQP), specialized expertise and experience
    • 1.5 Facilities: Specialized facilities and equipment
    • 1.6 Contributions from other NRC portfolios & collaborators/partners
  2. 2. Activities
    • Facilitate scientific, technical organizational capability
      • 2.1 Strategic planning (intelligence and foresight)
      • 2.2 Business development
      • 2.3 Human resource management (expertise/workforce development)
      • 2.4 Facility management
      • 2.5 Program and project management
    • Build scientific and technical capability
      • 2.6 Strategic R&D (internal R&D projects and collaborative R&D projects)
      • 2.7 Major/minor capital projects
    • Deploy scientific and technical capability
      • 2.8 R&D services (collaborative R&D projects, technical services)
      • 2.9 Multi-partner R&D projects & consortia (involving full supply chain)
  3. 3. Outputs
    • Technical capability
      • 3.1 Enhanced/maintained infrastructure
      • 3.2 Enhanced/maintained competencies.
    • Collaboration
      • 3.3 Clients/collaborators reached
      • 3.4 Value-added services for clients/collaborators
    • Scientific/technical achievements
      • 3.5 Scientific and technical knowledge
      • 3.6 Demonstrated/validated product, technology, or solution
      • 3.7 Intellectual property

Outcomes

  1. 4. Immediate (1-3 years)
    • Business innovations
      • 4.1 Market acceptance of advanced technology solution for use in Canada's natural resource and utility sectors
      • 4.2 Development and adoption of new technologies de-risked
  2. 5. Intermediate (4-6 years)
    • Advancement of knowledge
      • 5.1 Portfolio researchers and facilities recognized as world class
    • Government Policy Solutions
      • 5.2 Informal public policy and regulations
    • Business innovations
      • 5.3 Clients/collaborators commercialize new technologies
      • 5.4 Clients/collaborators improve productivity
      • 5.5 Clients/collaborators experience growth
      • 5.6 Strengthened Canadian supply chains in natural resource & utility sectors
  3. 6. Long term (7+ years)
    • Environmental impacts
      • 6.1 Reduced environmental risks in natural resource and utility sectors
    • Economic impacts
      • 6.2 Enhanced industrial productivity and competitiveness in natural resource and utility sectors
 

Appendix C - Methodology

The evaluation period covered 2012-13 to 2016-17 inclusively. The evaluation was carried out in accordance with the NRC's approved evaluation plan and Treasury Board policies. The Energy, Mining and Environment (EME) Research Centre and its programs had not been previously evaluated. The evaluation questions were developed based on consultations undertaken during the planning phase of the evaluation and a review of key documents. The evaluation questions are listed in table C1.

Table C1: Evaluation questions

  • Relevance and need
    • 1. To what extent is there a demonstrable need for the R&D offered by EME programs?
    • 2. Are EME program's current value propositions and market assessments still valid and aligned with program activities?
    • 3. To what extent are EME capabilities needed by NRC programs?
  • Stakeholder engagement
    • 4. Do EME programs have appropriate stakeholder engagement plans?
    • 5. To what extent have EME programs engaged key clients and collaborators?
  • Performance
    • 6. What progress has been made by EME programs to achieve the objectives articulated in their business case value propositions?
    • 7. To what extent are the achievements of EME's programs appropriate given the level of resources?
    • 8. To what extent is the research produced by EME leading-edge?
    • 9. To what extent have EME programs contributed to the ability of clients and collaborators to: Develop or adopt new technologies; Commercialize new technologies; Grow; Improve productivity; Reduce costs?
    • 10. To what extent have EME programs contributed to strengthened value chains in the natural resource and utility sectors?
    • 11. To what extent have EME programs contributed to government policy and regulations?
    • 12. Do EME programs have access to the capabilities needed to meet client needs and achieve intended outcomes?
    • 13. To what extent is EME project ideation and project management effective?
    • 14. What changes, if any, should be made to the scope of EME?
 

The evaluation approach and selection of methods were based on the information needs of the NRC Senior Management to support timely decision making. In order to maximize the possibility of generating useful, valid and relevant evaluation findings, mixed methods were used. This allowed for triangulation (i.e., convergence of results across lines of evidence) and complementarity (i.e., developing better understanding by exploring different facets of a complex issue).

Both qualitative and quantitative methods were used, and included:

  • Document and literature review
  • Market assessment
  • Data review
  • Interviews (internal and external)
  • Client survey
  • Peer review

A discussion of the approach used for each of these methods, as well as any limitations and/or challenges, is provided in the following paragraphs.

Document and literature review

Internal and external documents were reviewed, synthesized and integrated into the evaluation to provide context and history, and to complement other lines of evidence in assessing relevance and performance. Internal documents reviewed included Research Centre and program strategic plans, operating plans and business plans. In addition, external documentation and literature were reviewed, including documents produced by government departments and central agencies, as well as literature on the programs' targeted industries and markets.

Market assessment

The evaluation team updated the prospective economic impact assessments prepared by each program during their development with more recent quantitative data. This was done to determine whether the programs' forecasted economic impacts were still valid. The evaluation team used the same prospective economic model (i.e., same assumptions), however, updated the data with actual market data, where it was available.

Data review

Research Centre and program administrative and performance data for 2012-13 to 2016-17 were reviewed to provide information on program inputs (i.e., resources), outputs, and client reach. Valid and reliable data on facility use was not available.

Interviews (internal and external)

Interviews were conducted with internal EME staff as well as external clients and stakeholders (i.e., public and not-for-profit clients and industry associations) to collect information such as personal experiences, opinions and expert knowledge related to the relevance and performance of EME and its programs. This information was used to complement other lines of evidence and to contextualize quantitative information. In total, 36 internal NRC staff and ten external clients and stakeholders (e.g., industry associations, other government departments) were interviewed as part of the EME evaluation. While there was a limited number of interviews conducted with external stakeholders (due to challenges reaching them), this was mitigated through the use of peer review committees, which brought an additional external perspective from international experts.

Client survey

A web-based survey of EME's private sector clients was conducted to assess the impacts that EME and its programs have had on industry clients and collaborators. Other types of clients (e.g., government, industry associations) were consulted through interviews. The survey was designed by the NRC Office of Audit and Evaluation (OAE) and reviewed by the EME Business Management Support staff to ensure the language used was appropriate for industrial clients. Once finalized, the survey was administered by the NRC Communication Branch, using the NRC's online survey software (Fluid Surveys). The tool was internally tested before being launched. The survey population included industry clients and collaborators from 2012-13 to 2016-17, for which EME could provide contact information. This resulted in a final list of 78 industry clients and collaborators, representing approximately 60% of EME's industry clients and collaborators. An email notification about the survey was first sent to the targeted clients and collaborators from the Program Lead with whom the client had worked with. This was followed by an email with the survey hyperlink, sent by the NRC Communication Branch. The survey was open for two weeks, during which two reminder emails were sent. As a result of low response rates one week after the initial invite, the NRC OAE began contacting participants to complete the survey over the phone. In total, 25 EME industry clients and collaborators completed the survey, which represented a 35% response rate.

Peer review

Two peer review committees (PRC) were convened to assess EME's programs along four dimensions: relevance, stakeholder engagement, performance and appropriateness of resources, including capabilities. One PRC focused on EME's two energy programs - Bioenergy Systems for Viable Stationary Applications and Energy Storage for Grid Security and Modernization while the other PRC focused on EME's two mining programs – High Efficiency Mining and Environmental Advances in Mining.

Members

The energy PRC was composed of three members plus one chair. The Committee included individuals with expertise in energy storage and bioenergy and included national and international representatives from academia and R&D organizations. While one industry member was recruited and expected to participate on the Committee, this individual was unable to contribute due to unforeseen circumstances.

The mining PRC was composed of five members plus one chair. The Committee included individuals with expertise in process measurement and control, equipment durability and optimization, separation technology, prediction and prevention of acid rock drainage, effluent treatment, sensing and monitoring, environmental friendly mining, processing and separation, and remediation and reclamation technologies. The Committee included national representatives from academia, R&D organizations and industry. One of the industry representatives was only able to participate in the pre-assessment and not the site visit due to unforeseen circumstances. This representative approved the final peer review report as well.

Members of both PRCs were expected to participate in the review process in an objective,unbiased and credible manner, with no apparent or perceived conflict of interest. To this end, all members signed a confidentiality and conflict of interest agreement. The list of the energy and the mining PRC members is presented in Appendix D.

Tasks

Each peer review process included:

  1. Reviewing background material produced by the program and by the NRC evaluation team
  2. Participating in a pre-site visit teleconference to discuss the Committee's initial assessment of the programs, information gaps and questions.
  3. Participating in a two and a half day site visit to the NRC.

The total level of effort required by committee members, including the site visit was approximately four days. A representative from the NRC OAE acted as the Peer Review Coordinator.

Limitations

The use of one PRC to assess two programs resulted in efficiencies in terms of the number of committees convened and the number of site visits held. However, it increased the workload for the members of the Committee (i.e., double the material to review, double the reports to write). In addition, a two and a half day site visit to assess two programs reduced the amount of time available for Committee deliberations (as the time was needed to hear from the programs and tour facilities). Efforts were made to mitigate this challenge by providing the Committee with all of the materials in advance of the site visit, and by holding a pre-site visit teleconference to discuss initial assessments and identify areas on which to focus during the site visit. The use of two separate PRCs also did not allow for a peer review assessment at the research centre level. While this was to be mitigated by bringing together the EME Advisory Board with the two PRC Chairs, due to timing this meeting was not feasible within the scope of the evaluation.

 

Appendix D - Peer review committees

Energy Peer Review Committee members

Chair

Amit Kumar
Professor, University of Alberta (Department of Mechanical Engineering)

NSERC/Cenovus/Alberta Innovates Associate Industrial Research Chair in Energy and Environmental Systems Engineering

Cenovus Energy Endowed Chair in Environmental Engineering

Deputy Director – Future Energy Systems

Dr. Amit Kumar is a Professor in the Department of Mechanical Engineering at the University of Alberta. He currently leads a large research program in the areas of energy and environmental systems engineering and life cycle and techno-economic assessments of energy systems in collaboration with industry and government under Industrial Chair Program. The focus of his research program is on the assessment of both conventional and non-conventional sources of energy in terms of environmental and economic footprints. The overall goal of the research program is to develop information for policy formulation and investment decisions. Over last 20 years he has worked on a number of projects in the area of energy systems in collaboration with industry and governments. He has worked on joint projects with large and small scale industries including oil and gas companies, forest companies, renewable energy companies, utilities, consulting companies in the areas of techno-economic and life cycle assessments. He has also worked on energy projects for municipalities and their consortiums. He has served on several international and national expert review panels and steering committees including the European Commission (FP7 and Horizon 2020 programs), National Science Foundation (NSF), USA, Natural Resources Canada (NRCan), California Environmental Protection Agency (CalEPA), and Natural Sciences and Engineering Research Council of Canada (NSERC). He has co-authored more than 180 peer-reviewed journal publications and technical reports on energy systems. He has also been a part of more than 330 conference and other presentations. He is the Associate Editor of Canadian Biosystems Engineering. He holds a PhD in Mechanical Engineering from the University of Alberta, Canada and a B.Sc. in Energy Engineering from the Indian Institute of Technology, Kharagpur, India.

Members

Geza Joos
Professor, McGill University (Electrical and Computer Engineering)

NSERC/Hydro-Québec Industrial Research Chair on the Integration of Renewable Energies and Distributed Generation into the Electric Distribution Grid

Canada Research Chair in Powering Information Technologies, Tier 1

Geza earned his PhD from McGill University. He is a Professor in the Department of Electrical and Computer Engineering Department at McGill University since 2001. His research and consulting interests are in the areas of distributed generation and renewable energy, microgrids and applications of power electronic converters in power systems. He is active in working groups of the International Council on Large Electric Systems (CIGRE) Study Committee C6 on Distribution Systems and Dispersed Generation, and working groups of the Institute of Electrical and Electronics Engineers (IEEE) Power and Energy Society on power electronic applications to power systems. He is also involved in IEEE Standards Association projects on microgrids and static synchronous compensators (STATCOMs). He is a Fellow of the IEEE and the Canadian Academy of Engineering. Recently, he worked with Hydro-Québec and utilities on microgrid deployment, including battery energy storage systems. He also worked on the development of microgrid controller functional specifications (in the context of the industry supported development of IEEE Standard 2030.7- Standard for the Specification of Microgrid Controllers).

Xiaomei Li
Senior Advisor, Alberta Innovates, Clean Energy Division

Xiaomei earned her PhD from Oregon State University and has over 25 years' experience of developing and commercializing technologies that enable the sustainable development of agricultural/livestock industry and rural communities. She is a senior advisor with the Clean Energy Division within Alberta Innovates, leading a bioenergy portfolio. Previously she worked as a senior research scientist/program leader in waste management and bioenergy programs with InnoTech Alberta (formally Alberta Research Council) over 16 years and a chief science officer for Highmark Renewables for two years. She has over 180 publications.

Babu Chalamala
Department Head, Sandia National Laboratories (Energy Storage Technology and Systems)

Babu earned his PhD from the University of North Texas. He is the Department Head for Energy Storage Technology and Systems and Laboratory Program Manager for the Department of Energy Grid Energy Storage program at Sandia National Laboratories. Prior to joining Sandia in August 2015, he was a Corporate Fellow at MEMC/SunEdison for five years, where he led R&D and product development in grid scale energy storage. Before that, he founded two startup companies commercializing large format lithium batteries and digital x-ray sources. Earlier, as a research staff member at Motorola, Research Triangle Institute, and Texas Instruments, he made contribution to the development of electronic materials and device technologies. He is a Fellow of the Institute of Electrical and Electronics Engineers (IEEE) and the Academy of Sciences St. Louis, a Life Member of the Electrochemical Society, and a Member of the Materials Research Society. He currently serves as Secretary of the IEEE Power and Energy Society Energy Storage and Stationary Committee and as a member of the IEEE Fellow Committee.

Charles Mazzacato [regrets]
President, Emerging Power Technologies Inc.

Charles has over 30 years of senior executive experience focused on business development in the electrical equipment and power generation industries. He holds expertise in power systems for engineering including critical power systems for data centres, renewable energy generation and the operation and maintenance of these systems.

Mining Peer Review Committee members

Chair

Laeeque K. Daneshmend
Professor, Queen's University (The Robert M. Buchan Department of Mining; Cross-appointed to the Department of Mechanical & Materials Engineering)

Noranda-Falconbridge Chair in Mine Mechanical Engineering

Laeeque earned his Ph.D. in mechanical engineering from the Imperial College of Science and Technology in London in 1985. His current research interests include mine maintenance, production monitoring information from equipment, reliability based design/redesign of mobile equipment and reliability modeling for development in condition based maintenance. He has extensive experience in mine maintenance, mechanization and automation. He has worked with many mining equipment manufacturers and North American mines.

Members

Ferri Hassani
Professor, McGill University

Webster Chair in Mining Engineering

Ferri earned his Ph.D. in Mining Engineering from Nottingham University. His teaching and research relate principally to mine design, rock mechanics, renewable energy from mines, geosensing and mine backfill. He has been an advisor to multiple governments on mining issues and acted as a consultant to multiple major mining companies around the world. In addition, he was the co-founder of and chairman of the Canadian Mining Innovation Council (CMIC). He was awarded the most prestigious award of the Canadian Institute of mining and metallurgy "The CIM Distinguished Service Medal" in 2017.

Ian Wilson
Manager, Environmental Remediation, Saskatchewan Research Council

Ian Wilson earned his B.Sc. in Environmental Science from Royal Roads University and a MBA from London School of Business and Finance. He currently leads an inter-disciplinary team of scientists, engineers and project managers tasked with the remediation of 37 former cold war legacy uranium mine and mill sites. He has more than 17 years of environmental remediation experience and has successfully managed more than 200 assessment, remediation and site decommissioning projects around the world. His areas of expertise include remediation design, remediation project management, cost estimation, structural demolition, mine closure and waste management.

Peter Radziszewski
Vice President Research, Product RTD, Mining and Aggregated Consumables, Metso

Peter earned his Ph.D. from Université Laval in Mechanical Engineering in 1992. Prior to joining Metso, he was an Associate Professor in Mechanical Engineering at McGill University, a visiting professor at Canadian Space Agency, and a Professor at Université du Québec en Abitibi-Témiscamingue and a visiting scholar at the Julius Kruttschnitt Mineral Research Centre. He has expertise in materials, modeling and simulation, design engineering, numerical modeling, optimization, applied and computational mathematics and engineering thermodynamics. He currently oversees all issues related to consumables product development, wear performance simulation tools, wear simulation laboratory tests and grinding media tests.

Bernadette Conant
CEO, Canadian Water Network

Bernadette earned her M.Sc. in Hydrogeology from the University of Waterloo in 1991. Before joining the Canadian Water Network she was involved in groundwater issues for 17 years, including experience in research jobs in the public and private sector, and management of University-based research programs funded by government and industry partners. She is the author of a number of publications, including referenced journal articles, a book chapter, a US EPA report, and conference abstracts and proceedings.

Josée Méthot [participated in pre-assessment and approval of final report; was unable to attend the site visit]
President and CEO, Quebec Mining Association

Josée holds a M.Sc. degree in Business Administration from the École des Hautes Études Commerciales de Montréal as well as a B.Eng. in Chemical Engineering from McGill University. She has held senior executive positions for almost 20 years as Executive Director, President and CEO of non-profit and municipal organizations with the Montreal Center of Excellence for Site Remediation, Réseau Environnement and of the Intermunicipal Board of Waste Management.