Alternative format
PDF version (244 KB)
This report was prepared by Office of Audit and Evaluation, National Research Council Canada
This report was approved by NRC's President on October 31, 2016
Summary report
This report presents the results of the 2016 evaluation of National Research Council's (NRC) research, development and commercialization activities funded under the Roadmap for Canada's Official Languages 2013-2018 (hereafter referred to as the "Roadmap"). NRC's initiative under the Roadmap is one of 28 initiatives being implemented by 14 federal partners. As with the two previous five-year action plans, NRC received $10M over five years to contribute to the Roadmap. This funding is used by the Information and Communication Technology (ICT) Portfolio for R&D into natural language processing technologies in support of the growth and competitiveness of the Canadian language industry and other Canadian industries.
This evaluation was conducted by an independent evaluation team from the NRC Office of Audit and Evaluation (OAE). It provides information on the relevance and performance (including effectiveness and efficiency) of NRC's initiative under the Roadmap. Given the relatively small size of the NRC initiative and the fact that NRC previously conducted a comprehensive evaluation of the initiative in early 2012 Footnote 1, the OAE undertook a calibrated evaluation of the funded initiative. The evaluation was streamlined to narrow the scope and employ a limited number of methods compared to traditional evaluations. Calibrated evaluations allow for the effective utilization of evaluation resources, while maintaining the credibility and usability of the evaluation results. It is also worth noting that an evaluation of the broader ICT Portfolio is planned for 2017-18.
The evaluation methods included a review of internal and external documents (key sources included in the bibliography) as well as administrative data. In addition, in-depth interviews were conducted with internal (4) and external (4) stakeholders as well as with clients and collaborators (6).
Relevance
The following are the findings of the evaluation with regards to the relevance of the initiative:
- There is demand for Statistical Machine Translation (SMT) technology by the Canadian public sector and from industry domains outside the traditional focus of language translation. However, there is little evidence of significant industry pull for SMT technology from the principle target of NRC's initiative under the Roadmap - the Canadian translation industry.
- NRC's commitments under the Roadmap are consistent with its strategic outcomes as well as with federal priorities. NRC's initiative is also broadly aligned to the ultimate outcome of the Roadmap. However, the alignment of NRC's activities and outcomes to the objectives of the education pillar, where it has been placed within the Roadmap's results architecture, could be better articulated.
- NRC's role in research and development of natural language processing technologies is appropriate, given that it aligns with NRC's mandate and that internationally Canada's peer governments are making similar investments. NRC's ICT Portfolio has a critical mass of experts in natural language processing, who are recognized world-wide, supporting the notion that NRC's role in delivering this initiative is conducive to strong results.
- NRC plays a vital role in maintaining Canada's domestic capacity in natural language processing. In its absence, there is no other organization, public or private, willing or able to take on this role.
Performance
Performance was considered in the context of:
- the research team's contribution to the state-of-the-art in natural language processing;
- impact on Canadian language industry and other industries; and
- efficiency.
The following are the findings of the evaluation with regards to performance:
- The success of the initiative in terms of its excellence in research and development has not translated to significant uptake by the private sector. Presently, the receptor capacity for the technologies developed has been most apparent in the public sector.
- NRC's SMT software, developed under the initiative, has had a significant impact on the public sector.
- NRC's suite of language technologies, developed with assistance from Roadmap funding, has had a broader application beyond the Canadian language industry.
- NRC resources are being used efficiently in responding to the organization's commitments under the Roadmap.
Recommendations:
The evaluation put forth one recommendation:
Recommendation: NRC should consider its fit within the education pillar of the Roadmap and seek to better articulate how its activities and expected outputs align with the Roadmap's results architecture.
To view the management response, please see the full evaluation report.
Footnotes
- Footnote 1
-
A previous evaluation of the initiative was conducted in 2012; however, as the Department of Canadian Heritage (PCH) is currently conducting a horizontal evaluation of the broader Roadmap, in order to meet the terms of the Horizontal Coordination Framework for Roadmap 2013-2018, an evaluation covering the three-year period of 2013-14 to 2015-16 is needed. This current evaluation will serve as a source of data to support the horizontal evaluation.