Status: | Active |
---|---|
Effective date: | December 21, 2020 |
Prepared by: | Office of Audit and Evaluation, National Research Council Canada |
Approval: | NRC's President |
ISBN: | 978‑0‑660‑37304‑1 |
Related document: | Summary report - Evaluation of NRC's Grants for International Affiliations Program |
Alternate format: | Evaluation of NRC's Grants for International Affiliations Program (PDF, 757 KB) |
Acronyms
- APR
- Annual Performance Review
- CCISA
- Canadian Committee for International Scientific Affiliations
- CNC
- Canadian National Committee
- EDI
- Equality, Diversity and Inclusion
- FTE
- Full‑time equivalent
- GBA+
- Gender‑based analysis plus
- GIA
- Grants for International Affiliations program
- ICSU
- International Council for Science Unions
- IIO
- International Innovation Office
- IRO
- International Relations Office
- ISC
- International Science Council
- ISSC
- International Social Sciences Council
- NRC
- National Research Council of Canada
- OAE
- Office of Audit and Evaluation
- OGD
- Other governmental departments (Canadian federal)
- SSHRC
- Social Sciences and Humanities Research Council
- STEM
- Science, technology, engineering, and mathematics
Executive summary
The Grants for International Affiliations (GIA) program is a transfer payment program that maintains Canada's memberships in the International Science Council (ISC), an umbrella organization, and in 28 discipline‑specific scientific unions and associations. The NRC maintains partner agreements with domestic learned societies (such as scholarly societies or academic associations) and federal agencies that support Canada's affiliations to international organizations by maintaining Canadian National Committees (CNCs). Funds are transferred to the international organizations to allow the CNCs to engage with their respective international communities of scientists and policymakers.
The GIA program is administered by the International Relations Office, and the Secretary General holds accountability for the program. It has an annual grants and contributions budget of $560,000. As of 2019, the program has been guided by a steering committee, the Canadian Committee for International Scientific Affiliations, which is comprised of science‑based federal agencies and other national stakeholders; it is co‑chaired by the NRC and the Social Sciences and Humanities Research Council.
The evaluation of the National Research Council (NRC)'s Grants for International Affiliations (GIA) program covered the 2015‑16 to 2019‑20 fiscal year period. It was conducted by the NRC's Office of Audit and Evaluation (OAE). It assessed the program's relevance, effectiveness, and efficiency, drawing on a mixed‑methods approach including data review, document review and environment scan, and external and internal interviews.
Findings
Relevance
The GIA program is engaging with the most relevant umbrella organization for international scientific organizations, the International Science Council (ISC), as well as supporting 28 discipline‑specific international organizations and, indirectly, their Canadian counterparts. There are opportunities to engage more with the ISC, and to support additional affiliations relevant to Canada, but the program is limited by its budget. The new steering committee represents a balance between government and non‑government members from both the natural and social sciences, which will allow it to articulate and promote Canada's science priorities.
Effectiveness
With the support of the GIA program, Canadian scientists have led and participated in international science and technology efforts coordinated by their respective international scientific organizations. Participation has created new opportunities for Canadian scientists and students, including those from groups traditionally excluded from STEM fields, to network, collaborate, and partner with world leaders in their fields.
Efficiency
The GIA program maintains its affiliations with a static budget that has not increased in approximately 15 years. It is becoming increasingly difficult to maintain these affiliations with rising inflation and increases in membership fees. The NRC and the SSHRC need to come to an agreed upon governance model to clarify GIA roles and responsibilities. Changes to the annual performance reporting received positive feedback, however consistency in the process will facilitate better tracking of program performance.
Recommendations
Based on the findings of the evaluation, the OAE has made the following recommendations to strengthen the GIA program's relevance and efficiency:
- It is recommended that the International Relations Office (IRO), in consultation with the Canadian Committee for International Scientific Affiliations (CCISA), determine selection criteria for which affiliations it supports and to what extent if tiered memberships are offered. Criteria for consideration should include:
- strategic alignment with Canada's priorities in international science
- involvement of Canadians in leadership positions and participation in committees
- demonstrated contributions by Canadians to the advancement of knowledge and/or area(s) where Canada may want to advance its knowledge through international engagement
- It is the recommended that the IRO standardize the Annual Performance Review (APR) process, including consistent phrasing of questions, timing, and frequency. Questions should align with Canada's priorities in science, and with the selection criteria to be developed.
Introduction
The evaluation of NRC's Grants for International Affiliations (GIA) program covered the period of 2015‑16 to 2019‑20. It was carried out in accordance with NRC's approved evaluation plan and Treasury Board's Policy on Results (2016) as well as the requirements of the Financial Administration Act. The program was last evaluated in 2015‑2016 and covered the 2010‑11 to 2014‑15 fiscal year period.
This report begins by providing a profile of the GIA program. It then presents the evaluation findings on the relevance, effectiveness, and efficiency of the program. Following the conclusion are two recommendations for improvements within the program.
Evaluation approach
Methods
Mixed methods were used to maximize the generation of relevant evaluation findings. This approach also allowed for convergence of results across methods and contributed to a better understanding of complex issues by exploring different facets. Methods employed included:
- data review (financial, administrative, and performance data)
- document review and environment scan
- external and internal interviews (N=19)
For more detailed information on the methods, including challenges and limitations, refer to Appendix A.
Scope
This evaluation focused on the GIA program's support for the International Science Council and discipline‑specific scientific organizations. GIA also funds Canada's membership in Eureka/Eurostars, an intergovernmental body that facilitates funding for transnational research and development. The NRC delivers Canada's membership in Eureka/EuroStars, making it open to Canadian participants. The NRC's Industrial Research Assistance Program supports eligible small‑to‑medium enterprises with project funding. The NRC and Canada's participation in Eureka/Eurostars will be covered in a future evaluation.
Evaluation questions
Relevance
1. Is the GIA program engaging with the most relevant scientific organizations, both internationally and nationally?
2. Is the NRC the most appropriate administrator for the GIA program?
Effectiveness
3. To what extent has Canada's participation in GIA‑supported international scientific bodies contributed to the advancement of knowledge?
Efficiency
4. Is the GIA program delivered in a cost‑efficient manner, comparing it to similar Canadian programs (where possible)?
5. What has been the impact of recent changes made to the GIA program, including changes to,
- annual performance reviews?
- steering committee?
Profile
The Grants for International Affiliations (GIA) program is a transfer payment program that maintains Canada's memberships in the International Science Council (ISC), an umbrella organization, and in 28 discipline‑specific scientific unions and associations. The NRC maintains partner agreements with domestic learned societies and federal agencies that support Canada's affiliations to international organizations by maintaining Canadian National Committees (CNCs). Funds are transferred to the international organizations to allow the CNCs to engage with their respective international communities of scientists and policymakers.
Grants for International Affiliations program (GIA)
Sources: Data review, and document review and environment scan
- GIA is a transfer payment program that funds Canada's membership dues for non‑governmental Canadian delegates in selected international scientific organizations.
- GIA currently maintains memberships in the International Science Council (ISC), an umbrella organization, and in 28 discipline‑specific scientific unions and associations (herein referred to as organizations). All but four of the organizations are affiliated with the ISC (see the Relevance section for more on the purposes of the ISC and the supported organizations).
- The NRC maintains partner agreements with domestic learned societies and federal agencies that support Canada's affiliations to international organizations. These domestic or federal organizations are referred to as Canadian National Committees (CNCs). For example the Canadian Astronomical Society acts as the partner organization to support the CNC for the International Astronomical Union (see Appendix B for a complete list).
- The CNC for the ISC, the Canadian Committee for International Scientific Affiliations (CCISA), is co‑chaired by the NRC and the Social Sciences and Humanities Research Council (SSHRC) (see subsection on Program Guidance below).
- To keep informed about the activities of supported organizations, CNCs complete annual performance reviews (APRs). The APRs report on the organizations' yearly achievements, and the extent of Canadian participation.
Finances and resources
- The annual grants budget for GIA is $560,000, though the budget has been exceeded in some years to maintain existing affiliations (as high $645,000 in FY 2017‑18).
- The majority of GIA's funding covered membership dues with discipline‑specific organizations (62%) and membership in the ISC (12%), which have increased over time.
- CNCs are also eligible to apply for travel reimbursements. They may apply for funding to send one individual to their international affiliate's general assembly, which typically occurs once every three years. Attendance at assemblies gives Canadian scientists the opportunity to network and collaborate with their international peers. Travel costs are assessed regularly and account for only 2% of total costs.
Figure 1: Total grants and contributions during the evaluation period (2015‑16 to 2019‑20)
Note: As noted in the Scope subsection of the Introduction on Evaluation Approach, Eureka/EuroStars will be covered in a future evaluation.
Program governance
Sources: Document review and environment scan, and internal interviews
Program administration
- GIA is administered by the NRC's International Relations Office (IRO) and the Secretary General holds accountability for the program.
- The IRO allocates <1 FTE/year to GIA (estimated to be 25% of a program advisor's time, and 40% of a program officer's time). Annual administrative costs (salary) for FY 2019‑20 were estimated at $59,000.
- Program delivery challenges are assessed in the Efficiency section.
Program guidance
- The GIA program's steering committee, CCISA, acts as the CNC for ISC. Since 2019, it has been co‑chaired by the NRC and the SSHRC, as a result of a merger of natural and social science umbrella organizations (the merger and its impact are assessed in the Relevance section).
- CCISA serves two functions: one, a policy role, articulating Canada's position on science policy at the ISC; and two, an advisory role, providing guidance to the NRC on GIA.
- Members include science‑based federal agencies and other national stakeholders (see below). CCISA replaces a previous committee that was comprised exclusively of CNCs which was disbanded in 2012.
Figure 2.
Relevance
The GIA program is engaging with the most relevant umbrella organization for international scientific organizations, the International Science Council (ISC), as well as supporting 28 discipline‑specific international organizations and, indirectly, their Canadian counterparts. There are opportunities to engage more with the ISC, and to support additional affiliations relevant to Canada, but the program is limited by its budget. The new steering committee represents a balance between government and non‑government members from both the natural and social sciences, which will allow it to articulate and promote Canada's science priorities.
International Science Council
The GIA program is engaging with the most relevant umbrella organization for international scientific organizations, the International Science Council (ISC).
Sources: Document review and environment scan, external interviews, and internal interviews
Most relevant fit for the NRC
- The ISC is the most appropriate fit for the GIA program and the NRC, compared to other international science organizations that play a similar role. It is the largest such organization and includes a broad mix of national academies, research councils, and research and technology organizations that span the natural and social sciences.
- Other umbrella organizations tend to have narrower audiences. For example, the Belmont Forum is a partnership of funding agencies (like SSHRC) that supports projects through a request for proposals process. The InterAcademy Partnership, which is a global network of national academies (like the Royal Society of Canada), is focussed on capacity building and education (see Appendix C for detailed comparisons).
International influence, but could engage more
- Membership in the ISC gives Canada influence over international science policy, and fulfills Canada and NRC's commitment to supporting science and innovation, and evidence‑based decision making. It provides opportunities to shape international policies and responses to crises, nominate Canadians to international positions, and allow Canadian scientists to participate in collaborative research programs.
- Due to a static budget (see Efficiency section), staff have been largely limited to program administration (i.e., managing finances, conducting APRs) rather than attending ISC meetings, or supporting Canadian participation in ISC activities.
- There may be more opportunities for program staff to engage with the ISC, such as providing direct support to Canadian researchers or organizations to participate in ISC programs, or putting more effort towards coordination between ISC and Canadian stakeholders.
- Nonetheless, the ISC indicated that Canada is a valuable member of the ISC and has provided feedback on its policies and nominations to ISC executive positions.
Merging natural and social science communities
In 2018, the International Council of Science Unions (ICSU) and the International Social Sciences Council (ISSC) merged to become the International Science Council (ISC). Previously, the NRC and the SSHRC were adhering members to each respectively. The purpose of the merger was to strengthen international, interdisciplinary collaboration and support scientists to contribute solutions to complex and pressing matters of global public concern. The merger has implications on which affiliations the GIA program supports and its governance (see subsections below on Impact of Merger and Renewed Rationale).
Supported affiliations
The GIA program supports 28 discipline‑specific international scientific organizations and, indirectly, their respective Canadian National Committees (CNCs). These 28 organizations have demonstrated their relevance, but it is not known whether other discipline‑specific organizations - particularly in the social sciences - may also be relevant and merit support.
Sources: Document review, internal and external interviews
Supported organizations demonstrate value to Canada; additional affiliations may merit support
- All CNCs interviewed said that affiliating with their respective international unions or organizations supports collaboration and networking with international scientists. This was affirmed by the ISC and the interdisciplinary bodies that sit on the GIA steering committee.
- The current mix of supported organizations is a result of a historical, bottom‑up process, rather than recent or ongoing strategic selection. In the past, selection criteria and steering committee guidance informed which affiliations were supported, but this criteria has not been renewed in recent years. As such, the mix of supported organizations did not change during the evaluation period (2015‑16 to 2019‑20), and there may be gaps between the supported organizations and areas of Canada's strengths.
- The GIA program intends to revisit its selection criteria. In 2020‑21, the program's steering committee proposed a subcommittee to determine selection criteria, which would guide whether to add or end specific affiliations. To date, however, the program has been restricted by its budget and capacity. It has managed to maintain existing affiliations, but has not been able to fund additional ones or take a more top‑down approach to selection.
Impact of merger
- With the merger of the ICSU and ISSC in 2018, there may be opportunities to affiliate with more social science bodies. The merger added 17 social science unions or associations that could be supported.
- Currently all but three of the 28 supported organizations represent natural science disciplines. Affiliations with three social science organizations have been supported by the GIA since before the merger. As there was no GIA‑equivalent program at SSHRC or any other social science organization, CNCs accessed the GIA program for support.
- A 2019‑20 report commissioned by the GIA steering committee identified at least five international social science organizations that may align with Canada's strengths, but that have not be funded due to GIA program budget constraints.
Appropriateness
The NRC continues to be the most appropriate administrator of the GIA program. Given its mandate and national position in the scientific landscape, the NRC is well placed to align Canada's engagement with international science organizations with Canada's science priorities.
Sources: Document review and environment scan, external interviews, and internal interviews
The NRC is appropriate host as "national convenor"
- There was a general consensus among both internal and external stakeholders that the NRC is the most appropriate host for the program. Several OGDs and national organizations that were considered as alternate hosts in 2015‑16 are now members of the steering committee (e.g., Council of Canadian Academies, Global Affairs Canada, Natural Sciences and Engineering Research Council, and Royal Society of Canada).
- Hosting the GIA aligns with the NRC's role as a convenor of science‑based government agencies and national bodies; the program is now more able to articulate and promote Canada's science priorities because of the formal inclusion of other perspectives on the new steering committee.
GIA program does not duplicate other federal programs
- There is no other government department duplicating the NRC's role supporting international affiliations. OGDs' own international engagements are driven by their departmental mandates.
- Per interviews with CCISA members, the NRC and OGDs would benefit from a formal mapping of these engagements to better coordinate activities.
Affiliation practices differ internationally
- There is no international best practice or model for type of organization to administer the affiliation program. Practices differ between countries, reflecting their unique mixes of national science institutions and histories. The NRC is somewhat unique as a federal research organization as host, rather than a national academy (e.g., UK or USA) or a granting council (e.g., France, Germany, and Italy).
Renewed rationale
- The current rationale for keeping the GIA program at the NRC is focussed on the role the NRC plays in bridging science and policy. The NRC is uniquely equipped to translate science into policy options, given its close linkages to OGDs, its historic role as the program's administrator, and its involvement in joint research with international scientists.
- This is a change from five‑to‑ten years ago, when the 2010‑11 and 2015‑16 evaluations of the GIA program concluded that the GIA program no longer aligned with the NRC's overall strategy (when supporting business innovation was prioritized over advancement of knowledge or supporting evidence‑based policy).
Creation of the new steering committee
The new steering committee, the Canadian Committee for International Scientific Affiliations (CCISA), represents a better balance of government and non‑government members from the natural and social sciences compared to the previous committee. This committee was formed to steer the program's development and implementation of policy with respect to participation in international organizations. However, to date, few decisions on the future of the program have been made.
Sources: Document review and environment scan, external interviews and internal interviews
The new steering committee, co‑chaired by the NRC and the SSHRC, was convened in April 2019 and has since met three times (as of October 2020). Refer to the Profile section for the committee's roles and current membership.
Better positioned to articulate "Canadian priorities"
- The steering committee is composed of ten Canadian science institutions in and outside of government which creates opportunities to articulate "Canadian priorities."
- The inclusion of these members responded to the 2015‑16 Evaluation's recommendation to partner with organizations that represent Canada's scientific and engineering communities so as to bolster its representation of these communities.
- The committee's Terms of Reference (approved summer 2020) explains its mandate including (among others) to advise and assist the chairs in developing and implementing policy with respect to participation in international organizations under the ISC. For instance, committee members have suggested nominations of Canadian early career scientists to ISC posts.
- Although no date has been set, the committee intends to add six CNC representatives. Prior to the ISC merger, the GIA program had a committee comprised of scientists who were often CNC members.
Key decisions yet to be made
- The committee has focussed thus far on gathering information, having commissioned internal reports on the relevance of ISC affiliation, the performance of the GIA program, and on Canada's research strengths.
- The COVID‑19 pandemic has had an impact on the progress of committee decisions. Meetings were delayed, and committee members were engaged with their own institutional priorities.
- To date, the committee has not made major decisions on the future of the program, including whether to seek additional funding or make cuts in order to stay within the program's budget. Decisions include whether to reduce the total number of affiliations or reduce membership levels when tiered options are available. See the Efficiency section on program delivery challenges for more on this issue.
Effectiveness
With the support of the GIA program, Canadian scientists have led and participated in international science and technology efforts coordinated by their respective international scientific organizations. Participation has created new opportunities for Canadian scientists and students, including those from groups traditionally excluded from STEM fields, to network, collaborate, and partner with world leaders in their fields.
Supporting Canadian leadership and influence
With the support of the GIA, Canadian scientists participate in international science committees, which creates opportunities to influence international science policy towards Canadian priorities and support Canada's areas of research strength.
Sources: Data review, document review and environment scan, external interviews, and internal interviews
Canadians in leadership positions provide opportunities to influence decisions towards Canadian priorities
- As of 2019, two‑thirds of the supported affiliations had at least one Canadian in an executive position.
- While Canadian leadership rates decreased over the evaluation period, these roles are term‑based and some rotation is to be expected. For instance, in 2016 eight supported organizations had a Canadian president and seven had at least one Canadian vice‑president. In 2019, those figures decreased to four and three, respectively. GIA staff expect these rates to increase after the next rounds of appointments.
- The IRO monitors the rate of Canadians in leadership positions as it is a key performance indicator for the program. The target for 2019 was 69%.
- Canadian participation on executive and non‑executive committees provided opportunities to influence decisions and policy direction towards Canadian interests and priorities. For examples:
- influencing executive committee decisions on hiring and how to best engage early career scientists
- voting on project selection committees to support proposals from Canadians or those that align with Canada's research strengths
- participating in Equity, Diversity and Inclusion (EDI) committees and successfully advocating for organizations to adopt Canadian programs that encourage women's participation in STEM and sensitivity to Indigenous rights (see subsection below on Supporting New Opportunities).
Figure 3.
Supporting international science and technology efforts
Canadian researchers contribute to international science and technology efforts coordinated by their respective international organizations. Examples ranged from Nobel prize‑winning physics research conducted at subterranean Canadian laboratories to naming newly discovered stars and planets in the Cree language.
Sources: Data review, document review and environment scan, and external interviews
- Canadian labs and researchers contributed to physics research on neutrinos, including one Canadian researcher, Arthur Bruce McDonald, who won a Nobel Prize in 2015‑16 for contributions to this research which was planned through their CNC and international organization.
- Canadians led or held major roles on programs to trace contaminates in the ocean, and to measure sea ice and climate change impacts on ecosystems. The CNC also supported coastal management programs for developing nations who benefit from Canada's experience in this area.
- A Canadian social science researcher led an international project on religion and science supported by their international organization. Their CNC also participated in an ISC‑led project that sought historical and philosophical research to support their climate change advocacy projects.
- Canada has contributed to naming and terminology in astronomy and chemistry. In one case, CNCs were able to hold national naming competitions in 2020, ultimately naming a star and planet Nikâwiy and Awasis, translated as "my mother" and "child" in the Cree language.
Meeting Canada's international commitments via the ISC
Participation in the ISC helps Canada meet its commitments to several international agreements and policies on topics as diverse as biological diversity, chemical weapons, climate change, cybersecurity, disaster risk reduction, and urban development. For example, Canadian experts from natural and social sciences have been selected by the ISC to contribute to the UN's Global Sustainable Development Report.
Supporting new opportunities
The GIA program has facilitated new opportunities for Canadian scientists and students to network, collaborate, and partner with world leaders in their field and other international scientists. Most supported organizations have programming to extend these opportunities to early career scientists and groups traditionally excluded from STEM fields.
Sources: Data review, document review and environment scan, and external interviews
Opportunities for Canadians
- Affiliating with the supported international organizations has allowed Canada to host or attend international events that included thousands of Canadian researchers and students.
- CNCs successfully bid for and hosted four general assemblies for international science organizations that were held in Canada, attracting over 10,000 Canadian attendees. The success rate for bids by CNCs was 32%.
- Additionally, during the evaluation period, nearly 1,500 Canadian researchers and students attended international science organizations' annual general meetings; 1,015 of these researchers and students, delivered presentations and 605 of them formally represented Canada in organization business.
Supporting inclusion and diversity
- The supported international organizations have created opportunities and are building capacity for demographic groups traditionally excluded from STEM fields and for early career scientists.
- As of 2019, 88% of supported international organizations or their respective CNC had programs for early career scientists (e.g., travel grants, dedicated awards, co‑op placements), and 50% had programs for women (e.g., mentorship, virtual networking).
- Most CNCs interviewed also reported that their international organizations made supporting developing and emerging nations a priority (e.g., education and exchange programs).
- Two CNCs also reported new programming to engage better with Indigenous communities.
Why is considering diverse populations important? One of the Government of Canada's priorities is to consider Gender‑Based Analysis Plus (GBA+) in all of its activities. GBA+ assesses how diverse people (e.g., gender, race, ethnicity, religion, age, physical differences) may experience policies, programs, and initiatives.
Efficiency
The GIA program maintains its affiliations with a static budget that has not increased in approximately 15 years. It is becoming increasingly difficult to maintain these affiliations with rising inflation and increases in membership fees. The NRC and the SSHRC need to come to an agreed upon governance model to clarify GIA roles and responsibilities. Changes to the annual performance reporting received positive feedback, however consistency in the process will facilitate better tracking of program performance.
Program costs
The GIA program has managed to maintain its affiliation with the ISC and 28 international science organizations with a budget that has remained static since 2005, despite increases in membership fees and rising inflation. It is becoming increasingly challenging to maintain these affiliations, and the program may need to review which scientific bodies it supports and to what extent.
Sources: Data review, external interviews, and internal interviews
Figure 4: International affiliation costs increase annually
Managing increasing costs with limited budget and resources
- Annual affiliation costs have increased (see above figure) although the GIA program budget has remained static since 2005 (excluding an increase to affiliate Eureka/EuroStars; recall Profile).
- The IRO has managed increasing costs by spreading membership payments (which are paid per calendar year) across multiple fiscal years and taking advantage of favourable exchange rates when possible.
- The volatility of the Canadian dollar also poses a challenge for the program, as has been the case for other NRC transfer payment programs, such as the International Telescope Agreements and membership in the Bureau International des Poids et Mesures (BIPM).
- The practice of strategically timing payments is non‑compliant with Treasury Board Policy and Directives. NRC Finance encourages programs to manage grant payments within the allocated fiscal year budgets.
- The IRO has also had limited capacity to support program administration, with only two resources assigned to the program, in addition to their non‑GIA duties (less than one FTE annually).
Program delivery challenges
The program could be engaging more with its existing international affiliations and exploring additional ones, but is unable to at this time due to a limited budget. Additionally, there has been an impact on program governance with the merger of the ICSU and ISSC. As of summer 2020, the NRC, the SSHRC, and the steering committee were in the process of determining program roles and responsibilities. At the time of the evaluation, an agreed upon governance model had not been clarified.
Sources: Document review and environment scan, external interviews, and internal interviews
Roles & responsibilities of the NRC, the SSHRC unclear
The 2018 merger of the ICSU and ISSC into the ISC led to the NRC and the SSHRC working together to guide the program. Both organizations need to come to an agreed upon governance model to clarify GIA roles and responsibilities (see table below).
Potential opportunities missed with ISC, unsupported scientific organizations
"It has been a challenge with this program because it is so small, it is hard for it to get sufficient critical mass to be run efficiently. Our budget does not let us run the program efficiently or let us fully represent the Canadian science interest internationally. For example, when an international incident occurs and Canada is expected to respond."
- As mentioned in the relevance section, there are opportunities to engage more with the ISC and to fund more affiliations, particularly among the social sciences.
- Staff expressed concern that underfunding the program could create a reputational risk, as low engagement with the ISC or reducing affiliations could reflect poorly on Canada.
Context | Opportunity and challenges |
---|---|
Merger Following the merger, a new focus was placed on interdisciplinary collaboration within the ISC. |
Closer links between the NRC and the SSHRC ensure a stronger voice for social science perspectives which respond to the ISC mandate. For example, the NRC and the SSHRC worked together on nominations for the ISC's COVID Education Expert Panel in 2020. |
Relationship to ISC Both the NRC and the SSHRC cast votes at the ISC, with the NRC voting on Canada's behalf (informed by CCISA) and the SSHRC on its own behalf. |
The two‑vote issue is systemic to other ISC member nations that had unique adhering members to its predecessors, the ICSU and ISSC. This issue needs to be resolved by the ISC itself. |
Program responsibility Both the NRC and the SSHRC pay dues to the ISC, but the NRC is responsible for all other affiliation dues. |
As of summer 2020, the two agencies were in discussions about how program responsibilities and resourcing may change. |
Challenges common to NRC grants and contributions programs
This small program is unique within the Canadian natural sciences and social sciences landscape and there are no comparable programs elsewhere in the federal science landscape. In terms of efficiency, GIA faces some of the same challenges as other grants and contributions programs administered by the NRC.
Sources: Document review and environment scan, external interviews, and internal interviews
Comparable issues to other NRC programs
There is no program at the NRC comparable to GIA in terms of size, number of recipients, and purpose. Nonetheless, other grants and contribution programs at the NRC have faced similar capacity and exchange rate challenges, as indicated in recent evaluations and program reviews conducted by the Office of Audit and Evaluation (OAE).
Unique federal program
- Review of the federal science‑based OGDs (i.e., the Tri‑Council and the Chief Science Advisor) indicated that there are no comparable programs to the GIA program.
- CNCs saw the GIA program as unique. They engage with a number of granting programs, such as those managed by Tri‑Council members, but CNCs considered the GIA program and its purpose unique.
Program (Annual cost) | Common challenges (and differences) |
---|---|
Industrial Research Assistance Program (IRAP), contributions to firms and organizations: $770M in 2019‑20, maintained by IRAP division. |
|
International Telescope Agreement (ITA): $26M in 2019‑20, maintained by the Herzberg Astronomy and Astrophysics Research Centre. |
|
Membership in the Bureau International des Poids et Mesures (BIPM): $659,000 contribution in 2019‑20, maintained by the Metrology Research Centre. |
|
Changes to annual performance reviews
The changes made to the annual performance reviews (APRs) have reduced the burden on CNCs. However, inconsistency in APR administration, in terms of questions asked, timing, and frequency, have made it difficult to accurately track program performance, and establish metrics that may inform affiliation selection criteria.
Sources: Document review and environment scan, external interviews, and internal interviews
Ongoing changes made to reduce burden on CNCs
The previous evaluation recommended to decrease reporting burden on CNCs by reducing the length and complexity of the APRs. There have been ongoing efforts by the GIA program to streamline the APRs (summarized below). CNCs appreciated the changes but inconsistencies in the process have had an impact on performance measurement (continued below table).
Area | 2015‑16 | 2016‑17 | 2017‑18 | 2018‑19 | 2019‑20 |
---|---|---|---|---|---|
Questions | 11 open‑ended questions | 24 close‑ended questions | 24 close‑ended questions | 23 questions (18 written, 5 tables) | 16 questions (11 written, 5 tables) |
Mode | Web survey | Web survey | Email, then informal call by program staff | ||
Timing | January 2016 | January 2017 | January 2018 | January 2019 | June 2020 |
Scope | 2015 activity | 2016 activity | 2017 activity | 2018 activity | 2019 activity and 2020 COVID impact |
Respondents | All CNCs | All CNCs | Only three CNCs identified as lagging | All CNCs | All CNCs |
Executive committee tracking | Y | Y | Y | Y | Y |
Non‑executive committee tracking | X | X | X | Y | Y |
EDI tracking | Not asked, but self‑reported by some | Not asked, but self‑reported by some | Not asked, but self‑reported by some | Tracking international EDI programs | Tracking new international or CNC EDI programs |
Standardizing the process would enable more accurate performance tracking year‑over‑year, support the use of selection criteria for affiliations and decision‑making, and facilitate CNCs' ability to plan for and provide accurate information.
Sources: Document review and environment scan, external interviews, and internal interviews
Demand for information versus CNC burden
- CNCs appreciated the reduction of the APR questionnaire and consider the demand for information to be reasonable.
- In 2020, the IRO followed up on the APRs with one‑on‑one video conferences, giving CNC and IRO staff a chance to meet and elaborate on their responses.
- The follow‑up calls were a welcome addition that strengthened engagement between the program and its beneficiaries.
- However, by changing or omitting survey questions annually, it is challenging to compare results year‑over‑year. It may also make it more difficult for CNCs to prepare for the APR.
Need for selection criteria
- The steering committee has proposed a subcommittee to renew selection criteria for supported organizations, which would guide whether to add, modify or end specific affiliations.
- Standardized APRs could serve as a key line of evidence for this process. It would give the GIA program comparable data with which to gauge relative performance, and the ensure CNCs understand what is expected of them to continue receiving GIA program support.
Recommendations
Recommendations and supporting rationale
Determine selection criteria
- In the absence of increased funding and given the rising costs of memberships and inflation, it is necessary for the GIA program to determine how to decrease its spending to meet its budget. The current mix of supported organizations is the result of a bottom‑up process, rather than any recent strategic selection. Historically, CNCs approached the NRC for funding to affiliate with an international organization.
- The program has the opportunity, in consultation with the new steering committee, to play a greater role in the selection of which affiliations to continue supporting and which, if any, other affiliations could be supported.
- The steering committee has proposed a subcommittee to determine selection criteria, which would guide whether to add or end specific affiliations.
Recommendation 1
It is recommended that the International Relations Office (IRO), in consultation with the Canadian Committee for International Scientific Affiliations (CCISA), determine selection criteria for which affiliations it supports and to what extent if tiered memberships are offered. Criteria for consideration should include:
- strategic alignment with Canada's priorities in international science
- involvement of Canadians in leadership positions and participation in committees
- demonstrated contributions by Canadians to the advancement of knowledge and/or area(s) where Canada may want to advance its knowledge through international engagement
Standardize annual performance reviews
- Over the evaluation period (2015‑16 to 2019‑20), the APR process was conducted each FY for a total of five times. Most years some questions on the APR survey changed or were omitted.
- The timing of when the APR process was administered varied year by year. Additionally, in FY 2018 the APR was only administered to three CNCs. This did reduce administrative burden on other CNCs that year, however it made it more difficult to track performance and eliminated one of the few regular touchpoints between CNCs and the GIA program.
- Standardizing the process will enable more accurate performance tracking year‑over‑year, support the use of selection criteria for affiliations and decision‑making, and facilitate CNCs' ability to plan for and provide accurate information.
Recommendation 2
It is the recommended that the IRO standardize the Annual Performance Review (APR) process, including consistent phrasing of questions, timing, and frequency. Questions should align with Canada's priorities in science, and with the selection criteria to be developed.
Management response and action plan
Recommendation | Risk‑level associated with not addressing recommendation | Management response | Measure of achievements | Proposed person(s) responsible | Expected date of completion |
---|---|---|---|---|---|
1. It is recommended that the International Relations Office (IRO), in consultation with the Canadian Committee for International Scientific Affiliations (CCISA), determine selection criteria for which affiliations it supports and to what extent if tiered memberships are offered. Criteria for consideration should include:
|
High |
Response: Accepted. Action: CCISA sub‑committee to be established to determine selection criteria in alignment with three points above and advise CCISA on potential next steps. |
GIA program to draft selection criteria | Melanie Cullins, Executive Director, International Innovation Office | June 30, 2021 |
Establishment of CCISA sub‑committee | June 30, 2021 | ||||
CCISA finalizes selection criteria | June 30, 2022 | ||||
2. It is the recommended that the IRO standardize the Annual Performance Review (APR) process, including consistent phrasing of questions, timing, and frequency. Questions should align with Canada's priorities in science, and with the selection criteria to be developed. | Medium |
Response: Accepted. Action: APR questionnaire will be reviewed and revised to align with the selection criteria for affiliations to be determined as part of Recommendation 1 above. The standardized APR process will be carried out in a consistent manner over the following years. |
APR sent out in January 2021, restoring regular schedule | Melanie Cullins, Executive Director, International Innovation Office | January 2021 |
APR sent out in January 2022 using draft selection criteria | January 2022 | ||||
Beginning January 2023, APR administered using finalized selection criteria | January 2023 |
Appendices
Appendix A – Methodology
Data Review
GIA program administrative and performance data for 2015‑16 to 2019‑20 were reviewed to provide information on program inputs (i.e., resources), outputs, and client reach. This included financial data, and annual performance review (APR) data collected from the CNCs (available for 2016, 2018 and 2019).
Key informant interviews
Interviews were conducted with 19 stakeholders (6 internal and 13 external) to collect information such as personal experiences, opinions and expert knowledge related to the relevance and performance of IA. This includes interviews with current NRC employees (n=5) and former program staff (n=1). Interviews with the ISC (n=1), other interdisciplinary bodies (n=3) and CNC's to supported international bodies (n=9) provided an external review of IA.
This information was used to complement other lines of evidence and to contextualize quantitative information.
Document review and environment scan
Internal and external documents were reviewed to provide context and to complement other lines of evidence in assessing relevance and performance. Internal documents included previous GIA program evaluations and MRAP status reports, documentation for other NRC transfer payment programs, steering committee meeting minutes, steering committee member reports and draft terms of reference for CCISA.
External documents included ISC public statements and an environmental scan and report completed by Library and Information Management (LIMS). The LIMS report provided external evidence on international science organizations comparable to the ISC, and an overview of the science‑related fora that Canadian OGDs and other national science organizations are affiliated with.
Limitations and mitigation strategies
Although the evaluation encountered some challenges, methodological limitations were mitigated, where possible, through the use of multiple lines of evidence and the triangulation of data. This approach was taken in order to establish the reliability and validity of the findings and to ensure that conclusions and recommendations were based on objective and documented evidence. Details on limitations and their associated mitigation strategies are described below.
Issues with APR data
- APR data provided was inconsistent as several different versions were used between 2015‑2016 and 2019‑2020. This led to difficulties with comparing information and interpreting results.
- To mitigate this limitation, OAE sought out consistent indicators and worked closely with the program to ensure correct interpretations of data. It is recommended that GIA standardize the collection of APR data to prevent the recurrence of this limitation in future evaluations.
Issues with obtaining annual administrative costs
- Due to the small size of the program, NRC staff do not actively track time resources spent on GIA. For this reason, NRC‑Evaluation was unable to obtain true administration costs for the evaluation period.
- OAE was unable to obtain administrative costs from staff at SSHRC who co‑chair the program.
- To mitigate this limitation, NRC staff provided time estimates which were used to calculate estimated salary totals. OAE also consulted with former GIA personnel to establish a cost baseline.
Limited corporate memory due to turnover
- Both IRO and CNCs were subject to significant turnover during the evaluation period. IRO staff directly responsible for the program had been with the NRC for approximately one year. At CNCs, roles are generally term based so turnover is common and representatives often only had direct experience with a portion of the evaluation period.
- To mitigate this limitation, OAE interviewed past office holders. This included the GIA's former program advisor, who was present for most of the evaluation period. OAE also interviewed senior management who had longer experience with the program. For CNCs, in some instances OAE sought to interview past CNC representatives (e.g., past presidents) who had had more familiarity with the GIA's history and changes.
Appendix B – Supported international scientific organizations and their Canadian counterparts
Supported international scientific organization | Canadian National Committee | Partner Organization | Affiliated with ISC? | Natural or Social Science |
---|---|---|---|---|
International Commission on Illumination (CIE) | CNC‑CIE | National Research Council of Canada | Yes | Natural |
Committee on Data for Science and Technology (CODATA) | CNC‑CODATA | National Research Council of Canada | Yes | Natural |
Committee on Space Research (COSPAR) | CNC‑COSPAR | Canadian Space Agency | Yes | Natural |
International Association for the Properties of Water and Steam (IAPWS) | CNC‑IAPWS | CANDU Owners Group | No | Natural |
International Astronomical Union (IAU) | CNC‑IAU | Canadian Astronomical Society | Yes | Natural |
International Geographical Union (IGU) | CNC‑IGU | Canadian Association of Geographers | Yes | Mix |
International Mathematical Union (IMU) | CNC‑IMU | Canadian Mathematical Society | Yes | Natural |
International Union for Quaternary Research (INQUA) | CANQUA‑INQUA | Canadian Quaternary Association | Yes | Natural |
International Permafrost Association (IPA) | CNC‑IPA | Geological Survey of Canada, Natural Resources Canada | No | Natural |
International Society for Soil Mechanics and Geotechnical Engineering (ISSMGE) | CGS‑ISSMGE | Canadian Geotechnical Society | No | Natural |
International Union of Biochemistry and Molecular Biology (IUBMB) | CSMB‑IUBMB | Canadian Society for Molecular Biosciences | No | Natural |
International Union of Crystallography (IUCr) | CNC‑IUCr | Canadian Light Source | Yes | Natural |
International Union of Geodesy and Geophysics (IUGG) | CNC‑IUGG | Canadian Geophysical Union | Yes | Natural |
International Geological Sciences (IUGS) | CFES‑IUGG | Canadian Federation of Earth Sciences | Yes | Natural |
International Union of History and Philosophy of Science and Technology (IUPS)* | CNC‑IUPS | Canadian Society for the History and Philosophy of Science | Yes | Social |
International Union of Nutritional Sciences (IUNS) | CNS‑IUNS | Canadian Nutrition Society | Yes | Natural |
International Union of Pure and Applied Biophysics (IUPAB) | BSC‑IUPAB | Biophysical Society of Canada | Yes | Natural |
International Union of Pure and Applied Chemistry (IUPAC) | CNC‑IUPAC | National Research Council | Yes | Natural |
International Union of Pure and Applied Physics (IUPAP) | CNC‑IUPAP | National Research Council of Canada | Yes | Natural |
International Union of Basic and Clinical Pharmacology (IUPHAR) | CNC‑IUPHAR | Canadian Society of Pharmacology and Therapeutics | Yes | Natural |
International Union of Physiological Sciences (IUPS) | CNC‑IUPS | Canadian Physiological Society | Yes | Natural |
International Union of Psychological Science (IUPsyS) | CNC‑IUPsyS | Canadian Psychological Association | Yes | Social |
International Union of Theoretical and Applied Mechanics (IUTAM) | CNC‑IUTAM | Canadian Society for Mechanical Engineering | Yes | Natural |
Scientific Committee on Oceanic Research (SCOR) | CNC‑SCOR | Department of Fisheries and Oceans Canada | Yes | Natural |
Scientific Committee on Solar‑Terrestrial Physics (SCOSTEP) | CNC‑SCOTSTEP | Canadian Space Agency | Yes | Natural |
International Union of Radio Science (URSI) | CNC‑URSI | National Research Council | Yes | Natural |
World Climate Research Programme (WCRP) | CNC‑WCRP | Environment and Climate Change Canada | Yes | Natural |
*IUPS has distinct branches (IUPS‑DHST and IUPS‑DLMPST), each of which collect their own dues. They are counted as two supported organizations |
Appendix C – Comparing umbrella organizations for international scientific organizations
Organization | Number of member organizations or institutes | Member characteristics | Primary organizational objectives |
---|---|---|---|
International Science Council (ISC) |
|
|
|
InterAcademy Partnership (IAP) |
|
|
|
Belmont Forum |
|
|
|
Global Research Council (GRC) |
|
|
|