Evaluation of NRC’s Grants for International Affiliations Program

 
Information
Status: Active
Effective date: December 21, 2020
Prepared by: Office of Audit and Evaluation, National Research Council Canada
Approval: NRC's President
ISBN: 978‑0‑660‑37304‑1
Related document: Summary report - Evaluation of NRC's Grants for International Affiliations Program
Alternate format: Evaluation of NRC's Grants for International Affiliations Program (PDF, 757 KB)

Acronyms

APR
Annual Performance Review
CCISA
Canadian Committee for International Scientific Affiliations
CNC
Canadian National Committee
EDI
Equality, Diversity and Inclusion
FTE
Full‑time equivalent
GBA+
Gender‑based analysis plus
GIA
Grants for International Affiliations program
ICSU
International Council for Science Unions
IIO
International Innovation Office
IRO
International Relations Office
ISC
International Science Council
ISSC
International Social Sciences Council
NRC
National Research Council of Canada
OAE
Office of Audit and Evaluation
OGD
Other governmental departments (Canadian federal)
SSHRC
Social Sciences and Humanities Research Council
STEM
Science, technology, engineering, and mathematics

Executive summary

The Grants for International Affiliations (GIA) program is a transfer payment program that maintains Canada's memberships in the International Science Council (ISC), an umbrella organization, and in 28 discipline‑specific scientific unions and associations. The NRC maintains partner agreements with domestic learned societies (such as scholarly societies or academic associations) and federal agencies that support Canada's affiliations to international organizations by maintaining Canadian National Committees (CNCs). Funds are transferred to the international organizations to allow the CNCs to engage with their respective international communities of scientists and policymakers.

The GIA program is administered by the International Relations Office, and the Secretary General holds accountability for the program. It has an annual grants and contributions budget of $560,000. As of 2019, the program has been guided by a steering committee, the Canadian Committee for International Scientific Affiliations, which is comprised of science‑based federal agencies and other national stakeholders; it is co‑chaired by the NRC and the Social Sciences and Humanities Research Council.

The evaluation of the National Research Council (NRC)'s Grants for International Affiliations (GIA) program covered the 2015‑16 to 2019‑20 fiscal year period. It was conducted by the NRC's Office of Audit and Evaluation (OAE). It assessed the program's relevance, effectiveness, and efficiency, drawing on a mixed‑methods approach including data review, document review and environment scan, and external and internal interviews.

Findings

Relevance

The GIA program is engaging with the most relevant umbrella organization for international scientific organizations, the International Science Council (ISC), as well as supporting 28 discipline‑specific international organizations and, indirectly, their Canadian counterparts. There are opportunities to engage more with the ISC, and to support additional affiliations relevant to Canada, but the program is limited by its budget. The new steering committee represents a balance between government and non‑government members from both the natural and social sciences, which will allow it to articulate and promote Canada's science priorities.

Effectiveness

With the support of the GIA program, Canadian scientists have led and participated in international science and technology efforts coordinated by their respective international scientific organizations. Participation has created new opportunities for Canadian scientists and students, including those from groups traditionally excluded from STEM fields, to network, collaborate, and partner with world leaders in their fields.

Efficiency

The GIA program maintains its affiliations with a static budget that has not increased in approximately 15 years. It is becoming increasingly difficult to maintain these affiliations with rising inflation and increases in membership fees. The NRC and the SSHRC need to come to an agreed upon governance model to clarify GIA roles and responsibilities. Changes to the annual performance reporting received positive feedback, however consistency in the process will facilitate better tracking of program performance.

Recommendations

Based on the findings of the evaluation, the OAE has made the following recommendations to strengthen the GIA program's relevance and efficiency:

  1. It is recommended that the International Relations Office (IRO), in consultation with the Canadian Committee for International Scientific Affiliations (CCISA), determine selection criteria for which affiliations it supports and to what extent if tiered memberships are offered. Criteria for consideration should include:
    1. strategic alignment with Canada's priorities in international science
    2. involvement of Canadians in leadership positions and participation in committees
    3. demonstrated contributions by Canadians to the advancement of knowledge and/or area(s) where Canada may want to advance its knowledge through international engagement
  2. It is the recommended that the IRO standardize the Annual Performance Review (APR) process, including consistent phrasing of questions, timing, and frequency. Questions should align with Canada's priorities in science, and with the selection criteria to be developed.

Introduction

The evaluation of NRC's Grants for International Affiliations (GIA) program covered the period of 2015‑16 to 2019‑20. It was carried out in accordance with NRC's approved evaluation plan and Treasury Board's Policy on Results (2016) as well as the requirements of the Financial Administration Act. The program was last evaluated in 2015‑2016 and covered the 2010‑11 to 2014‑15 fiscal year period.

This report begins by providing a profile of the GIA program. It then presents the evaluation findings on the relevance, effectiveness, and efficiency of the program. Following the conclusion are two recommendations for improvements within the program.

Evaluation approach

Methods

Mixed methods were used to maximize the generation of relevant evaluation findings. This approach also allowed for convergence of results across methods and contributed to a better understanding of complex issues by exploring different facets. Methods employed included:

  • data review (financial, administrative, and performance data)
  • document review and environment scan
  • external and internal interviews (N=19)

For more detailed information on the methods, including challenges and limitations, refer to Appendix A.

Scope

This evaluation focused on the GIA program's support for the International Science Council and discipline‑specific scientific organizations. GIA also funds Canada's membership in Eureka/Eurostars, an intergovernmental body that facilitates funding for transnational research and development. The NRC delivers Canada's membership in Eureka/EuroStars, making it open to Canadian participants. The NRC's Industrial Research Assistance Program supports eligible small‑to‑medium enterprises with project funding. The NRC and Canada's participation in Eureka/Eurostars will be covered in a future evaluation.

Evaluation questions

Relevance

1. Is the GIA program engaging with the most relevant scientific organizations, both internationally and nationally?

2. Is the NRC the most appropriate administrator for the GIA program?

Effectiveness

3. To what extent has Canada's participation in GIA‑supported international scientific bodies contributed to the advancement of knowledge?

Efficiency

4. Is the GIA program delivered in a cost‑efficient manner, comparing it to similar Canadian programs (where possible)?

5. What has been the impact of recent changes made to the GIA program, including changes to,

  1. annual performance reviews?
  2. steering committee?

Profile

The Grants for International Affiliations (GIA) program is a transfer payment program that maintains Canada's memberships in the International Science Council (ISC), an umbrella organization, and in 28 discipline‑specific scientific unions and associations. The NRC maintains partner agreements with domestic learned societies and federal agencies that support Canada's affiliations to international organizations by maintaining Canadian National Committees (CNCs). Funds are transferred to the international organizations to allow the CNCs to engage with their respective international communities of scientists and policymakers.

Grants for International Affiliations program (GIA)

Sources: Data review, and document review and environment scan

  • GIA is a transfer payment program that funds Canada's membership dues for non‑governmental Canadian delegates in selected international scientific organizations.
  • GIA currently maintains memberships in the International Science Council (ISC), an umbrella organization, and in 28 discipline‑specific scientific unions and associations (herein referred to as organizations). All but four of the organizations are affiliated with the ISC (see the Relevance section for more on the purposes of the ISC and the supported organizations).
  • The NRC maintains partner agreements with domestic learned societies and federal agencies that support Canada's affiliations to international organizations. These domestic or federal organizations are referred to as Canadian National Committees (CNCs). For example the Canadian Astronomical Society acts as the partner organization to support the CNC for the International Astronomical Union (see Appendix B for a complete list).
  • The CNC for the ISC, the Canadian Committee for International Scientific Affiliations (CCISA), is co‑chaired by the NRC and the Social Sciences and Humanities Research Council (SSHRC) (see subsection on Program Guidance below).
  • To keep informed about the activities of supported organizations, CNCs complete annual performance reviews (APRs). The APRs report on the organizations' yearly achievements, and the extent of Canadian participation.

Finances and resources

  • The annual grants budget for GIA is $560,000, though the budget has been exceeded in some years to maintain existing affiliations (as high $645,000 in FY 2017‑18).
  • The majority of GIA's funding covered membership dues with discipline‑specific organizations (62%) and membership in the ISC (12%), which have increased over time.
  • CNCs are also eligible to apply for travel reimbursements. They may apply for funding to send one individual to their international affiliate's general assembly, which typically occurs once every three years. Attendance at assemblies gives Canadian scientists the opportunity to network and collaborate with their international peers. Travel costs are assessed regularly and account for only 2% of total costs.

Figure 1: Total grants and contributions during the evaluation period (2015‑16 to 2019‑20)

 
Figure 1
Figure 1 - Text version

A horizontal bar graph depicting total grants and contributions during the evaluation period (2015‑16 to 2019‑20) subdivided to:

  • Dues for scientific organizations totals $1.8 million
  • Eureka/Eurostars totals $731.2 thousand
  • ISC membership totals $356.8 thousand
  • Travel reimbursements totals $47.0 thousand
 

Note: As noted in the Scope subsection of the Introduction on Evaluation Approach, Eureka/EuroStars will be covered in a future evaluation.

Program governance

Sources: Document review and environment scan, and internal interviews

Program administration

  • GIA is administered by the NRC's International Relations Office (IRO) and the Secretary General holds accountability for the program.
  • The IRO allocates <1 FTE/year to GIA (estimated to be 25% of a program advisor's time, and 40% of a program officer's time). Annual administrative costs (salary) for FY 2019‑20 were estimated at $59,000.
  • Program delivery challenges are assessed in the Efficiency section.

Program guidance

  • The GIA program's steering committee, CCISA, acts as the CNC for ISC. Since 2019, it has been co‑chaired by the NRC and the SSHRC, as a result of a merger of natural and social science umbrella organizations (the merger and its impact are assessed in the Relevance section).
  • CCISA serves two functions: one, a policy role, articulating Canada's position on science policy at the ISC; and two, an advisory role, providing guidance to the NRC on GIA.
  • Members include science‑based federal agencies and other national stakeholders (see below). CCISA replaces a previous committee that was comprised exclusively of CNCs which was disbanded in 2012.

Figure 2.

 
 Figure 2
Figure 2 - Text version

An Illustration of GIA program governance using two flowcharts. The flowcharts describe the following relationships:

Program administration for the GIA at the NRC is as follows: The program is administered by the IRO. The IRO is a unit within the IIO. Ultimate responsibilities for the program is the Secretary General.

Program guidance for the GIA at the NRC is as follows:

The program receives guidance from the CCISA. CCISA acts at the CNC for the ISC. The CCISA is composed of the following committee membership organizations:

  • Canadian Foundation for Innovation (CFI)
  • Canadian Institute of Health Research (CIHR)
  • Council of Canadian Academies (CCA)
  • Global Affairs Canada (GAC)
  • Innovation, Science and Economic Development (ISED)
  • National Research Council (NRC)
  • Natural Sciences and Engineering Research Council (NSERC)
  • Office of the Chief Science Advisor (OCSA)
  • Royal Society of Canada (RSC)
  • Social Sciences and Humanities Research Council (SSHRC)
  • Six CNC Representatives – to be determined
 

Relevance

The GIA program is engaging with the most relevant umbrella organization for international scientific organizations, the International Science Council (ISC), as well as supporting 28 discipline‑specific international organizations and, indirectly, their Canadian counterparts. There are opportunities to engage more with the ISC, and to support additional affiliations relevant to Canada, but the program is limited by its budget. The new steering committee represents a balance between government and non‑government members from both the natural and social sciences, which will allow it to articulate and promote Canada's science priorities.

International Science Council

The GIA program is engaging with the most relevant umbrella organization for international scientific organizations, the International Science Council (ISC). 

Sources: Document review and environment scan, external interviews, and internal interviews

Most relevant fit for the NRC

  • The ISC is the most appropriate fit for the GIA program and the NRC, compared to other international science organizations that play a similar role. It is the largest such organization and includes a broad mix of national academies, research councils, and research and technology organizations that span the natural and social sciences.
  • Other umbrella organizations tend to have narrower audiences. For example, the Belmont Forum is a partnership of funding agencies (like SSHRC) that supports projects through a request for proposals process. The InterAcademy Partnership, which is a global network of national academies (like the Royal Society of Canada), is focussed on capacity building and education (see Appendix C for detailed comparisons).

International influence, but could engage more

  • Membership in the ISC gives Canada influence over international science policy, and fulfills Canada and NRC's commitment to supporting science and innovation, and evidence‑based decision making. It provides opportunities to shape international policies and responses to crises, nominate Canadians to international positions, and allow Canadian scientists to participate in collaborative research programs.
  • Due to a static budget (see Efficiency section), staff have been largely limited to program administration (i.e., managing finances, conducting APRs) rather than attending ISC meetings, or supporting Canadian participation in ISC activities.
  • There may be more opportunities for program staff to engage with the ISC, such as providing direct support to Canadian researchers or organizations to participate in ISC programs, or putting more effort towards coordination between ISC and Canadian stakeholders.
  • Nonetheless, the ISC indicated that Canada is a valuable member of the ISC and has provided feedback on its policies and nominations to ISC executive positions.

Merging natural and social science communities
In 2018, the International Council of Science Unions (ICSU) and the International Social Sciences Council (ISSC) merged to become the International Science Council (ISC). Previously, the NRC and the SSHRC were adhering members to each respectively. The purpose of the merger was to strengthen international, interdisciplinary collaboration and support scientists to contribute solutions to complex and pressing matters of global public concern. The merger has implications on which affiliations the GIA program supports and its governance (see subsections below on Impact of Merger and Renewed Rationale).

Supported affiliations

The GIA program supports 28 discipline‑specific international scientific organizations and, indirectly, their respective Canadian National Committees (CNCs). These 28 organizations have demonstrated their relevance, but it is not known whether other discipline‑specific organizations - particularly in the social sciences - may also be relevant and merit support.

Sources: Document review, internal and external interviews

Supported organizations demonstrate value to Canada; additional affiliations may merit support

  • All CNCs interviewed said that affiliating with their respective international unions or organizations supports collaboration and networking with international scientists. This was affirmed by the ISC and the interdisciplinary bodies that sit on the GIA steering committee.
  • The current mix of supported organizations is a result of a historical, bottom‑up process, rather than recent or ongoing strategic selection. In the past, selection criteria and steering committee guidance informed which affiliations were supported, but this criteria has not been renewed in recent years. As such, the mix of supported organizations did not change during the evaluation period (2015‑16 to 2019‑20), and there may be gaps between the supported organizations and areas of Canada's strengths.
  • The GIA program intends to revisit its selection criteria. In 2020‑21, the program's steering committee proposed a subcommittee to determine selection criteria, which would guide whether to add or end specific affiliations. To date, however, the program has been restricted by its budget and capacity. It has managed to maintain existing affiliations, but has not been able to fund additional ones or take a more top‑down approach to selection.

Impact of merger

  • With the merger of the ICSU and ISSC in 2018, there may be opportunities to affiliate with more social science bodies. The merger added 17 social science unions or associations that could be supported.
  • Currently all but three of the 28 supported organizations represent natural science disciplines. Affiliations with three social science organizations have been supported by the GIA since before the merger. As there was no GIA‑equivalent program at SSHRC or any other social science organization, CNCs accessed the GIA program for support.
  • A 2019‑20 report commissioned by the GIA steering committee identified at least five international social science organizations that may align with Canada's strengths, but that have not be funded due to GIA program budget constraints.

Appropriateness

The NRC continues to be the most appropriate administrator of the GIA program. Given its mandate and national position in the scientific landscape, the NRC is well placed to align Canada's engagement with international science organizations with Canada's science priorities.

Sources: Document review and environment scan, external interviews, and internal interviews

The NRC is appropriate host as "national convenor"

  • There was a general consensus among both internal and external stakeholders that the NRC is the most appropriate host for the program. Several OGDs and national organizations that were considered as alternate hosts in 2015‑16 are now members of the steering committee (e.g., Council of Canadian Academies, Global Affairs Canada, Natural Sciences and Engineering Research Council, and Royal Society of Canada).
  • Hosting the GIA aligns with the NRC's role as a convenor of science‑based government agencies and national bodies; the program is now more able to articulate and promote Canada's science priorities because of the formal inclusion of other perspectives on the new steering committee.

GIA program does not duplicate other federal programs

  • There is no other government department duplicating the NRC's role supporting international affiliations. OGDs' own international engagements are driven by their departmental mandates.
  • Per interviews with CCISA members, the NRC and OGDs would benefit from a formal mapping of these engagements to better coordinate activities.

Affiliation practices differ internationally

  • There is no international best practice or model for type of organization to administer the affiliation program. Practices differ between countries, reflecting their unique mixes of national science institutions and histories. The NRC is somewhat unique as a federal research organization as host, rather than a national academy (e.g., UK or USA) or a granting council (e.g., France, Germany, and Italy).

Renewed rationale

  • The current rationale for keeping the GIA program at the NRC is focussed on the role the NRC plays in bridging science and policy. The NRC is uniquely equipped to translate science into policy options, given its close linkages to OGDs, its historic role as the program's administrator, and its involvement in joint research with international scientists.
  • This is a change from five‑to‑ten years ago, when the 2010‑11 and 2015‑16 evaluations of the GIA program concluded that the GIA program no longer aligned with the NRC's overall strategy (when supporting business innovation was prioritized over advancement of knowledge or supporting evidence‑based policy).

Creation of the new steering committee

The new steering committee, the Canadian Committee for International Scientific Affiliations (CCISA), represents a better balance of government and non‑government members from the natural and social sciences compared to the previous committee. This committee was formed to steer the program's development and implementation of policy with respect to participation in international organizations. However, to date, few decisions on the future of the program have been made.

Sources: Document review and environment scan, external interviews and internal interviews

The new steering committee, co‑chaired by the NRC and the SSHRC, was convened in April 2019 and has since met three times (as of October 2020). Refer to the Profile section for the committee's roles and current membership.

Better positioned to articulate "Canadian priorities"

  • The steering committee is composed of ten Canadian science institutions in and outside of government which creates opportunities to articulate "Canadian priorities."
  • The inclusion of these members responded to the 2015‑16 Evaluation's recommendation to partner with organizations that represent Canada's scientific and engineering communities so as to bolster its representation of these communities.
  • The committee's Terms of Reference (approved summer 2020) explains its mandate including (among others) to advise and assist the chairs in developing and implementing policy with respect to participation in international organizations under the ISC. For instance, committee members have suggested nominations of Canadian early career scientists to ISC posts.
  • Although no date has been set, the committee intends to add six CNC representatives. Prior to the ISC merger, the GIA program had a committee comprised of scientists who were often CNC members.

Key decisions yet to be made

  • The committee has focussed thus far on gathering information, having commissioned internal reports on the relevance of ISC affiliation, the performance of the GIA program, and on Canada's research strengths.
  • The COVID‑19 pandemic has had an impact on the progress of committee decisions. Meetings were delayed, and committee members were engaged with their own institutional priorities.
  • To date, the committee has not made major decisions on the future of the program, including whether to seek additional funding or make cuts in order to stay within the program's budget. Decisions include whether to reduce the total number of affiliations or reduce membership levels when tiered options are available. See the Efficiency section on program delivery challenges for more on this issue.

Effectiveness

With the support of the GIA program, Canadian scientists have led and participated in international science and technology efforts coordinated by their respective international scientific organizations. Participation has created new opportunities for Canadian scientists and students, including those from groups traditionally excluded from STEM fields, to network, collaborate, and partner with world leaders in their fields.

Supporting Canadian leadership and influence

With the support of the GIA, Canadian scientists participate in international science committees, which creates opportunities to influence international science policy towards Canadian priorities and support Canada's areas of research strength.

Sources: Data review, document review and environment scan, external interviews, and internal interviews

Canadians in leadership positions provide opportunities to influence decisions towards Canadian priorities

  • As of 2019, two‑thirds of the supported affiliations had at least one Canadian in an executive position.
  • While Canadian leadership rates decreased over the evaluation period, these roles are term‑based and some rotation is to be expected. For instance, in 2016 eight supported organizations had a Canadian president and seven had at least one Canadian vice‑president. In 2019, those figures decreased to four and three, respectively. GIA staff expect these rates to increase after the next rounds of appointments.
  • The IRO monitors the rate of Canadians in leadership positions as it is a key performance indicator for the program. The target for 2019 was 69%.
  • Canadian participation on executive and non‑executive committees provided opportunities to influence decisions and policy direction towards Canadian interests and priorities. For examples:
    • influencing executive committee decisions on hiring and how to best engage early career scientists
    • voting on project selection committees to support proposals from Canadians or those that align with Canada's research strengths
    • participating in Equity, Diversity and Inclusion (EDI) committees and successfully advocating for organizations to adopt Canadian programs that encourage women's participation in STEM and sensitivity to Indigenous rights (see subsection below on Supporting New Opportunities).

    Figure 3.

     
    Figure 3
    Figure 3 - Text version

    The top figure is a pictogram that uses 10 globe icons to represent the supported organizations.  7 globes are fully or partially coloured purple representing 67% of supported organization that had at least one Canadian on their executive team in 2019.

    The bottom figure is a pictogram that uses 10 globe icons to represent the supported organizations.  9 globes are fully or partially coloured blue representing 83% of supported organization that had at least one Canadian on their non‑executive team in 2019.

     

    Supporting international science and technology efforts

    Canadian researchers contribute to international science and technology efforts coordinated by their respective international organizations. Examples ranged from Nobel prize‑winning physics research conducted at subterranean Canadian laboratories to naming newly discovered stars and planets in the Cree language.

    Sources: Data review, document review and environment scan, and external interviews

    • Canadian labs and researchers contributed to physics research on neutrinos, including one Canadian researcher, Arthur Bruce McDonald, who won a Nobel Prize in 2015‑16 for contributions to this research which was planned through their CNC and international organization.
    • Canadians led or held major roles on programs to trace contaminates in the ocean, and to measure sea ice and climate change impacts on ecosystems. The CNC also supported coastal management programs for developing nations who benefit from Canada's experience in this area.
    • A Canadian social science researcher led an international project on religion and science supported by their international organization. Their CNC also participated in an ISC‑led project that sought historical and philosophical research to support their climate change advocacy projects.
    • Canada has contributed to naming and terminology in astronomy and chemistry. In one case, CNCs were able to hold national naming competitions in 2020, ultimately naming a star and planet Nikâwiy and Awasis, translated as "my mother" and "child" in the Cree language.

    Meeting Canada's international commitments via the ISC

    Participation in the ISC helps Canada meet its commitments to several international agreements and policies on topics as diverse as biological diversity, chemical weapons, climate change, cybersecurity, disaster risk reduction, and urban development. For example, Canadian experts from natural and social sciences have been selected by the ISC to contribute to the UN's Global Sustainable Development Report.

    Supporting new opportunities

    The GIA program has facilitated new opportunities for Canadian scientists and students to network, collaborate, and partner with world leaders in their field and other international scientists. Most supported organizations have programming to extend these opportunities to early career scientists and groups traditionally excluded from STEM fields.

    Sources: Data review, document review and environment scan, and external interviews

    Opportunities for Canadians

    • Affiliating with the supported international organizations has allowed Canada to host or attend international events that included thousands of Canadian researchers and students.
    • CNCs successfully bid for and hosted four general assemblies for international science organizations that were held in Canada, attracting over 10,000 Canadian attendees. The success rate for bids by CNCs was 32%.
    • Additionally, during the evaluation period, nearly 1,500 Canadian researchers and students attended international science organizations' annual general meetings; 1,015 of these researchers and students, delivered presentations and 605 of them formally represented Canada in organization business.

    Supporting inclusion and diversity

    • The supported international organizations have created opportunities and are building capacity for demographic groups traditionally excluded from STEM fields and for early career scientists.
    • As of 2019, 88% of supported international organizations or their respective CNC had programs for early career scientists (e.g., travel grants, dedicated awards, co‑op placements), and 50% had programs for women (e.g., mentorship, virtual networking).
    • Most CNCs interviewed also reported that their international organizations made supporting developing and emerging nations a priority (e.g., education and exchange programs).
    • Two CNCs also reported new programming to engage better with Indigenous communities.

    Why is considering diverse populations important? One of the Government of Canada's priorities is to consider Gender‑Based Analysis Plus (GBA+) in all of its activities. GBA+ assesses how diverse people (e.g., gender, race, ethnicity, religion, age, physical differences) may experience policies, programs, and initiatives.

    Efficiency

    The GIA program maintains its affiliations with a static budget that has not increased in approximately 15 years. It is becoming increasingly difficult to maintain these affiliations with rising inflation and increases in membership fees. The NRC and the SSHRC need to come to an agreed upon governance model to clarify GIA roles and responsibilities. Changes to the annual performance reporting received positive feedback, however consistency in the process will facilitate better tracking of program performance.

    Program costs

    The GIA program has managed to maintain its affiliation with the ISC and 28 international science organizations with a budget that has remained static since 2005, despite increases in membership fees and rising inflation. It is becoming increasingly challenging to maintain these affiliations, and the program may need to review which scientific bodies it supports and to what extent.

    Sources: Data review, external interviews, and internal interviews

    Figure 4: International affiliation costs increase annually

     
    Figure 4
    Figure 4 - Text version

    A horizontal line chart depicting international affiliation costs increase between 2015 and 2019. Two lines present data on the costs of dues for 28‑discipline specific affiliations and for the annual ISC membership:

    • The line representing dues for 28 discipline‑specific affiliations indicates in 2015 costs totaled at $329 thousand, increasing by 15 percent to $377 thousand in 2019.
    • The line representing annual ISC membership indicates in 2015 this cost totalled at $76 thousand increasing by 24 percent to $94 thousand in 2019.
     

    Managing increasing costs with limited budget and resources

    • Annual affiliation costs have increased (see above figure) although the GIA program budget has remained static since 2005 (excluding an increase to affiliate Eureka/EuroStars; recall Profile).
    • The IRO has managed increasing costs by spreading membership payments (which are paid per calendar year) across multiple fiscal years and taking advantage of favourable exchange rates when possible.
    • The volatility of the Canadian dollar also poses a challenge for the program, as has been the case for other NRC transfer payment programs, such as the International Telescope Agreements and membership in the Bureau International des Poids et Mesures (BIPM).
    • The practice of strategically timing payments is non‑compliant with Treasury Board Policy and Directives. NRC Finance encourages programs to manage grant payments within the allocated fiscal year budgets.
    • The IRO has also had limited capacity to support program administration, with only two resources assigned to the program, in addition to their non‑GIA duties (less than one FTE annually).

    Program delivery challenges

    The program could be engaging more with its existing international affiliations and exploring additional ones, but is unable to at this time due to a limited budget. Additionally, there has been an impact on program governance with the merger of the ICSU and ISSC. As of summer 2020, the NRC, the SSHRC, and the steering committee were in the process of determining program roles and responsibilities. At the time of the evaluation, an agreed upon governance model had not been clarified.

    Sources: Document review and environment scan, external interviews, and internal interviews

    Roles & responsibilities of the NRC, the SSHRC unclear

    The 2018 merger of the ICSU and ISSC into the ISC led to the NRC and the SSHRC working together to guide the program. Both organizations need to come to an agreed upon governance model to clarify GIA roles and responsibilities (see table below).

    Potential opportunities missed with ISC, unsupported scientific organizations

    "It has been a challenge with this program because it is so small, it is hard for it to get sufficient critical mass to be run efficiently. Our budget does not let us run the program efficiently or let us fully represent the Canadian science interest internationally. For example, when an international incident occurs and Canada is expected to respond."

    Internal staff
    • As mentioned in the relevance section, there are opportunities to engage more with the ISC and to fund more affiliations, particularly among the social sciences.
    • Staff expressed concern that underfunding the program could create a reputational risk, as low engagement with the ISC or reducing affiliations could reflect poorly on Canada.
    Table 1
    Context Opportunity and challenges
    Merger
    Following the merger, a new focus was placed on interdisciplinary collaboration within the ISC.
    Closer links between the NRC and the SSHRC ensure a stronger voice for social science perspectives which respond to the ISC mandate. For example, the NRC and the SSHRC worked together on nominations for the ISC's COVID Education Expert Panel in 2020.
    Relationship to ISC
    Both the NRC and the SSHRC cast votes at the ISC, with the NRC voting on Canada's behalf (informed by CCISA) and the SSHRC on its own behalf.
    The two‑vote issue is systemic to other ISC member nations that had unique adhering members to its predecessors, the ICSU and ISSC. This issue needs to be resolved by the ISC itself.
    Program responsibility
    Both the NRC and the SSHRC pay dues to the ISC, but the NRC is responsible for all other affiliation dues.
    As of summer 2020, the two agencies were in discussions about how program responsibilities and resourcing may change.

    Challenges common to NRC grants and contributions programs

    This small program is unique within the Canadian natural sciences and social sciences landscape and there are no comparable programs elsewhere in the federal science landscape. In terms of efficiency, GIA faces some of the same challenges as other grants and contributions programs administered by the NRC.

    Sources: Document review and environment scan, external interviews, and internal interviews

    Comparable issues to other NRC programs

    There is no program at the NRC comparable to GIA in terms of size, number of recipients, and purpose. Nonetheless, other grants and contribution programs at the NRC have faced similar capacity and exchange rate challenges, as indicated in recent evaluations and program reviews conducted by the Office of Audit and Evaluation (OAE).

    Unique federal program

    • Review of the federal science‑based OGDs (i.e., the Tri‑Council and the Chief Science Advisor) indicated that there are no comparable programs to the GIA program.
    • CNCs saw the GIA program as unique. They engage with a number of granting programs, such as those managed by Tri‑Council members, but CNCs considered the GIA program and its purpose unique.
    Table 2
    Program (Annual cost) Common challenges (and differences)
    Industrial Research Assistance Program (IRAP), contributions to firms and organizations: $770M in 2019‑20, maintained by IRAP division.
    • Capacity issues, namely insufficient time allotted to implementing new initiatives, thus incurring higher costs than estimated, or lacking capacity to support new initiatives during their roll‑out.
    • This program is of a far greater magnitude and includes direct support and engagement with grant recipients.
    International Telescope Agreement (ITA): $26M in 2019‑20, maintained by the Herzberg Astronomy and Astrophysics Research Centre.
    • Subject to fluctuations in exchange rates in spite of a capped funding limit.
    • These contribution agreements cover three programs that support Canadian and international telescopes, including a mix of operation costs and access fees.
    Membership in the Bureau International des Poids et Mesures (BIPM): $659,000 contribution in 2019‑20, maintained by the Metrology Research Centre.
    • Subject to fluctuations in exchange rates in spite of a capped funding limit.
    • This is a single membership grant governed by a treaty. It has negligible administrative costs, unlike GIA.

    Changes to annual performance reviews

    The changes made to the annual performance reviews (APRs) have reduced the burden on CNCs. However, inconsistency in APR administration, in terms of questions asked, timing, and frequency, have made it difficult to accurately track program performance, and establish metrics that may inform affiliation selection criteria.

    Sources: Document review and environment scan, external interviews, and internal interviews

    Ongoing changes made to reduce burden on CNCs

    The previous evaluation recommended to decrease reporting burden on CNCs by reducing the length and complexity of the APRs. There have been ongoing efforts by the GIA program to streamline the APRs (summarized below). CNCs appreciated the changes but inconsistencies in the process have had an impact on performance measurement (continued below table).

    Table 3
    Area 2015‑16 2016‑17 2017‑18 2018‑19 2019‑20
    Questions 11 open‑ended questions 24 close‑ended questions 24 close‑ended questions 23 questions (18 written, 5 tables) 16 questions (11 written, 5 tables)
    Mode Web survey Web survey Email Email Email, then informal call by program staff
    Timing January 2016 January 2017 January 2018 January 2019 June 2020
    Scope 2015 activity 2016 activity 2017 activity 2018 activity 2019 activity and 2020 COVID impact
    Respondents All CNCs All CNCs Only three CNCs identified as lagging All CNCs All CNCs
    Executive committee tracking Y Y Y Y Y
    Non‑executive committee tracking X X X Y Y
    EDI tracking Not asked, but self‑reported by some Not asked, but self‑reported by some Not asked, but self‑reported by some Tracking international EDI programs Tracking new international or CNC EDI programs

    Standardizing the process would enable more accurate performance tracking year‑over‑year, support the use of selection criteria for affiliations and decision‑making, and facilitate CNCs' ability to plan for and provide accurate information.

    Sources: Document review and environment scan, external interviews, and internal interviews

    Demand for information versus CNC burden

    • CNCs appreciated the reduction of the APR questionnaire and consider the demand for information to be reasonable.
    • In 2020, the IRO followed up on the APRs with one‑on‑one video conferences, giving CNC and IRO staff a chance to meet and elaborate on their responses.
    • The follow‑up calls were a welcome addition that strengthened engagement between the program and its beneficiaries.
    • However, by changing or omitting survey questions annually, it is challenging to compare results year‑over‑year. It may also make it more difficult for CNCs to prepare for the APR.

    Need for selection criteria

    • The steering committee has proposed a subcommittee to renew selection criteria for supported organizations, which would guide whether to add, modify or end specific affiliations.
    • Standardized APRs could serve as a key line of evidence for this process. It would give the GIA program comparable data with which to gauge relative performance, and the ensure CNCs understand what is expected of them to continue receiving GIA program support.

    Recommendations

    Recommendations and supporting rationale

    Determine selection criteria

    • In the absence of increased funding and given the rising costs of memberships and inflation, it is necessary for the GIA program to determine how to decrease its spending to meet its budget. The current mix of supported organizations is the result of a bottom‑up process, rather than any recent strategic selection. Historically, CNCs approached the NRC for funding to affiliate with an international organization.
    • The program has the opportunity, in consultation with the new steering committee, to play a greater role in the selection of which affiliations to continue supporting and which, if any, other affiliations could be supported.
    • The steering committee has proposed a subcommittee to determine selection criteria, which would guide whether to add or end specific affiliations.

    Recommendation 1

    It is recommended that the International Relations Office (IRO), in consultation with the Canadian Committee for International Scientific Affiliations (CCISA), determine selection criteria for which affiliations it supports and to what extent if tiered memberships are offered. Criteria for consideration should include:

    1. strategic alignment with Canada's priorities in international science
    2. involvement of Canadians in leadership positions and participation in committees
    3. demonstrated contributions by Canadians to the advancement of knowledge and/or area(s) where Canada may want to advance its knowledge through international engagement

    Standardize annual performance reviews

    • Over the evaluation period (2015‑16 to 2019‑20), the APR process was conducted each FY for a total of five times. Most years some questions on the APR survey changed or were omitted.
    • The timing of when the APR process was administered varied year by year. Additionally, in FY 2018 the APR was only administered to three CNCs. This did reduce administrative burden on other CNCs that year, however it made it more difficult to track performance and eliminated one of the few regular touchpoints between CNCs and the GIA program.
    • Standardizing the process will enable more accurate performance tracking year‑over‑year, support the use of selection criteria for affiliations and decision‑making, and facilitate CNCs' ability to plan for and provide accurate information.

    Recommendation 2

    It is the recommended that the IRO standardize the Annual Performance Review (APR) process, including consistent phrasing of questions, timing, and frequency. Questions should align with Canada's priorities in science, and with the selection criteria to be developed.

    Management response and action plan

    Table 4
    Recommendation Risk‑level associated with not addressing recommendation Management response Measure of achievements Proposed person(s) responsible Expected date of completion

    1. It is recommended that the International Relations Office (IRO), in consultation with the Canadian Committee for International Scientific Affiliations (CCISA), determine selection criteria for which affiliations it supports and to what extent if tiered memberships are offered.

    Criteria for consideration should include:

    1. strategic alignment with Canada's priorities in international science
    2. involvement of Canadians in leadership positions and participation in committees
    3. demonstrated contributions by Canadians to the advancement of knowledge and/or area(s) where Canada may want to advance its knowledge through international engagement
    High

    Response: Accepted.

    Action: CCISA sub‑committee to be established to determine selection criteria in alignment with three points above and advise CCISA on potential next steps.

    GIA program to draft selection criteria Melanie Cullins, Executive Director, International Innovation Office June 30, 2021
    Establishment of CCISA sub‑committee June 30, 2021
    CCISA finalizes selection criteria June 30, 2022
    2. It is the recommended that the IRO standardize the Annual Performance Review (APR) process, including consistent phrasing of questions, timing, and frequency. Questions should align with Canada's priorities in science, and with the selection criteria to be developed. Medium

    Response: Accepted.

    Action: APR questionnaire will be reviewed and revised to align with the selection criteria for affiliations to be determined as part of Recommendation 1 above. The standardized APR process will be carried out in a consistent manner over the following years.

    APR sent out in January 2021, restoring regular schedule Melanie Cullins, Executive Director, International Innovation Office January 2021
    APR sent out in January 2022 using draft selection criteria January 2022
    Beginning January 2023, APR administered using finalized selection criteria January 2023

    Appendices

    Appendix A – Methodology

    Data Review

    GIA program administrative and performance data for 2015‑16 to 2019‑20 were reviewed to provide information on program inputs (i.e., resources), outputs, and client reach. This included financial data, and annual performance review (APR) data collected from the CNCs (available for 2016, 2018 and 2019).

    Key informant interviews

    Interviews were conducted with 19 stakeholders (6 internal and 13 external) to collect information such as personal experiences, opinions and expert knowledge related to the relevance and performance of IA. This includes interviews with current NRC employees (n=5) and former program staff (n=1). Interviews with the ISC (n=1), other interdisciplinary bodies (n=3) and CNC's to supported international bodies (n=9) provided an external review of IA.

    This information was used to complement other lines of evidence and to contextualize quantitative information.

    Document review and environment scan

    Internal and external documents were reviewed to provide context and to complement other lines of evidence in assessing relevance and performance. Internal documents included previous GIA program evaluations and MRAP status reports, documentation for other NRC transfer payment programs, steering committee meeting minutes, steering committee member reports and draft terms of reference for CCISA.

    External documents included ISC public statements and an environmental scan and report completed by Library and Information Management (LIMS). The LIMS report provided external evidence on international science organizations comparable to the ISC, and an overview of the science‑related fora that Canadian OGDs and other national science organizations are affiliated with.

    Limitations and mitigation strategies

    Although the evaluation encountered some challenges, methodological limitations were mitigated, where possible, through the use of multiple lines of evidence and the triangulation of data. This approach was taken in order to establish the reliability and validity of the findings and to ensure that conclusions and recommendations were based on objective and documented evidence. Details on limitations and their associated mitigation strategies are described below.

    Issues with APR data
    • APR data provided was inconsistent as several different versions were used between 2015‑2016 and 2019‑2020. This led to difficulties with comparing information and interpreting results.
    • To mitigate this limitation, OAE sought out consistent indicators and worked closely with the program to ensure correct interpretations of data. It is recommended that GIA standardize the collection of APR data to prevent the recurrence of this limitation in future evaluations.
    Issues with obtaining annual administrative costs
    • Due to the small size of the program, NRC staff do not actively track time resources spent on GIA. For this reason, NRC‑Evaluation was unable to obtain true administration costs for the evaluation period.
    • OAE was unable to obtain administrative costs from staff at SSHRC who co‑chair the program.
    • To mitigate this limitation, NRC staff provided time estimates which were used to calculate estimated salary totals. OAE also consulted with former GIA personnel to establish a cost baseline.
    Limited corporate memory due to turnover
    • Both IRO and CNCs were subject to significant turnover during the evaluation period. IRO staff directly responsible for the program had been with the NRC for approximately one year. At CNCs, roles are generally term based so turnover is common and representatives often only had direct experience with a portion of the evaluation period.
    • To mitigate this limitation, OAE interviewed past office holders. This included the GIA's former program advisor, who was present for most of the evaluation period. OAE also interviewed senior management who had longer experience with the program. For CNCs, in some instances OAE sought to interview past CNC representatives (e.g., past presidents) who had had more familiarity with the GIA's history and changes.

    Appendix B – Supported international scientific organizations and their Canadian counterparts

    Appendix B – Supported international scientific organizations and their Canadian counterparts
    Supported international scientific organization Canadian National Committee Partner Organization Affiliated with ISC? Natural or Social Science
    International Commission on Illumination (CIE) CNC‑CIE National Research Council of Canada Yes Natural
    Committee on Data for Science and Technology (CODATA) CNC‑CODATA National Research Council of Canada Yes Natural
    Committee on Space Research (COSPAR) CNC‑COSPAR Canadian Space Agency Yes Natural
    International Association for the Properties of Water and Steam (IAPWS) CNC‑IAPWS CANDU Owners Group No Natural
    International Astronomical Union (IAU) CNC‑IAU Canadian Astronomical Society Yes Natural
    International Geographical Union (IGU) CNC‑IGU Canadian Association of Geographers Yes Mix
    International Mathematical Union (IMU) CNC‑IMU Canadian Mathematical Society Yes Natural
    International Union for Quaternary Research (INQUA) CANQUA‑INQUA Canadian Quaternary Association Yes Natural
    International Permafrost Association (IPA) CNC‑IPA Geological Survey of Canada, Natural Resources Canada No Natural
    International Society for Soil Mechanics and Geotechnical Engineering (ISSMGE) CGS‑ISSMGE Canadian Geotechnical Society No Natural
    International Union of Biochemistry and Molecular Biology (IUBMB) CSMB‑IUBMB Canadian Society for Molecular Biosciences No Natural
    International Union of Crystallography (IUCr) CNC‑IUCr Canadian Light Source Yes Natural
    International Union of Geodesy and Geophysics (IUGG) CNC‑IUGG Canadian Geophysical Union Yes Natural
    International Geological Sciences (IUGS) CFES‑IUGG Canadian Federation of Earth Sciences Yes Natural
    International Union of History and Philosophy of Science and Technology (IUPS)* CNC‑IUPS Canadian Society for the History and Philosophy of Science Yes Social
    International Union of Nutritional Sciences (IUNS) CNS‑IUNS Canadian Nutrition Society Yes Natural
    International Union of Pure and Applied Biophysics (IUPAB) BSC‑IUPAB Biophysical Society of Canada Yes Natural
    International Union of Pure and Applied Chemistry (IUPAC) CNC‑IUPAC National Research Council Yes Natural
    International Union of Pure and Applied Physics (IUPAP) CNC‑IUPAP National Research Council of Canada Yes Natural
    International Union of Basic and Clinical Pharmacology (IUPHAR) CNC‑IUPHAR Canadian Society of Pharmacology and Therapeutics Yes Natural
    International Union of Physiological Sciences (IUPS) CNC‑IUPS Canadian Physiological Society Yes Natural
    International Union of Psychological Science (IUPsyS) CNC‑IUPsyS Canadian Psychological Association Yes Social
    International Union of Theoretical and Applied Mechanics (IUTAM) CNC‑IUTAM Canadian Society for Mechanical Engineering Yes Natural
    Scientific Committee on Oceanic Research (SCOR) CNC‑SCOR Department of Fisheries and Oceans Canada Yes Natural
    Scientific Committee on Solar‑Terrestrial Physics (SCOSTEP) CNC‑SCOTSTEP Canadian Space Agency Yes Natural
    International Union of Radio Science (URSI) CNC‑URSI National Research Council Yes Natural
    World Climate Research Programme (WCRP) CNC‑WCRP Environment and Climate Change Canada Yes Natural
    *IUPS has distinct branches (IUPS‑DHST and IUPS‑DLMPST), each of which collect their own dues. They are counted as two supported organizations

    Appendix C – Comparing umbrella organizations for international scientific organizations

    Appendix C – Comparing umbrella organizations for international scientific organizations
    Organization Number of member organizations or institutes Member characteristics Primary organizational objectives
    International Science Council (ISC)
    • 135 member organizations
    • 40 member unions and associations
    • 30 affiliated members
    • Natural and social science unions/associations
    • National and regional scientific organizations and academies
    • Research councils, institutes and foundation
    • Promotes "Science‑for‑Policy": supports international scientific research, promotes researchers from marginalized groups and contributes to international policy issues.
    • Promotes "Policy‑for‑Science": contributes to scientific policy development in addressing major global challenges
    • Promotes "Free Science": promotes free and responsible scientific practice
    InterAcademy Partnership (IAP)
    • 140 national and regional member academies of science, engineering, and medicine
    • Academies of science, medicine and engineering
    • Capacity building (e.g., for new and less experienced member academies)
    • Science advice
    • Education
    Belmont Forum
    • 25 member countries
    • 29 member organizations
    • Funding organizations
    • International science entities
    • Regional consortia
    • Advancement of transdisciplinary science that focuses on understanding, mitigation and adoption knowledge on environmental change
    Global Research Council (GRC)
    • 45 member countries
    • Science and engineering funding agencies
    • Improve co‑operation among funding agencies
    • Promote data and process sharing among funding agencies