Bias in Peer Review – A Training Module

Introduction

Welcome to bias in peer review, a training module for peer reviewers.

This interactive module is designed to promote an understanding of unconscious bias and how it can affect the peer review process. It will also provide strategies for mitigating bias during the review process.

Objectives

By the end of this course, you will be able to:

• understand what unconscious bias is;
• understand how unconscious bias can impact the peer review process; and
• integrate methods for mitigating the influence of unconscious bias.

Objective 1

In this section, we will explore the difference between explicit bias and implicit (or unconscious) bias.

How biases are formed ^{Footnote 1}

Biases are shortcuts our brain forms based on:

• our own culture;
• experiences;
• things other people tell us;
• institutional influences; and
• other external influences, such as social media.

When faced with situations or people, we use mental maps and patterns to classify them by making a number of automatic associations. Not surprisingly, our perceptions and assumptions based on these automatic associations are not always correct.

Because our unconscious biases tend to be ingrained, it takes some work to disrupt them, but it can be done through active reflection and practicing inclusive behaviours. Doing this work benefits us, the people around us and the peer review process; it also contributes to research excellence.

Consider the following scenario:

You are out with friends and see someone wearing a sports jersey from your favourite team. You decide that you like them and decide to have a friendly conversation with them.

In this situation, you are consciously aware of the reason that you like that person. You make a deliberate decision to talk to them and you are aware of the process by which you reached that decision. This is an example of explicit bias.

What is explicit bias? ^{Footnote 2}

Explicit bias is a result of conscious thought and can be deliberately regulated. People are more motivated to control their biases if there are social norms in place, which dictate that prejudice is not socially acceptable.

Every peer reviewer is asked to sign a Conflict of Interest Agreement, which states that a conflict may be deemed to exist if reviewers feel for any reason unable to provide an impartial review of the application.

Explicit bias is an example of being unable to provide an impartial review, and is therefore a conflict of interest. For this reason, explicit bias is not explored in this module.

Consider the following scenario:

You are out with friends and see someone at the table beside you. The person gives you an uneasy feeling and you feel uncomfortable. Your friend notices and asks you what is wrong, but you aren’t sure. You cannot describe why you feel that way.

In this situation the person at the table beside you might remind you of someone from your past with whom you had a negative experience. You are not aware of this, and the feelings are unconscious.

This is an example of unconscious bias. The feeling was involuntarily formed and you cannot explain why you feel that way. The feeling is caused by a past experience and you were unaware of it until you felt uncomfortable.

What is unconscious bias? ^{Footnote 1 Footnote 3 Footnote 4 Footnote 5}

An unconscious bias is an implicit attitude, stereotype, motivation or assumption that can occur without one’s knowledge, control or intention. Unconscious bias is a result of our life experiences and affects all types of people. Examples of unconscious bias include gender bias, cultural bias, age bias, language and institutional bias.

Unconscious biases are important to recognize in instances when quality, relevance and competence are being evaluated. Examples of these instances include peer review of grants, search committees, reference letters and peer review of manuscripts.

How does unconscious bias work?

Unconscious bias affects our judgment without us realizing.

For example, these tables appear to be different sizes. We think they are different just by looking at them; however, in reality, they are the same size.

This is an example of how our brains form shortcuts, and of how unconscious bias occurs.

Effects of bias ^{Footnote 6 Footnote 7}

Unconscious bias may not significantly affect every single interaction or decision but it can have lasting consequences. Decisions made based on unconscious bias can compound over time to significantly impact the lives and opportunities of others who are affected by the decisions we make.

Small biases in peer review of career boosting resources (like grants and journal publications) can make a big difference in the long-term career of an investigator.

In this example, a computer simulation studied the effects of a one per cent bias and found that cycle had an impact on how the blues and oranges were represented over time, and ultimately the representation of each colour drastically changed as a result of this unconscious bias. It is therefore very important to mitigate unconscious bias in peer review.

Biased indicators of excellence ^{Footnote 20 Footnote 21 Footnote 22 Footnote 23 Footnote 24 Footnote 25}

The granting agencies fund excellent research and researchers, but what if the indicators of excellence used in evaluation of candidates are biased?  Many reviewers may assume that they are applying an objective standard of excellence when they look at three traditional metrics:

• years of experience;
• number of publications and citations; and
• size of a candidate’s research grants.

However, it has been shown that all three of these metrics are subject to unconscious bias.

Did you know?

Research has shown that underrepresented groups are more likely to face systemic barriers (such as more expectation to sit on committees) that limit their lab time, result in less research funding and contend with the view that their publications are less significant.

In addition, women face less integration into the scientific community, for example, by holding positions and memberships in scientific associations and on the editorial boards of journals. These barriers have resulted in women publishing 20 per cent fewer publications than men in the natural sciences, despite being equally qualified.

This bias has extended to the peer review process, where female first authors suggested female reviewers 21 per cent of the time, while men suggested women just 15 per cent of the time. In addition, men cite their own papers at a 35 per cent higher rate than women.

When looking at aggregated data, it has also been shown that women self-promote their own works at a lower level and their publications receive lower recognition (citations) from the scientific community despite being published in journals with higher impact factors.

Unconscious bias is also a key factor when evaluating the productivity of researchers with career leaves and slowdowns. Members of underrepresented populations are more likely to face career interruptions because of parental, family or medical reasons. For example, recent newcomers to Canada may need leaves or face slowdowns in order to address immigration and settlement issues.

This can understandably have an effect of the productivity of the researcher, meaning two equally qualified candidates may not have the same number of publications, but this does not make the candidate with a career leave any less excellent. Therefore, relying on publications as an indicator of excellence without taking into consideration the impact on productivity of the leaves unduly disadvantages researchers with career leaves.

For example, a recent study found that after a parental leave, male economists had a 19 percentage-point rise in the probability of tenure in their first job. In contrast, women’s chances of gaining tenure fell by 22 percentage points. This is because men are more able to take advantage of this leave to publish in top-tier journals. In short, not taking gender into account has the consequence of raising the bar for tenure even further for women.

Despite this, there are many studies that show that greater diversity leads to better research. In other words, the research enterprise won’t reach its full potential for excellence unless it is diverse. This underscores the need to use unbiased indicators of excellence in the peer review process.

Bias in peer review

There are many types of unconscious bias that affect our daily lives, and this is no different in the review process. There are, however, types of unconscious bias that are more likely to appear in the review process.

These include gender bias, institutional bias, age bias, culture bias (including Indigenous bias and geographic bias) and language bias.

Bias can be influenced by the characteristics of both the reviewer and the applicant, as well as the nature of the application and where the research is being conducted and by whom.

It is important to note that bias may not only be towards someone with different characteristics than you; bias can also occur towards someone with the same characteristics as you. For example, female reviewers can be just as unconsciously biased against female applicants as male reviewers.

Objective 2

In this section, we will explore different types of unconscious bias and how they can impact the peer review process.

What is gender bias? ^{Footnote 10}

Unconscious gender bias has been defined as the differential treatment of men and women, the impact of which may be positive, negative or neutral.

Evidence of gender bias by linguistic analysis ^{Footnote 11 Footnote 12}

In a 2003 study in the United States, academic linguists reviewed 312 letters of recommendation for faculty hired at a major medical school.

Letters written for female applicants were shorter and less focused on the candidate’s record of accomplishment; they used more gendered terms such as “intelligent young lady” or “insightful young woman.” Letters for women included more grindstone adjectives such as hardworking, conscientious, dependable, thorough, dedicated, careful or meticulous, whereas letters for male applicants included more standout adjectives such as excellent, superb, outstanding or unique.

This finding suggests that women’s success is more often associated with effort while men’s success is associated with ability.

Letters written for female applicants included more doubt raisers such as “she worked hard on the project that she accepted,” and were significantly more likely to have references to one’s personal life than those written for men.

In contrast, letters written for male applicants were more likely to have references to their CV, publications or patents.

It is therefore important to evaluate each candidate’s entire application, and not rely too heavily on only one element.

Institutional bias ^{Footnote 13 Footnote 14}

Another common bias in peer review is institutional bias. Institutional bias refers to bias that occurs due to the reputation of the institution the size, type, or location of the institution or prior research conducted by that institution.

Prestige bias and affiliation bias can influence institutional bias as well—either due to the status of the institution or due to a reviewer’s current or previous affiliation with the institution.

Examples of institutional bias from reviews have focused on the availability of resources, the size of the institution and collaboration with other institutions.

Read the application and then answer the question:

A research proposal has been submitted by a principal investigator at a small institution with limited access to resources. The research proposal will study broadband internet systems in rural locations across Canada.

Which of the following is an unbiased response to the research proposal?

1. Given the size of the institution, the applicant has limited training experience and is unsuitable for this grant.
2. The institution does not have a viable research capacity due to its size and will not be able to conduct the research proposed successfully.
3. The affiliated institution lacks resources needed to support this research and the applicant has not secured these resources elsewhere.
4. The institution should not support a grant because of its size and limited capacity.

Age bias

Age bias refers to bias that occurs due to the age of the applicant and can be directed to people who are older or younger.

Read the application and then answer the question:

The principal applicant of an application is someone that you know is thinking of retiring within the next couple of years. The application specifies a grant duration of five years.

In this situation, the possible retirement of the principal investigator is reason to reject the application

1. True
2. False

Indigenous bias

Indigenous bias can be experienced in a variety of ways including applying (explicitly or unconsciously) assumptions on the quality, merit, value, relevance, importance, success and competence of either an Indigenous or non-Indigenous allied scholar, or Indigenous research methodologies, based on racial, historical, and cultural biases and stereotypes.

This is critical to be cognizant of when reviewing any works involving First Nations, Inuit, or Métis individuals (and/or communities).

Historically, the Indigenous peoples of Canada have been marginalized, disadvantaged, and excluded from all mainstream opportunities, through continued systemic barriers, assumptions, and biases.

There is a growing movement to create an equitable, fair relationship; these longstanding biases are slowly being challenged and addressed.

With the federal acceptance of the Final Report of the Truth and Reconciliation Commission and the signing of the UN Declaration on the Rights of Indigenous Peoples, a new respectful relationship can be built among Indigenous peoples and Canada, confronting the nation’s old biases.

It is therefore of utmost importance, through the peer review process, that the reviewer  have the knowledge, expertise, openness, and humility to be able to fairly assess an application based on Indigenous Ways of Knowing or using Indigenous research methodologies.

The reviewer must have respect for, knowledge of, and uphold the TCPS-2 Chapter 9 guidelines in reviewing an application involving Indigenous Peoples.

Language bias ^{Footnote 26 Footnote 27 Footnote 28}

Canada’s linguistic duality can give rise to bias during peer reviews, in particular for francophones. First of all, scientific culture not only in Canada but also throughout the world increasingly places greater value on research conducted and published in English.

This is against the principle of respect for Canada’s linguistic duality, as well as the principles of transferring knowledge to and exchanging knowledge with local populations.

Furthermore, if French applications are reviewed by committee members who do not have excellent French skills, their understanding of the proposal may be impacted. This may affect their confidence in their comprehension and assessment, and subsequently implicit preference to review in their own first language. Applicants and reviewers may also experience concern (or stereotype threat) with their second language skills being evaluated by their peers, and this can reduce their performance.

Research also shows that one cause of linguistic bias is “editors and reviewers having little awareness of and sensitivity to the challenges of nonnative English-speaking contributors in writing in English for publication.

Reviewers’ assessments may be impacted by their impression of what constitutes well-written applications if the quality of the writing does not meet their expectations.

Objective 3

In this section, we will learn about how to integrate methods for mitigating the influence of unconscious bias.

Identifying bias ^{Footnote 15 Footnote 16}

It is important to take steps to mitigate bias. In order to do so, you must first recognize that you have biases.

Identify what those biases are and take steps to mitigate bias in your thought process. The following will offer you some tools to help mitigate bias.

Implicit Association Test

Please take some time to do the Implicit Association Test.

The Implicit Association Test measures the strength of associations between concepts and evaluations or stereotypes. The main idea is that making a response is easier when closely related items share the same response key.

Once you have identified potential sources of unconscious bias, continue reading to learn how to mitigate biases.

How to mitigate bias ^{Footnote 17 Footnote 18 Footnote 19 Footnote 12}

The steps below are evidence based behavioural strategies that can be practiced while reading an application to mitigate bias in the review process.

• Stereotype replacement: Think about a stereotype that you hold and consciously replace it with accurate information. 
• Positive counterstereotype imaging: Picture someone who counterstereotypically fills a traditionally stereotyped role.
• Perspective taking: Take the perspective of someone in a stereotyped group (example below).
• Individuation: Gather specific information about an applicant to prevent group stereotypes from leading to potentially inaccurate assumptions (example below).   

The following scenario is an example of perspective taking:

Before evaluating a research proposal written by a member of a minority group, imagine in detail what it is like to be a person in a stereotyped group. For example, imagine what it is like to be an Indigenous researcher and to be questioned on the validity of Indigenous research methodologies, based on racial, historical, and cultural biases and stereotypes.

The following scenario is an example of individuation:

A late career female investigator has submitted a research proposal. While reviewing her application, make sure you gather specific information about the applicant in order to prevent group stereotypes from leading to potentially inaccurate assumptions. Use the adjudication criteria and sub-criteria outlined in the peer review manual to gather the facts and information about the applicant.

Additional tips ^{Footnote 12}

Here are some additional tips for minimizing the influence of bias and assumptions.

Spend sufficient time evaluating each applicant. Studies have shown that evaluators who were busy or distracted by other tasks gave women lower scores than men for the same written evaluation of job performance.

Apply the criteria consistently to all applicants. Research shows that different standards may be used to evaluate applicants of different genders.

Evaluate each candidate’s entire application. Don't rely too heavily on only one element of the application to evaluate an applicant—recall the linguistic analysis that revealed unconscious gender bias.

Periodically evaluate your judgments and consider whether evaluation biases are influencing your decisions; ask yourself questions such as:

• Are underrepresented candidates subject to different expectations or standards in order to be considered as qualified as the majority?
• Is research from smaller institutions or minority groups being undervalued?
• Have accomplishments, ideas or findings of underrepresented candidates been unfairly attributed to research directors or collaborators despite evidence to the contrary?

Conclusion

Being aware of unconscious bias is the first step in mitigating these sources of bias. By learning about unconscious bias, doing the Implicit Association Test and using the additional resources provided, you can review applications in a manner that is more conscious, fair and avoids unequal outcomes.

Greater awareness of unconscious bias will lead to an improved review process and greater parity in grant and award distribution.

Survey

Before concluding this module, please complete a survey to assist the Natural Sciences and Engineering Research Council (NSERC) and the Social Sciences and Humanities Research Council (SSHRC) in tracking the uptake, and improving the quality, of the learning.

Congratulations, you have now completed the learning on unconscious bias. You should now be able to understand what unconscious bias is, understand how unconscious bias can impact the peer review process and integrate methods for mitigating the influence of unconscious bias in the review process.

Acknowledgments

A special thanks to all those at the Canadian Institutes of Health Research who developed the original version of this module. Thanks in particular to:

• the CIHR Institute of Gender and Health, and Knowledge Translation program at the Li Ka Shing Knowledge Institute at St. Michael’s Hospital in Toronto, Ontario, for their development of the information relevant to gender bias;
• the CIHR Institute of Aboriginal Peoples’ Health, for their development of information relevant to Indigenous bias;
• the Association des collèges et universités de la francophonie canadienne, for their development of information relevant to language bias; and
• the many people who reviewed the content, including (but not limited to) Dr. Carole Lee (University of Washington), Dr. Molly Carnes (University of Wisconsin) and Dr. Calvin Lai (Harvard University).

Resources

Training resources:

• Association of American Medical Colleges Unconscious Bias Training
• The Canada Research Chair Program’s Equity, Diversity and Inclusion Practices
• Canada Research Chairs Program, Limiting Unconscious Bias
• CIHR Peer Review Manual for Grant Applications, 2016
• How to Survive Peer Review, BMJ Books
• Implicit Association Test
• NSERC’s equity, diversity and inclusion webpage
• SSHRC’s Manual for Adjudication Committee Members 2016-17
• SSHRC’s Guidelines for the Merit Review of Aboriginal Research
• Tri-Council Policy Statement on Research Involving the First Nations, Inuit and Métis Peoples of Canada

Additional online/training resources:

• Bornmann, Lutz, Rüdiger Mutz and Hans-Dieter Daniel (2012), “Does gender matter in grant peer review?” Zeitschrift für Psychologie, Vol. 220(2), pp. 121-129.
• Bornmann, Lutz, Rüdiger Mutz and Hans-Dieter Daniel, Gender differences in grant peer review: A meta-analysis.
• Bornmann, Lutz and Hans-Dieter Daniel (2004), Reliability, fairness and predictive validity of committee peer review, B.I.F Futura, Vol. 19, pp. 7-19.
• CIA Diversity and Inclusion Strategy (2016-2019)
• Cook Ross Inc., Proven Strategies for Addressing Unconscious Bias in the Workplace.
• Council of Canadian Academies (2012), Strengthening Canada’s Research Capacity: The Gender Dimension.
• Drummond, Rennie, Annette Flanagin and Fiona Godlee (2015), The Eighth International Congress on Peer Review and Biomedical Publication: A Call for Research. The JAMA Network, 313(20):2031-2032. doi:10.1001/jama.2015.4665.
• Emerson, Gwendolyn B., et al. (2010), Testing for the Presence of Positive-Outcome Bias in Peer Review: A Randomized Controlled Trial, Arch Intern Med, 170(21):1934-1939. doi:10.1001/archinternmed.2010.406.
• Facebook, Managing Unconscious Bias.
• Garfunkel, Joseph M., et al. (1994), Effect of Institutional Prestige on Reviewers' Recommendations and Editorial Decisions, The JAMA Network 272(2):137-138. doi:10.1001/jama.1994.03520020063017.
• Gender Bias Learning Project
• Hames, Irene (2007), Peer Review and Manuscript Management in Scientific Journals, doi: 10.1002/9780470750803.
• Hopewell, Sally, et al. (2014), Impact of peer review on reports of randomised trials published in open peer review journals: retrospective before and after study, BMJ Books, 349: g4145.
• Kaatz, Anna, Belinda Gutierrez and Molly Carnes (2014), Threats to objectivity in peer review: the case of gender, Trends in pharmacological sciences, 35(8):371-373. doi:10.1016/j.tips.2014.06.005.
• Leading Women, Unconscious Bias at the Office – Does Your Company Have a Plan?
• Luukkonen, T. (2012), “Conservatism and risk-taking in peer review: Emerging ERC practices,” Research Evaluation, 21:48–60.
• Miller, David I., Alice H. Eagly and Marcia C. Linn (August 2015), “Women’s representation in science predicts national gender-science stereotypes: Evidence from 66 nations,” Journal of Educational Psychology, Vol. 107(3), 631-644.
• National Collaborating Centre for Aboriginal Health — Aboriginal Research Designs
• Science Magazine (2015), A little bias in peer review scores can translate into big money, simulation finds.
• Smith, Richard (2006), “Peer review: a flawed process at the heart of science and journals,” Journal of the Royal Society of Medicine, 99(4):178-182.
• Unconscious Bias — Global Diversity Practice

Works cited