Sep 1, 2019 9:00 AM

Scientists Avoid Bias When They Know They're Being Tested

A new paper tracked possible bias among scientists awarding prestigious research positions.

When it comes to gender, science suffers from what has been called a "leaky pipeline." In some fields, like biology, women make up the majority of the individuals entering graduate school in the field. But at each successive career stage—post-doctoral fellowships, junior faculty, tenured faculty—the percentage of women drops. The situation is even worse in fields where women are in the minority at the graduate level.

It's difficult to figure out why so many women drop out of the career pipeline. Progressing through a research career is a struggle for everyone, and it can be tough to suss out subtle sources of bias that can make it harder for women to push through. It's possible to do statistical analyses of outcomes—say, how many women received a particular type of grant—but then it becomes hard to determine whether they were all equally qualified. It's possible to set up artificial test situations that are better controlled, but people's behavior changes when they know they're being tested.

That is precisely the message that's driven home by a new paper that looks at gender bias in the awarding of research positions in France. While the study has some significant limitations, it examined the outcomes in two situations: the year the test was announced and all the people on the award committees were aware of it, and the year after, when the same people returned but had probably not realized the test was being repeated.

Bias, overt and otherwise

In France, allocation of research positions at elite institutions is done through a two-step process. Candidates are first selected based on a set of predetermined criteria (such as number of publications and their impact). Those that clear this cut off are all highly qualified; at that point, a committee of scientists in their field looks at intangibles and other hard-to-quantify aspects of the applicants' research careers. A Canadian-French team of scientists received access to this process and was able to track the outcomes of the committee's decisions for two years.

To an extent. While there were over 400 committee members, information on how they voted for different candidates was kept confidential. All that could be determined was whether the committee as a whole had chosen them for a research position.

The first year the test was run, the researchers got all the committee members to take a standard test for what's called implicit bias. This is the sort of thing that people do without thinking, like unconsciously associating being a scientist with being male. While this is typically not a bias people are even aware of—it's common in women and among those who push for greater gender equality—it is associated with differences in women's participation in the sciences.

These tests showed that, when it comes to implicit bias, the French scientists were equivalent to France as a whole, in that there was a significant tendency to associate science success to being male.

But, ideally, scientists are a relatively rational bunch, and these were involved in a deliberative group activity. So, the hope is that at least some of them would be able to overcome this implicit bias. The authors' hypothesis is that, if people are thinking about the potential for bias, they'll be more likely to overcome their implicit biases.

The test for this was rather complicated. In the first year of the experiment, all the committee members were made aware the experiment was taking place. As such, everyone would be expected to be thinking about the possibility of bias and be able to overcome their implicit bias in favor of males. The year following, however, it was assumed that this knowledge had worn off and that defaulting to implicit bias would be common. The exception would be those groups of reviewers whose default state is to be aware of bias issues.

To get a measure of this last possibility, in the first year, everyone was asked to complete a survey in which they were asked about the gender disparities in science and were given the choice of possible reasons, ranging from things like the challenges of balancing work and family, personal choices, or a lack of ability. This was converted to a score that represented an awareness of some of the hurdles women face in the sciences.

So, their hypothesis was that, in the first year, everyone would be in a position to overcome their innate biases. In the second, only the groups that had a default awareness of the hurdles women face would be in a position to do so.

Analysis and limitations

The researchers figured out the gender mix in each committee's pool of candidates, then determined whether the gender mix of the successful candidates was consistent with the starting percentage. Overall, there was no evidence of significant gender differences in either year of the testing, suggesting that, as a whole, the committees did a good job of overcoming their biases.

Still, there were differences among the committees. Nearly half of them had a majority of members who felt that women's lack of progress in the sciences was influenced by gender discrimination. And, if that belief was combined with a high level of implicit bias, then these committees had the largest decline in selection of women in the second year of testing. In other words, if a high implicit bias is combined with a low sense that women face barriers to advancement, then the committee selected relatively fewer women in the second year of testing.

This supports the researchers' hypothesis that an awareness of the issues that women face in the sciences provides a degree of protection against the implicit biases that many of us have internalized. Fortunately, that's now true for a majority of the committees in France, which seems to be enough to prevent an overall bias from creeping into this selection process.

Or at least creeping in at a statistically significant level. As the researchers acknowledge, their study is very small, as it was forced to do evaluations at the committee level, rather than analyzing the individual choices of the 400-plus committee members. There are a lot of tests that could potentially provide us with more information, but the data was too limited to show a significant effect. Another issue they point out is correlation; while the data they see is consistent with their hypothesis, it doesn't demonstrate the causal link among these factors.

To an extent, the strongest thing the researchers demonstrate is how hard it is to get good data on this issue. If studies are done using surveys, then scientists will likely end up aware that they're part of a test and may be more conscious of their implicit biases. If it's done in a real-world context, as in this study, then the need to keep everything confidential can limit the data available.

Still, if the overall hypothesis holds up in any further tests, then it's a problem that's relatively easy to correct, as a simple reminder about implicit biases prior to things like grant and tenure reviews would go a long way toward correcting things.

This story originally appeared on Ars Technica.