A few weeks ago, a paper came out about the fate of research papers in ecology and evolution (my field!) pre- and post-publication, comparing outcomes between male and female authors.

I want to focus on just one aspect of their nuanced analysis (you can read the whole paper here for free). In this section, the authors gathered citation data on over 100,000 papers published in 142 journals in our field.

They found a slight, but significant, difference in how often papers by men and by women got cited. On average, controlling for the impact factor (a proxy for quality[1]), papers by women accrued 2% less citations.

A hundred thousand papers! That’s a lot, and it’s why the authors could detect this small difference. The sample size allowed them to do a quite powerful analysis.

It also revealed some interesting interactions. For example, papers with female last authors (which often indicates seniority or leadership of the research group) were cited less than those with male last authors at high impact journals, but the effect was reversed at low-impact journals.

Because more people cite papers in high-impact journals, this meant that overall, women were cited less often. [2]

I guess this shouldn’t have been a surprise, but it’s not something I had seen data about before, and consequently something I hadn’t really thought about.

But it is something that matters. Like it or not, citations are a metric of success that is easy to measure, and therefore whether others cite your work is a good piece of evidence that you’re a valuable scientist when you’re up for a job position, tenure, or an award. It’s not great for women if there is bias preventing them from doing well on this metric. [3]

Sounds About Right

Like a lot of research and headlines about challenges facing women in science, this got me mad. I’m a feminist, and this stuff pisses me off.

I see the experiences of my female friends and colleagues, and see when they are treated differently than their male peers. Not always, of course, but enough to make a pattern of our own anecdotal experience.

Then there’s the fact that in my study topics, almost every giant in the field is a man. The ones that defined the discipline and published the equations in a top journal? Men. There are women there, doing great work, but they are less famous.

When data on various aspects of academic life backs up our experience it’s like, “yup, sounds about right.”

Academia is harsh on most people; it’s a place with a lot of rejection, high standards, low pay, job insecurity, and so many power dynamics. The fact that academic science is even harder on women is just not fair.

And that’s not getting into the more complex and devastating nature of the structural problems in academia, which are even worse for other minorities, especially minority women. This paper found that on average women are cited 2% less than men; how much less often are minority women cited?

Anyway, I read this paper, and I was mad.

What I Did Next

Being mad doesn’t accomplish all that much. I tried to think about what we, in our daily lives as scientists, could do to work against this problem.

As an individual early-career scientist, I can only do a very little bit about peer review outcomes or paper acceptances.

But citations? That’s different. I am always writing papers, and I am always citing others’ work. I realized that this was a small step I could take to try to contribute to supporting female scientists. It sounds trivial, but I can cite their work. Could we all do this?

I brought this idea to our lab group, and was curious what they would think.

Some colleagues immediately recognized that this was a problem, and something we don’t think about enough. There are a handful of big names in our subfield, and we mostly cite them over and over. But do we need to do that? Are their papers that we cite every time actually the most relevant? Maybe not. There is probably a lot of work being done around the world we could cite that instead, or at least in addition to the now-traditional canon.

Another common reaction was for someone to say that they don’t think about the gender of authors when they search for papers, read them, or cite them. Here it diverged: a few people said, “I don’t think about it and now I realize that I should.” Others said, “I don’t think about the gender of authors when I’m citing them, therefore I’m not part of this problem.”

I challenged this statement. Does that really mean you’re not part of the problem? Maybe not, I said – and if not, that’s great for you, good work! But instead of assuming that not consciously citing more papers by men means no harm is done, check your reference list on the paper you’re writing. What’s the author breakdown? Do you think this strategy is really working?

I don’t think I was popular for making this callout.

Rubber Meets the Road

As I mentioned in a tweet a few weeks ago, “caring means walking the walk.”

In my paper-reading project, I track the gender of the authors I read, and I found that I am reading fewer papers by women than by men. I was curious about what the ratio might be of the papers I end up citing.

I went through the references section of my current manuscript with a blue and a pink highlighter (I know, supporting stereotypes, etc., it was lazy). The results were not pretty.

I was frankly surprised at how few of the papers I had cited were by women. I knew it wouldn’t be 50/50 for a lot of reasons (discussed below), but I didn’t think it would be that bad.

I basically proved to myself that in order to cite women, you have to do it on purpose. Just not intentionally excluding them isn’t enough.

It’s like how colorblind policies don’t work. Not intentionally doing harm is not the same thing as not doing harm. As Evelyn Carter writes about claiming to be colorblind with respect to race, “if you ‘don’t see’ race, but you say you care about inclusion, how can you advance inclusion efforts that will effectively target communities of color?”

There is a lot of unconscious bias and systematic barriers that lead us to contribute to inequality. Working for equality and to recognize contributions made by women and minorities means actively working to overcome those unconscious biases and systematic barriers.

Shifting the Balance

After my discouraging experiment with the highlighters, I went through the paper and looked at the places I had cited different work. In some places, I was able to find a paper by a woman that I could cite instead – and often even one that supported my point even better than the paper I had cited originally.

That was the most delightful aspect of this task I had set out for myself: I discovered papers in my core area of study, that were by women and that I had never read. And they were really good and very interesting!

The point of reading and citing work by women isn’t just to check boxes and give women a fair shot at career metrics. The main reason is to do better science. Science is creative; it involves having ideas, being exposed to new things, thinking outside whatever box you’re in. Reading work by more different people will necessarily help that process.

I’m really glad that I found these new-to-me papers.

As I discussed with a different colleague later, a lot of this issue comes down in part to poor citation practice, which is endemic across academia. We cite something because everyone else cites it, even though maybe we haven’t read the whole paper. Or we cite something because it’s already in our reference library and we are in a hurry. I’m completely guilty of this, and I’m quite sure everyone else I work with is, too.

If we spent more time reading, and more time looking for and getting familiar with other work that’s related to what we’re doing, I think all of our papers would have much more diverse author lists – at least in terms of evenness and not being dominated by the famous people in our field.

Our papers might also just simply be better, because of all the ideas we would be having.

Don’t Worry, I’m Still Citing Men

After this citation overhaul, my reference list was still majority male first authors. You need to cite what is relevant, and in a lot of cases this work is by men. One reason is that over the history of ecology, there is a lot more work published by men than by women, especially the farther back you go. [4]

Plus, if you need to cite a classic paper, it doesn’t matter who it’s by. That’s the one you need to cite. That’s a second thing. [5]

And likewise, if you need to cite something about your specific study organism or system, there might be only a handful (or a few handfuls) of people in the world who publish on this very specialized niche. They are who they are. If you need to cite peer-reviewed literature, you may have limited choices, and you need to cite the best and/or most relevant work out of that array. [6]

So there are a lot of reasons you can’t just take your reference list and manipulate it towards 50/50 gender equality. I want to make it completely clear: I am not advocating for a departure from citing good and relevant science!

When I mentioned this idea to colleagues – that citing women was a seemingly small, basic thing that we could do in our everyday lives as scientists to make a difference in structural biases – some were deeply uncomfortable that if they sought out work by women, then they would have to leave out other papers.

Listen: there is so much research out there, it boggles the mind. It’s growing exponentially. It’s insane. And in no paper do we cite all the possibly important research on a topic. We’re going to leave things out anyway. It’s just that now, we seem to be structurally leaving out work by women. Again, to overcome this bias is going to require intentionality. This is not a problem that we can just hope will go away because we are good people and don’t mean to cause harm.

No matter what we do, we’re going to keep citing a lot of great research by men.

Tokenism?

Among women, there was another layer. Many of the women I talked to did not want to be cited just because they were women and someone needed to move their reference list towards gender parity. They wanted to be cited because someone genuinely thought their research was the best and most relevant.

And I get that. I was invited to give a talk once because the organizer was looking for a replacement female speaker after the originally-invited woman couldn’t participate. I was so excited for this opportunity, but at the same time it felt weird to get it explicitly because they were looking for a woman.

I’m not sure what to do with this concern, but I think it comes back to proper citation practice. Is your work relevant to the topic, and being cited correctly? Then you deserve to be cited. If papers by women are accepted less often and cited less often, then part of the reason you are not currently being cited might be simply because someone isn’t familiar with your work.

And The Inevitable Question, Does Quality Limit Equality

Finally, we had some conversations about whether this might be because women’s work just wasn’t as good as men’s.

It’s pretty hard to assess that (although the paper I started this blog post with tried to, by using journal impact factor as a covariate).

I have two thoughts. One is that I doubt it’s true. There was an interesting graphic and paper that went around Twitter – about economics, not ecology – showing that papers by women are better written than those by men, that women incorporate reviewer comments more often, and that they improve at presenting information over the course of their careers.

It’s provocative and I’m not sure if it’s also true in my field or not, but I believe it. I think we are culturally trained from a young age to try to please, and so we might be likely to try to pacify reviewers, and to make a revision extra perfect if it was rejected the first time around.

Also, in the sciences, women have to be better to be evaluated as being as qualified as men.

Second, about the content. If it is possible that the data and experiments presented by women are less strong, why might that be?

It is interesting to think about how structural problems would lead to this. Could it be because women get less funding to do their research, and thus might have less resources or support? Could it be that women are asked to do more departmental service and teaching, and have less time to do research?

If the research really is worse – which I’m not convinced it is, but there’s little way to objectively assess, at least not for a dataset of any size – this very well might be a result of the same structural issues that cause the citation patterns.

What Next

What do you think about this? Are there any other ways to work on this problem?

Notes

[1] Impact factor is not a perfect measure of journal quality. It is an estimate of how often work there gets cited, which is a traditional metric for how important it is. For any individual paper, the impact factor of the journal it is published in doesn’t say much about the quality of that paper. However, I think it was the best covariate the authors could find to control for the differences in quality between papers when comparing men’s and women’s outcomes. Also, for better or for worse scientists pay attention to impact factors, so it may affect citation practice even if it’s not an actual metric of “quality” per se.

[2] I have an interesting, harebrained idea about this: if papers by women are less likely to be accepted, do good papers with women as last authors end up in lower-impact journals? That could explain why the better-cited papers from those journals are by women. Totally spitballing here.

[3] Citations shouldn’t define one’s value as a scientist, colleague, or employee anyway, but… that’s a discussion for another day.

[4] Is it really though? Women probably contributed to many important ideas. They typed the manuscripts. Maybe they did some of the work. There are tons of cases in science where people had the same idea independently, and one person got famous, and the other didn’t. Sometimes both these people were white men. I’m guessing there’s a lot of times when some of these people were women, minorities, scientists from outside Europe and North America, and people who were/are otherwise excluded from the elite science community.

[5] Given what I just wrote, I think I – and we all – need to put more effort into finding papers that were written around the same time as the famous ones that “came up with” ideas, and represent contributions of the community-building of those ideas by other people.

[6] When you’re looking for very specific things, there might be a lot of important and relevant data in theses and lower-impact papers by students who did not continue in science. This is valuable literature to search for, and might expand what you think the contributions of women and minorities are, given that these people are less likely to continue in academic careers either by choice or by exclusion.