#openscience and peer review: write it like you plan to sign it (even if you don’t)

I submitted my first big federal grant proposal last week! Yay for me!1 It’s been taking up most of my time since I got back from maternity leave, and has resulted in a small pile of half-written blog posts saved as drafts, hopefully to see the light of day some day.2

After I submitted, I rubbed my eyes, shook my head, and set to work on a handful of manuscript reviews that had come in while I was focussed on the grant. And in the spirit of moving towards open science, I decided to sign some of the reviews- which was not easy.

For those of you not overly familiar with the academic peer review process, traditionally, when you submit a manuscript to a journal, an editor will assess the MS for basic quality and subject area, and then identify 2-4 scientists with expertise in that subject area and contact them individually to see if they would be willing to review the manuscript. If the prospective reviewer agrees, they are given some time period in which they are expected to complete a review. Precisely what the reviewer assesses the manuscript for can vary with the journal, but all reviews are supposed to evaluate the manuscript for quality of science, interpretation, and communication. In traditional ‘blind’ peer review, the reviewers act anonymously- that is, the reviewer sends the review back to the editor, who removes any identifying information, synthesizes the reviews, and then returns them to the authors with a decision- the authors never know who reviewed their work.

There are good things and bad things about this system of anonymous review. Reviewers are on the front lines- they’re essentially the gatekeepers of science- the ones who make the decision- will the manuscript make it to the literature, the permanent scientific record, or will it fade into obscurity? This is a serious job.3 Anonymity serves to protect the reviewer, to allow them to be as critical as necessary without fear of retribution from authors. This can be particularly important when the power differential is great– imagine a plucky young grad student finding a major flaw in the work of an old, established scientist.

But reviewers are human, too, with biases, positions, conflicts of interests, and sometimes they fall asleep at the wheel.4. Science doesn’t benefit from this- in fact, it can be outright dangerous when the gatekeepers don’t do their job in earnest. Anonymity also has the tendency to bring out the nastiest side of people5– you’re on the internet, you, gentle reader, have probably seen this. This is bad for authors, but there are aspects of anonymous peer review that are bad for reviewers, too. Reviewing a paper can be a thankless task- one that is both time consuming and counts for very little in the metrics we academics use to evaluate ourselves- because there’s no real proof of contribution.

An alternative to blind review which can alleviate some of these issues is open review. Open review is nearly identical to the traditional process, except the identity of the reviewers are made known, and often, the content of the reviews are also published alongside the paper. In general, when reviewers know they won’t be anonymous, they produce higher quality, more constructive reviews. However, it’s also more difficult to find reviwers using the open model, probably due to the protections anonymity offers. PeerJ is one journal that uses this model for peer review.

As part of my self-challenge to be as open as possible, as much of the time as possible, I decided to sign a couple of the reviews I’ve done recently. I also consciously decided *not* to sign a couple of other reviews I’ve done recently. So, as you can see, I’m on the fence about this whole issue. I’ll try to unpack it all here.

I am a junior scientist. A woman. A postdoc on contract. A foreigner. The sole provider for a family of four. My situation is, as far as scientific career stages go, precarious. Crossing someone who may, in the future, decide my fate, with a harsh review, is clearly not in my best interests, even if the review is completely justified. However, I do benefit from open review, in increased visibility and increased credibility6. Also, as an author and an advocate, I want to help foster a scientific community that’s more open in general.

So how do I decide to sign a review, or not? I think it comes down to two interacting factors for me- the culture of the journal, and the quality of the paper I’m reviewing. I’m much more likely to sign a positive review.7 It’s also pretty easy to decide to *not* sign a negative review- those papers where you just can’t see anything salvageable because their experimental design was crap, or the writing style is so bad you can’t even tell what they did, or even worse, you uncover some kind of misconduct and call them on it. Clearly, there is going to be a difference of opinion between you and the authors, and that’s going to breed a sense of hostility that doesn’t serve the author or reviewer to put a name to. But the ones in between, that’s where it’s harder, and the journal culture is what tends to tip things one way or the other. I always, always always try to offer constructive feedback in my reviews, but sometimes, even constructive criticism can feel like an attack. I know, because I’ve been on both sides of the review. It’s a difficult thing to balance, although I get a bit more comfortable with it each time I sign a review.

So, what I’m saying is, offering honest, constructive feedback is scary and potentially dangerous, and we can’t expect reviewers to always feel comfortable sticking their neck out. I don’t really have an answer to all this, so I’m going to end with this. Kindness, constructiveness, and thoroughness in peer review benefits all of us- regardless of whether you sign your review or not, write it like you plan to sign it.


1. So knock on wood, do an elaborate interpretive dance, drink a liter of a homeopathic remedy (it’s good to stay hydrated), conjure some positive vibes for me. I don’t know what the exact process beyond the point of submission, so I figure it can’t hurt to keep us all busy while we wait to hear the news. Who knows? Maybe the review panel is doing this, too.
2. Someday, soon hopefully, you will learn, in excruciating detail, why the Jenks natural breaks algorithm has become one of my (many) data presentation pet peeves. It’s not just sigfigs anymore, folks, though that’s still up there:

And also:

3. Maybe I romanticise, but when I review, I kind of feel like a GateKeeper of Truth. I may or may not wear my superhero cape while reviewing (please do not ask my officemate about this).
4. Would the dumbhead reviewers of the Wakefield MMR/Autism study please raise their hands? See? People can die when reviewers suck. Don’t suck as a reviewer.
5. Would the dumbhead reviewer who made *overtly* sexist comments in his review of a paper please stand up? See? We create a culture hostile to women and minorities when reviewers suck. Don’t suck as a reviewer.
6. “Gosh, Dr. B, that was an astute comment there over at PeerJ. Yes, we read it when we Googled you. You’re exactly the type of scientist we want in our department. Here are the keys to your new lab.”
7. “That was great! Gold stars everywhere! I especially liked the part where you solved world hunger.”

Advertisements

About cbahlai

Hi! I'm Christie and I'm an applied ecologist and postdoc in the midwestern US. I am an #otherpeoplesdata wrangler, stats enthusiast, and, of course, a bug counter. I cohabitate with five other vertebrates: one spouse, one preschooler, one teeny baby and two cats.
This entry was posted in Uncategorized and tagged , , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s