Peer review is research’ most powerful instrument. Having your manuscript reviewed by independent researchers in your own field improves the odds that your published work is valid—and valuable. The drawback of this mechanism is that many researchers are often on reviewer duty; I find myself reviewing several papers a month. It’s not hard to imagine that sloppiness can creep in sometimes… And sadly, there are not a lot of ways to prevent this: reviews in the research community remain largely anonymous. This means that, if a reviewer has a bad day or doesn’t want to read a paper with their full attention, they cannot be held accountable for that. If you have written a grounded opinion, why don’t you put your name on it?

31 July 2014

Here is how peer-reviewed science usually works:

1. You write a paper or article and submit it.
2. A number of reviewers read your paper and write down their opinion.
3. You receive a decision, based on the combination of all reviews.

This decision (usually a variation on “accept” or “reject”) is accompanied by the original reviews. This allows you to understand why the decision was made and gives you pointers for improving your manuscript.

It seems like a fair process and it usually is—under the assumption that reviewers honestly try their very best to give full attention to the paper and its contents.

And that’s where the shoe pinches: reviewers lack the incentives to put a lot of effort into reviews. Sure, many reviewers take pride in their task and have a strong intrinsic motivation to thoroughly read a paper and write an in-depth critique. But unlike with scientific papers, there is no accountability if you do a half-baked job, because reviews generally are anonymous. That means you can skim through the paper, draft a quick review, and submit it—your review will not be reviewed in turn. The “worst” thing that could happen if you mess up is that you’re not invited to review next year (which you probably didn’t want anyway).

Perceived needs for anonymity

Anonymous peer review, as this practice is called, is considered the default style to conduct reviews. It’s hard to find official reasons why this is so popular, but the concerns I heard most often are:

• “Nameless reviews emphasize the objectivity of the process.”
• “If my name is on it, the paper’s authors will retaliate against me.”

Let me debunk that first myth: the objectivity of reviews is an illusion. Science is performed by humans, not by machines. There is no black, there is no white; perhaps there is gray, but we’ll never know how many shades of it. Sure, good research provides the knowledge and tools to reproduce it, but that doesn’t automatically make the conclusions of that paper correct. And even if a paper is completely scientifically accurate, that still doesn’t make it relevant or important. As such, one reviewer might see a certain paper as a highly valuable contribution, while another reviewer considers that same paper rubbish. And they can both be right.

Multiple reviews should average this effect out, but sometimes agreement is simply not possible. Maybe none of the reviewers is competent enough to give the final answer—and that’s quite a problem, since reviewers are usually experts in their domain. If they cannot agree on the value of a contribution, then who can?
So objectivity is out of the question; value judgements—even on scientific matters—will always be subjective.

Then there’s the question of retaliation. If I give an author a bad review, time will come when that author gives me a bad review in turn, for non-scientific, personal reasons.
Perhaps surprisingly, non-anonymous reviews actually prevent retaliation: the author’s name would appear on that bad review. If it’s intentionally bad, you would have reasons to step towards the chair and explain the situation.

This is what accountability is all about. If you write a review with your name on it, you will think twice before making ungrounded or unscientific arguments. Who would hold an honest, informed opinion against you? People who do, clearly should not be part of the review process. That problem thus solves itself.

There’s only one real drawback for non-anonymous reviewers: they have to read the paper very well in order to form that honest, informed opinion. But this can only strongly benefit the quality of published research papers.

A plea for open peer review

Recently, I was invited to review an article for the Semantic Web Journal, which employs an open peer review process. Even though reviewers can opt for anonymity, the default is that your name is published along with your review. For instance, check this example in which four well-known researchers are glad to publish critical, in-depth reviews with their name on it.

I’m quite sure they won’t risk their “good name” by only reading half of the paper and rejecting it (or accepting it!) based on vague excuses. As a reviewer, it really is your duty to try to understand the paper as good as you can. That takes work.

I once heard an interesting argument against putting a lot of effort in a review:

“Sometimes, it is already clear from the abstract or introduction that the authors did not put a lot of effort in the paper (serious spelling mistakes, missing words, …). Why should I then put effort in my review?”

Oddly enough, this argument seems to imply that the reviewer would use anonymity to reject the paper without fully reading it, without having to admit this.
But why not simply state this in the review then?
“For this and that reason, I have not fully read the paper. I advise the authors to revise their manuscript in this and that way in order to enable a full review.”
Seems much more helpful—and honest—than to write a vague review because you (understandably) didn’t want to read the paper.

Start with more openness in your own reviews

Given that most of the conferences and journals don’t have an open review policy, what can we already do today?

I was quite charmed by a paragraph at the end of one of the reviews I received on one of my own papers:

If you wish to contact me after reading this review,
I’m Raphaël Troncy (e-mail & @rtroncy on Twitter).

When I thanked Raphaël for this, he replied hadn’t been actually contacted yet. But that’s not the important part: allowing people to contact you and taking ownership and responsibility for a review is a very sensible thing.

While I do admit it can be inconvenient to be named if your co-reviewers are anonymous, it is a strong signal and a step in the right direction.
If you want to change the world, start with yourself ;-)

And by the way, while I’m at it, here are some very valuable guides for reviewers:

• A quick guide to writing a solid peer review
• How to become good at peer review
• Peer review: how to get it right

If you follow that advice, you won’t be ashamed to put your name under your review!