A short post to express a bit of anguish and to vent. I apologize to my gentle readers in advance. The subject of the peer review system has been discussed elsewhere in greater detail (a notable example is an excellent post by Jonathan Eisen, blogger and Professor at UC Davis). I believe in it, because I firmly believe that serious, conscientious peer review can promote the cause of science and scientific progress.
A while back, I had received an invitation to review a manuscript for a fairly well-circulated journal, and agreed to do it, as the subject matter was well within my area of expertise. Upon reading the manuscript, I found that there were several lacunae in it, in terms of methodology, data interpretation and conclusion. I, however, didn’t recommend outright rejection, because there was scientific merit in the conception of the work, and I thought it necessary to have the information – of the kind embodied in the paper – out in the scientific literature. That, and also because I am painfully aware of the urgencies associated with publishing one’s work. I went through the manuscript assiduously, checking references, eventually pointing out line by line, page by page, where the problems lay, and suggesting how the authors could improve it. I sent it back with the recommendation to ‘Modify’.
It appears that the Editor for the article agreed with me, and returned a decision of ‘Modify’ to the authors. I received a notification to that effect. Initially, I had a quick case of warm fuzzies, because I thought I was able to help the authors; with some modification, their work could be published.
And then I saw something, scrolling down, which completely deflated me. The notification contained all the comments from all the reviewers. Following my detailed, eleven-paragraph response (as reviewer 1) to the authors, I found the responses from reviewer 2 and reviewer 3 – they wrote precisely ONE paragraph each, giving a very general and vague summary of the study, without making any recommendation as to the acceptability of the paper.
Several questions quickly coursed through my mind:
- Is this the true face of peer review – one single hastily scribbled paragraph, determining the value of painstaking work by researchers spending time and money and effort?
- Was I being unfair on the other reviewers, who may be so much more experienced than I am, that one look at the manuscript and they were able to visually separate the grain from the chaff?
- Was I – a mere postdoc who didn’t know any better – an idiot, an irredeemable non compos mentis, to have spent time and effort and care to review this paper in a constructive way?
- Would I want my own papers to be evaluated in this way?
Of course, like many other puzzling mysteries of life, these questions, too, leave me clueless.
Huh. I don’t know what your field is; in my field, nobody writes one-paragraph “reviews”.
The grass, green and sides, David? Hehehe. I work in, broadly, microbiology-immunology, and specifically, in host pathogen interaction, but I am also interested in immunological assay development.
That’s a dark side of peer-review, isn’t it? I would think though that it’s the editor’s job to notify reviewers 2 and 3 so as to ask each of them for more detailed reviews. Surely, he bears some responsibility as well.
Nice post – I know exactly where you’re coming from here, I’ve a tendency towards long reviews and have seen others write a single vague paragraph for the same ms I’ve dissected in (probably too much) detail. With my editor’s hat on, people who give poor quality reviews get a reputation, although whether not being asked to review again is punishment or reward I’m not sure! (Also worth noting that most journals offer a ‘comments to editor’ box which won’t appear in the decision letter that all reviewers see – so it’s possible, though unlikely, the ‘lazy’ reviewers have written more there.)
I do, however, think it’s really valuable as a reviewer to see all reviews – of course sometimes that just enables you to identify the kind of sharp practice you note, but more often all reviews are pretty thorough, which (especially when starting out as a reviewer) can help to reassure yourself that you’ve said sensible things.
Good point, Khalil. But perhaps the role of editor is rather limited; if s/he doesn’t like the brevity of a particular review, perhaps the only other thing s/he can do is call for another review – which would potentially delay the publication for no fault of the authors.
Tom, thank you. I appreciate your empathy. I am, of course, aware that the strange situation I encountered is, thankfully, not general or generalizable. 🙂
Maybe not generalisable, though “I’ve wondered at it”:http://www.scilogs.com/the_mawk_moth_profligacies/peered-review/. I wasn’t in the position to see the (other) reviewers’ comments. But approaching the paper as one qualified to review it, I had to wonder…
(Back blogging with a vengeance, Kausik – excellent!)