Peer reviewing

One of the foundations of the scientific method is publication of results so that they can be assessed and reproduced by the wider community of researchers.  Peer review is the first step in that publication process, voluntary, anonymous review of work by experts in the field, with comments returned to the author so that the paper can be improved prior to publication.

I serve as a peer reviewer for several journals and probably get requests about twice a month to comment on a paper.  Generally it takes about an hour to understand the work, to assess whether it is correct, clear, open to alternative interpretation,and properly positioned within the relevant landscape of background literature. I try to write balanced comments, praising a clear hypothesis, a well-reasoned discussion, while perhaps suggesting where figures could be more clear or an alternative method might yield better results.  The revised paper comes back the next month and I generally pass it if the author has replied to my points (whether or not those result in changes).

Occasionally I run into situations where I struggle.  I received one where the author had copied an earlier paper, word for word, yet not cited it. I talked with my editor (the lead author was fairly senior) and we recommended that he include the citation: it was enough to alert him that we’d found the earlier work.   Another was proposing pages of dense calculation for an obscure visualization technique that, in the end,m didn’t reveal anything new in the data.  I think that, in six revisions, we did everything but rewrite the paper for him.

Last month, I received a work that was analyzing structure in short-and long-term heart rate changes.  It’s a tricky analysis, since it’s statistical rather than deterministic, and the author had calculated some crude ratios to support his hypothesis.  Worse, his underlying technique destroyed the very ordering among data points that he based his hypothesis on.  I pointed out that the results weren’t supported by the method, suggested some alternative approaches fo analysis, and worked out an example to illustrate my point.  Throughout I was constructive and respectful (I though), allowing that I might have missed his point, but didn’t follow the logic.

The authors absolutely blistered me in their reply.

…there are many comments made by the Reviewers with which we disagree, comments which we consider unhelpful and comments which are just wrong…we strongly believe that the Reviewers should be aware of the fact that, even though they are anonymous, they should put time and effort in reading and reviewing the papers. We are frequent reviewers for a number of peer-reviewed journals,and we always try to helpful for the Authors and never write from the position of someone who, by definition, knows better.

Most of my comments were subsequently dismissed with this is an opinion and there is nothing we can do about it, except state that we disagree.  Fair enough: I went back through my analysis and still cannot link their conclusions with their data using their method.  I had to just recuse myself after that: while we might have solved the disagreements over a beer and a whiteboard, their reaction was so strong and their comments so dismissive, that there just weren’t actually any explanations to discuss.

I felt badly about the comments and about the outcome: I do take the job seriously and give every paper a good read.  I know how much work it takes to do research and to write it up.  In the end, if they are right, they should be able to refute the counter argument and to say why.  If they are wrong, they need to make corrections.

And, fundamentally, I disagree with them that Reviewers are there to serve the Authors. 

Reviewers are present to serve Readers.

If, as a first reader, I think that a paper is wrong, I have a responsibility to (nicely) say so, suggest why, and present alternatives.  Maybe it’s a real mistake, maybe I just don’t understand, maybe the Methods contain omissions or ambiguities.  But honest and differing perspectives should be addressed openly.

We might argue that every author is entitled to his view and the Readers will sort things out. Eventually, yes.   But errors enter indexed literature, then pop up on searches, news reports, and blogs.

The Letters to the Editor don’t have much impact, and are presented as a debate that obscures rather than resolves issues.  It can take years before the truth is sorted by Readers, years of contradictions and misinterpretations in which true facts have been undermined and genuine debates get muddied. 

The argument presented by these Authors, that Reviewers should only check for clarity of language and labels on graphs, is wrong.  The peer review process is intended to engage experts as surrogate Readers, not as supplemental Editors.