Peer review, why do I bother?

For those of you who know about peer review, you know that the system is broken. But for those of you who aren't in the know, peer review is one of the ways that science journals try to ensure that the studies they publish are properly performed, that the conclusions match the data, and to reduce issues such as fraud.

In theory the process works as follows: You submit your study to a journal, and if it passes the reviewers initial look-over, your study gets sent out to two or three experts in your field who then review your work. They are anonymous (although the authors of the study are not) to protect them from "retribution" if you don't like their review. In theory, the reviewers are supposed to look at the paper, identify any holes, and recommend ways to fix those holes.

Unfortunately, this isn't always what happens. Some reviewers don't put much of an effort into reviewing, so you get half-baked reviews that don't do much to improve the paper but waste a lot of time. Other's will foist off reviews onto untrained grad students or postdocs, and don't vet the review before sending it in. While not always bad, the lack of expertise these individuals have often leads to them concentrating on minutia that is often irrelevant. Other reviewers use the review process as a way to hamper their competitors (whose papers they often receive for review) by recommending unreasonable experiments or even recommending rejection of the paper. And the absolute worst is when you get a real ass hole who thinks you did everything wrong, and demands you rodo every experiment, and rewrite the paper to fit the way they think it should have worked.

To fight this I do my absolute best when reviewing papers, in the vague hope that I may set a good example. I take time to read the paper through - usually several hours, and try to provide reasonable and helpful feedback. In most cases the authors seem to appreciate this and try to address my concerns.

And then there was the last paper I reviewed.

I cannot speak of specifics, due to confidentiality, but the long story short is a group developed a method that is a pretty big advance to one of my fields. Basically, they developed a mass-screening assay for a field that has traditionally used low-throughput methods. As I said, its a big deal - it would have cut my PhD in half, for example. There was one critical problem with their paper though - they didn't actually do the test to confirm that their new readout actually "readsout" what was being measured. To fix that was simple - they had stills from a video already in the paper. All they needed was to do 5min of work and actually analyse that video. I pointed that out, provided a few references in case they didn't know how to do the analysis, and then sent in my review.

In reply, they added the video - without any analysis - to the paper. Or, in other words, they ignored the one recommendation I had, and there is still a gaping hole in their paper.

ARRRRRRRRRRRRRRRRRRRGGGGGGGGGGGGGGGGGGG

So I did the analysis myself - and it turns out their assay works perfectly. It took me a whopping 3min; 2 of which was booting my computer and loading matlab. But this puts me in a real pickle - without that data their paper has a serious flaw, but their method is correct. Normally I'd re-write the editor saying "they need to do this analysis, or reject you need to reject this paper", but since this is a big advance - that works extremely well - I also want to see the paper published ASAP.

Makes me wonder why I bothered putting any effort into the paper. I don't know what I'm going to do yet - but flipping a coin tops the "options list".