Misunderstanding Peer Review
Sunday morning. I've got a busy week coming up, having to head to Canberra tomorrow for my induction into the Australian Research Council (ARC) College, then a few board of examiners meetings, the allocation of supercomputer time, and then Christmas drinkies on the Botany Lawn.
So a quick post on peer review.
There has been an awful lot written on this, and I will focus on academic peer review, especially when it comes to publications. I read this interesting article on peer review, which points out that the press does not seem to understand the difference between a published article, and things written in popular science magazines, on blogs, or simply chatted about down the pub.
But before I start, let me remind you how a paper gets published in a journal.
It starts by you having your idea, doing your work, and deciding that the results are significant enough to be interesting to everyone else.
You need to decide where you are going to publish it. Journals are tiered. There are top tier journals (for astronomers, these are Nature and Science), then there are the middle tier, which are the main journals (again, for astronomers, these are The Astrophysical Journal, Astronomical Journal, Monthly Notices of the Royal Astronomical Society and Astronomy & Astrophysics). There there are other journals (but I won't name them :) I'll tell you what sets the journal tiers a little later.
You bundle you paper into the appropriate format for your journal of choice and send it off. The journal receives the paper and an editor takes a quick look. The editor will then make a choice. Is the science of enough importance to publish? Is it appropriate for the journal? If not, it gets rejected.
If it is appropriate, then it goes out to the referees. This is the part that people typically see as peer review.
The referees are generally experts in the field, and typically review the paper for free. Many find this a bit weird, why would anyone do the work for the journal for free?
Honestly, I don't really know. It's just one of things expected of academics. I got my first paper to referee as a student, and was chuffed that someone thought I was enough of an expert to examine someone elses work. And so it be began, and I now referee around 8-10 papers per year.
The referee may not be as impressed as you about the importance of your work, or might find a flaw in your reasoning or approach. They could recommend that you paper be rejected on these grounds.
But they might like it, and recommend publication. Then your paper is officially "accepted" and will appear in the journal. Yay!
This is the part that people have a problem with. They argue that as only a few eyes have scanned the paper before it was accepted, it could still be flawed or wrong. Peer review is a failure, they cry.
Here's my take on the situation, and I will put it in bold as I think it is very important: Peer Review Does Not Stop Once You Paper Has Been Accepted. In Fact, It Has Only Just Begun!
You might say "Huh?" to this. To professionals, it is not the fact that your paper has been accepted that is important, it is how it is received my the community. What you want is for your work to me noticed, and more importantly, cited. This is why there is a growing obsession with bibliometrics, not only with individual researchers, but also with governments.
The new buzz word is impact. Not only does your science have to be excellent, it has to be noticed, it has to be cited, it has to be used, it has to have impact.
And impact is also related to where you paper is published, and it is generally thought that if you can get your article into journals with a high impact factor, it clearly must be good science (this is what sets the tiers of journals). This is not a universally held view.
So, the referee process is nothing part of the first step of the journey of your paper. You might publish it and it may sink like a stone, unnoticed by the community. These sunken papers may reappear 10, 50 or 100 years in the future as a new direction in science, unrealised during your life, comes to fruition.
But what scientists want is for their research to be noticed, and this is nothing but a continuous review by your peers.