Sunday, 2 December 2012

Misunderstanding Peer Review

Sunday morning. I've got a busy week coming up, having to head to Canberra tomorrow for my induction into the Australian Research Council (ARC) College, then a few board of examiners meetings, the allocation of supercomputer time, and then Christmas drinkies on the Botany Lawn.


So a quick post on peer review.

There has been an awful lot written on this, and I will focus on academic peer review, especially when it comes to publications. I read this interesting article on peer review, which points out that the press does not seem to understand the difference between a published article, and things written in popular science magazines, on blogs, or simply chatted about down the pub.

But before I start, let me remind you how a paper gets published in a journal. 

It starts by you having your idea, doing your work, and deciding that the results are significant enough to be interesting to everyone else. 

You need to decide where you are going to publish it. Journals are tiered. There are top tier journals (for astronomers, these are Nature and Science), then there are the middle tier, which are the main journals (again, for astronomers, these are The Astrophysical JournalAstronomical JournalMonthly Notices of the Royal Astronomical Society and Astronomy & Astrophysics). There there are other journals (but I won't name them :) I'll tell you what sets the journal tiers a little later.

You bundle you paper into the appropriate format for your journal of choice and send it off. The journal receives the paper and an editor takes a quick look. The editor will then make a choice. Is the science of enough importance to publish? Is it appropriate for the journal? If not, it gets rejected.

If it is appropriate, then it goes out to the referees. This is the part that people typically see as peer review.

The referees are generally experts in the field, and typically review the paper for free. Many find this a bit weird, why would anyone do the work for the journal for free?
Honestly, I don't really know. It's just one of things expected of academics. I got my first paper to referee as a student, and was chuffed that someone thought I was enough of an expert to examine someone elses work. And so it be began, and I now referee around 8-10 papers per year.

The referee may not be as impressed as you about the importance of your work, or might find a flaw in your reasoning or approach. They could recommend that you paper be rejected on these grounds.

But they might like it, and recommend publication. Then your paper is officially "accepted" and will appear in the journal. Yay!

This is the part that people have a problem with. They argue that as only a few eyes have scanned the paper before it was accepted, it could still be flawed or wrong. Peer review is a failure, they cry.

Here's my take on the situation, and I will put it in bold as I think it is very important: Peer Review Does Not Stop Once You Paper Has Been Accepted. In Fact, It Has Only Just Begun!

You might say "Huh?" to this. To professionals, it is not the fact that your paper has been accepted that is important, it is how it is received my the community. What you want is for your work to me noticed, and more importantly, cited. This is why there is a growing obsession with bibliometrics, not only with individual researchers, but also with governments. 

The new buzz word is impact. Not only does your science have to be excellent, it has to be noticed, it has to be cited, it has to be used, it has to have impact.

And impact is also related to where you paper is published, and it is generally thought that if you can get your article into journals with a high impact factor, it clearly must be good science (this is what sets the tiers of journals). This is not a universally held view.

So, the referee process is nothing part of the first step of the journey of your paper. You might publish it and it may sink like a stone, unnoticed by the community. These sunken papers may reappear 10, 50 or 100 years in the future as a new direction in science, unrealised during your life, comes to fruition. 

But what scientists want is for their research to be noticed, and this is nothing but a continuous review by your peers. 


3 comments:

  1. "it is generally thought that if you can get your article into journals with a high impact factor, it clearly must be good science (this is what sets the tiers of journals). This is not a universally held view."

    This is clearly rubbish. The whole idea of impact is that citations are important. That in itself is not obvious (at least, the number of citations and the reasons for them need to be understood), but let's assume that the more citations the better. OK, a journal has a high impact factor if its articles have a high average citation rate. So, some people are so deluded that they think that publishing in such a journal must mean their paper is good. However, if they believe that the number of citations is good, then why not just look at the number of citations for their own paper?

    ReplyDelete
  2. Second time to write this. While I generally agree with you, I don't think that IF are read the way you say. There is a correlation between IF and the robustness of the refereeing stage. Nature and Science are harder to publish in than ApJ and MNRAS, and they are harder than the Bulgarian Journal of Astronomical Delights. There is a corresponding decrease in impact factor, and so it is taken as a proxy for the refereeing stage. But this does not guarantee citations. These only come with peer review..

    ReplyDelete
    Replies
    1. Yes, Nature and Science are harder to publish in, and the IF is higher. However, the main reason they are harder to publish in is that many people want to publish there and they cover all the sciences, not just astronomy. Supply and demand. I think the actual refereeing is actually not significantly better than standard journals. Rather, I think they go for "sexy" topics.

      Thus, if someone gets published in one of these high-IF journals, my conclusion would be that the paper is on a "hot" topic and is of reasonable quality, not that it is necessarily outstanding in quality.

      Some things are expensive because they have to be (parts, labour etc). Some are expensive because they are artificially limited to maintain an exclusive reputation.

      Delete