Misunderstanding Peer Review

Sunday morning. I've got a busy week coming up, having to head to Canberra tomorrow for my induction into the Australian Research Council (ARC) College, then a few board of examiners meetings, the allocation of supercomputer time, and then Christmas drinkies on the Botany Lawn.


So a quick post on peer review.

There has been an awful lot written on this, and I will focus on academic peer review, especially when it comes to publications. I read this interesting article on peer review, which points out that the press does not seem to understand the difference between a published article, and things written in popular science magazines, on blogs, or simply chatted about down the pub.

But before I start, let me remind you how a paper gets published in a journal. 

It starts by you having your idea, doing your work, and deciding that the results are significant enough to be interesting to everyone else. 

You need to decide where you are going to publish it. Journals are tiered. There are top tier journals (for astronomers, these are Nature and Science), then there are the middle tier, which are the main journals (again, for astronomers, these are The Astrophysical Journal, Astronomical Journal, Monthly Notices of the Royal Astronomical Society and Astronomy & Astrophysics). There there are other journals (but I won't name them :) I'll tell you what sets the journal tiers a little later.

You bundle you paper into the appropriate format for your journal of choice and send it off. The journal receives the paper and an editor takes a quick look. The editor will then make a choice. Is the science of enough importance to publish? Is it appropriate for the journal? If not, it gets rejected.

If it is appropriate, then it goes out to the referees. This is the part that people typically see as peer review.

The referees are generally experts in the field, and typically review the paper for free. Many find this a bit weird, why would anyone do the work for the journal for free?
Honestly, I don't really know. It's just one of things expected of academics. I got my first paper to referee as a student, and was chuffed that someone thought I was enough of an expert to examine someone elses work. And so it be began, and I now referee around 8-10 papers per year.

The referee may not be as impressed as you about the importance of your work, or might find a flaw in your reasoning or approach. They could recommend that you paper be rejected on these grounds.

But they might like it, and recommend publication. Then your paper is officially "accepted" and will appear in the journal. Yay!

This is the part that people have a problem with. They argue that as only a few eyes have scanned the paper before it was accepted, it could still be flawed or wrong. Peer review is a failure, they cry.

Here's my take on the situation, and I will put it in bold as I think it is very important: Peer Review Does Not Stop Once You Paper Has Been Accepted. In Fact, It Has Only Just Begun!

You might say "Huh?" to this. To professionals, it is not the fact that your paper has been accepted that is important, it is how it is received my the community. What you want is for your work to me noticed, and more importantly, cited. This is why there is a growing obsession with bibliometrics, not only with individual researchers, but also with governments. 

The new buzz word is impact. Not only does your science have to be excellent, it has to be noticed, it has to be cited, it has to be used, it has to have impact.

And impact is also related to where you paper is published, and it is generally thought that if you can get your article into journals with a high impact factor, it clearly must be good science (this is what sets the tiers of journals). This is not a universally held view.

So, the referee process is nothing part of the first step of the journey of your paper. You might publish it and it may sink like a stone, unnoticed by the community. These sunken papers may reappear 10, 50 or 100 years in the future as a new direction in science, unrealised during your life, comes to fruition. 

But what scientists want is for their research to be noticed, and this is nothing but a continuous review by your peers. 


Location: Sydney NSW, Australia

Comments

  1. Phillip Helbig3 December 2012 at 22:47

    "it is generally thought that if you can get your article into journals with a high impact factor, it clearly must be good science (this is what sets the tiers of journals). This is not a universally held view."

    This is clearly rubbish. The whole idea of impact is that citations are important. That in itself is not obvious (at least, the number of citations and the reasons for them need to be understood), but let's assume that the more citations the better. OK, a journal has a high impact factor if its articles have a high average citation rate. So, some people are so deluded that they think that publishing in such a journal must mean their paper is good. However, if they believe that the number of citations is good, then why not just look at the number of citations for their own paper?

    ReplyDelete
  2. Cusp6 December 2012 at 08:25

    Second time to write this. While I generally agree with you, I don't think that IF are read the way you say. There is a correlation between IF and the robustness of the refereeing stage. Nature and Science are harder to publish in than ApJ and MNRAS, and they are harder than the Bulgarian Journal of Astronomical Delights. There is a corresponding decrease in impact factor, and so it is taken as a proxy for the refereeing stage. But this does not guarantee citations. These only come with peer review..

    ReplyDelete
    Replies
    1. Phillip Helbig14 December 2012 at 01:39

      Yes, Nature and Science are harder to publish in, and the IF is higher. However, the main reason they are harder to publish in is that many people want to publish there and they cover all the sciences, not just astronomy. Supply and demand. I think the actual refereeing is actually not significantly better than standard journals. Rather, I think they go for "sexy" topics.

      Thus, if someone gets published in one of these high-IF journals, my conclusion would be that the paper is on a "hot" topic and is of reasonable quality, not that it is necessarily outstanding in quality.

      Some things are expensive because they have to be (parts, labour etc). Some are expensive because they are artificially limited to maintain an exclusive reputation.

      Delete

Post a Comment

Popular posts from this blog

Falling into a black hole: Just what do you see?

Image
Everyone loves black holes. Immense gravity, a one-way space-time membrane, the possibility of links to other universes. All lovely stuff. A little trawl of the internets reveals an awful lot of web pages discussing black holes, and discussions about spaghettification, firewalls, lost information, and many other things. Actually, a lot of the stuff out there on the web is nonsense, hand-waving, partly informed guesswork. And one of the questions that gets asked is "What would you see looking out into the universe?" Some (incorrectly) say that you would never cross the event horizon, a significant mis-understanding of the coordinates of relativity. Other (incorrectly) conclude from this that you actually see the entire future history of the universe play out in front of your eyes. What we have to remember, of course, is that relativity is a mathematical theory, and instead of hand waving, we can use mathematics to work out what we will see. And that's what I did.
Read more

Journey to the Far-Side of the Sun

Image
There was a movie, in the old days, Journey to the Far-Side of the Sun  (also known as Doppleganger) which (spoiler alert) posits that there is a mirror version of the Earth hidden on the other side of the Sun, sharing the orbit with our Earth. The idea is that this planet would always be hidden behind the Sun, and so we would not know it there there. This idea comes up a lot, over and over again. In fact, it came up again last week on twitter. But there's a problem. It assumes the Earth is on a circular orbit. I won't go into the details here, but one of the greatest insights in astronomy was the discovery of Kepler's laws of planetary motion , telling us that planets move on elliptical orbits. With this, there was the realisation that planets can't move at uniform speeds, but travel quickly when closer to the Sun, while slowing down as their orbits carry them to larger distance.  There has been a lot of work examining orbits in the Solar System, and you can  s
Read more

Proton: a life story

Proton: a life story  by Geraint F. Lewis 10 35 years: I’ve lived a long and eventful life, but I know that death is almost upon me. Around me, my kind are slowly melting into the darkness that is now the universe, and my time will eventually come. I’ve lived a long and eventful life… 10 -43 seconds: A time of unbelievable light, unbelievable heat! I don’t remember the time before I was born, but I was there, disembodied, ethereal, part of the swirling, roaring fires of the universe coming in to being. But the universe cooled. From the featureless inferno, its character crystalized into a seething sea of particles and forces. Electrons and quarks tore about, smashing and crashing into photons and neutrinos. The universe continued to cool. 1 second: The intensity of the heat steadily died away, and I was born. In truth, there was no precise moment of my birth, but as the universe cooled my innards, free quarks, bound together, and I was suddenly there! A p
Read more