Ranking Astronomers

If you want to fire up astronomers (and scientists in general), start discussing the topics of research impact and research metrics. These are the buzz words at the moment, as governments around the world are carrying out assessment of research done with public funds. Here in Australia we are in the current round of The Excellence in Research for Australia, where research in universities is scored on a scale which compares it to international standards.

I could write pages on attitudes to such exercises, and what it means, but I do note that the number of papers an academic has was a factor considered in a recent round of redundancies at the University of Sydney. But what I will focus upon is an individual ranking, the h-index.

You can read the details of the h-index at wikipedia, but simply put, take all the papers written by an academic, and find out how many times each of them has been cited. Order the papers from the most cited to the least, and where the number of the paper (going down the list) matches the number of citations, then  this is the academics h-index. Here's the piccy from wikipedia:


If you check me out at Google Scholar, I have an h-index of 49, which means I have 49 papers with at least 49 citations. Things like Google Scholar, and the older (but still absolutely excellent) Astrophysical Data Service, make calculating the h-index for astronomers (and even academics in general) extremely easy. The result is that people now write papers about people h-indicies, papers like this one which ranks Australian astronomers in terms of their output over particular periods.

Now, a quick google search will turn up a bucket load of articles for and against the h-index. There are a lot of complaints about the h-index, that it does not take things like the field of research, or number of authors, or the time needed to build up citations (slow cooker research which is not recognised at the time, but becomes influential after a long period, sometimes after the researcher has died!). 

Others seem to have a bit of a dirty feeling about the h-index, that ranking astronomers and research is somewhat below being an academic, especially things like producing league tables as in the paper above. 

The problem is that, in reality, scientists are ranked all the time, be it for grant applications, telescope time, jobs etc. and in all of these, it is necessary to compare one researcher to another. Such comparisons can be very difficult. When faced with a mountain of CVs, with publication records as long as your arm, grand success and even outside interests (why, oh why, in job applications do people feel it is necessary to tell me they like socialising and reading fantasy books??), it can be hard to compare John Smith with Jayne Smythe.

This is why I am a fan of the h-index. 

But let's be clear why and how. I know the age-old statement that "Past performance is not an indicator of future success" but when hiring someone, or allocating grant money or telescope time, people are implicitly looking at a return on their investment, they want to see success. And to judge that, you need to look at peoples' past record and extrapolate into the future. If Joe Blogs has received several grants in the past and nothing has come of it, then do you really want to give them more money? And what if they received quantities of telescope time and never publish the results? Is it a good idea to give them more time? 

But the stakes are higher. "Impact" is key, and one view of impact is that your research is read, and more importantly cited, by other academics around the world. What if Joe is a prolific publisher, and all of his papers appear in the Bulgarian Journal of Moon Rocks, with no evidence that anyone is reading his papers? Do you want to fund him to write more papers that no-one is going to read?

Now, some will say that academic freedom means that Joe should be free to work on whatever he like, and I agree that this is true. But as the opening of this post pointed out, governments, and hence universities, are assessing research and funding, and this assessment wants to see "dollars = impact". 

So, when looking at applications, be it for a job or grants, then the h-index is a good place to start; does this researcher have a track record of publishing work that is cited by others? Especially for jobs, it appears that the h-index actually has some predictive powers (although I know this is not globally true, as I know a few early hot-shots that fell off the table).

But let me stress, the h-index is a good place to start, not end, the process. 

I agree with Bryan Gaensler's statement that we should "Reward ideas, not CVs", and the next "big thing" might come from left-field, from a little known researcher who has yet to establish themselves, but realistically a research portfolio should be a mix of ventures; research that is guaranteed to produce solid results, with some risky ideas that might pay off big time, and we have to judge that by looking at the research proposal as a whole (and I think this should be true for an individual's research portfolio, or a department, or even a country).


Anyway, professional academics know that they are being assessed and ranked, and know that those that count beans are watching. I know there are a myriad of potential metrics that one can use to assess a researcher (and funnily enough, many researchers like the one that they look good in :), and I also know that you should look at the whole package when assessing the future potential of a researcher.


But the h-index is a good place to start.



Comments

  1. Phillip Helbig14 May 2012 at 18:33


    But let's be clear why and how. I know the age-old statement that "Past performance is not an indicator of future success" but when hiring someone, or allocating grant money or telescope time, people are implicitly looking at a return on their investment, they want to see success.

    Past performance can be used as a proxy to estimate future success. That's fine as far as it goes, but doesn't go far enough. People shouldn't be hired as a reward for past accomplishments, but rather because they will do good stuff in the future. In practice, someone who was, say, a Hubble Fellow will probably make the short list automatically, while others will be scrutinized more. However, one should actually demand more of the Hubble Fellow, because he had a better environment in which to work than the typical post-doc. In practice, people probably demand less. The other side of the coin is that someone who hasn't published as much due to circumstances beyond his control doesn't rank as high. One should factor in the difficulty of his situation---not out of sympathy, but because in the future the working conditions of A or B will be essentially the same, since they are applying for the same job. So the person doing the hiring should, in his own interest, multiply the h-index, or whatever, by the difficulty of the candidate's situation. (Yes, someone who was a Hubble Fellow presumably got it because he was good (though of course sometimes other factors play a role in hiring), but he has already been rewarded for that, so to speak, by the Hubble Fellowship itself. If this is an automatic passage to the next job and so on, then all one has to do is land a good job at the beginning of one's career and then coast from there, which is especially dangerous if the first good job was not awarded entirely on the basis of qualifications. But even if it were, people can burn out etc.)

    The h-index is probably better than counting number of papers, or number of citations, and most objections to it apply to bibliometry in general. However, I think that one should take into account the number of authors. Two candidates with the same h-index (and let's say same number of papers, same total number of citations etc) are certainly not comparable if A's papers are all single-author papers and B's are papers with several authors. So, divide the number of citations of a paper by the number of authors before calculating the h-index.

    I think that, if one wants to use one number, the g-index is a bit better than the h-index.

    The biggest problem in bibliometry, though, is probably the fact that the criteria for becoming an author vary from place to place and even within an institution. In other words, the amount of effort required to move from the acknowledgements to co-author status varies extremely widely.

    ReplyDelete

Post a Comment

Popular posts from this blog

Falling into a black hole: Just what do you see?

Image
Everyone loves black holes. Immense gravity, a one-way space-time membrane, the possibility of links to other universes. All lovely stuff. A little trawl of the internets reveals an awful lot of web pages discussing black holes, and discussions about spaghettification, firewalls, lost information, and many other things. Actually, a lot of the stuff out there on the web is nonsense, hand-waving, partly informed guesswork. And one of the questions that gets asked is "What would you see looking out into the universe?" Some (incorrectly) say that you would never cross the event horizon, a significant mis-understanding of the coordinates of relativity. Other (incorrectly) conclude from this that you actually see the entire future history of the universe play out in front of your eyes. What we have to remember, of course, is that relativity is a mathematical theory, and instead of hand waving, we can use mathematics to work out what we will see. And that's what I did. ...
Read more

Journey to the Far-Side of the Sun

Image
There was a movie, in the old days, Journey to the Far-Side of the Sun  (also known as Doppleganger) which (spoiler alert) posits that there is a mirror version of the Earth hidden on the other side of the Sun, sharing the orbit with our Earth. The idea is that this planet would always be hidden behind the Sun, and so we would not know it there there. This idea comes up a lot, over and over again. In fact, it came up again last week on twitter. But there's a problem. It assumes the Earth is on a circular orbit. I won't go into the details here, but one of the greatest insights in astronomy was the discovery of Kepler's laws of planetary motion , telling us that planets move on elliptical orbits. With this, there was the realisation that planets can't move at uniform speeds, but travel quickly when closer to the Sun, while slowing down as their orbits carry them to larger distance.  There has been a lot of work examining orbits in the Solar System, and you can  s...
Read more

Moving Charges and Magnetic Fields

Image
Still struggling with grant writing season, so another post which has resulted in my random musings about the Universe (which actually happens quite a lot). In second semester, I am teaching  electricity and magnetism to our First Year Advanced Class . I really enjoy teaching this class as the kids are on the ball and can ask some deep and meaningful questions. But the course is not ideal. Why? Because we teach from a textbook and the problem is that virtually all modern text books are almost the same. Science is trotted out in an almost historical progression. But it does not have to be taught that way. In fact, it would be great if we could start with  Hamiltonian and Lagrangian  approaches, and derive physics from a top down approach. We're told that it's mathematically too challenging, but it really isn't. In fact, I would start with a book like  The Theoretical Minimum , not some multicoloured compendium of physics. We have to work with what we have! ...
Read more