Tuesday 18 December 2012

My year in review - Reviewing reviewing

Like most scientists, from time to time I get asked to review papers, and a few years ago I decided to keep track of the reviews I did for various journals. In 2010, 2011, and 2012 I was asked to review 7, 6 and 7 times respectively.

I've been trying to figure out is this a reasonable level of reviewing? I guess the question is, am I doing more reviewing for the chemistry community than they are doing for me? Not that I mind those freeloading slackers dumping it all on me - I'm just curious.

So what's the other side of the equation? In the same three years I've had my name attached to 8 peer-reviewed publications. If each was reviewed 2.5 times (a reasonable guesstimate), that's also 20 reviews.

And so the balance is maintained.

Note: I know what you're thinking...(eerie isn't it!)...you want to adjust the figures for multiple authors. But both of the values would have to be adjusted in the same way and so it just cancels out. (I think.)

Update (02/01/2013): The previous note is a load of rubbish. As Felix points out in the comments, I should be correcting for multiple authors. Oh well...

Image credit: After the Edit by Laura Ritchie (LMRitchie on Flickr)

8 comments:

jpo said...

I reviewed about 20 this year, and had about 10 published - I do think though that journals are getting ruder with reviewers. I have to do these things in my own time, so getting notes to say I'm late for reviewing is a bit cheeky.

To be honest I can't properly review the vast majority of the papers I see - I either can't have the data discussed to verify anything, I don't have the software, it would take a ridiculous amount of time to repeat an analysis on similar data, etc. etc. So everything factual stated is taken on trust.

Also, I can't check prior art for many papers, since I don't have access to cited papers, or papers that should be cited - one recent example would have meant me spending over $200 on Elsevier papers to more deeply review a paper for an Elsevier journal - now there's a good business model!

I also reject about two thirds of the papers I see, and then get a little annoyed to see my judged feedback ignored. Yes, I'm thinking of you PLoS journals!

Xmas rant over.

jpo

Egon Willighagen said...
This comment has been removed by the author.
Egon Willighagen said...

John, very recognizable! In numbers (except that I restrict myself to around 6 papers), and in sentiment.

It takes me 4-8 hours to review a paper, and that is without attempts to reproduce the results, but just reading it for sound argumentation, matching it against prior knowledge etc.

Noel O'Boyle said...

@JPO: The good news is that 2/3 rejection rate takes you right off my list of potential reviewers, which should reduce your reviewing burden going forward. :-)

@Egon: Just FYI and FWIW, I would be more in the 2 to 3 hour range per review.

And just for the record, there was an eigth paper this year I was asked to review, but I passed on it due to conflict of interest.

Andrew Dalke said...

In the last two years I've reviewed two papers, and published 0. I've been asked to review more papers, but I've decided that I have the luxury of ignoring requests from non-open access journals since publication numbers don't really affect my work.

While there's definitely papers for J. Cheminf. which I would have liked to reviewed. :)

Felix said...

if your coauthors are eligible as referees, then I think some freeloading slackers are dumping stuff on you ... :)

assuming I am in a group of five people and we published five papers with all of us as authors. Then these papers required ten referee reports (assuming 2 per paper). In return our group of five people should do 10 reports, as well. This means only two per person, right (unless all 10 just go to the group leader)?

anyway I will be excited about the first time when someone trusts me enough to ask me to do a referee report :)

Noel O'Boyle said...

@Felix: Dude - you are right. I am an idiot. I *do* need to correct for multiple authors, or at least, multiple reviewing authors.

There's only one solution I can see that is win-win for everyone: from now on, I'm going to send all my reviews to you to do.

Jan Jensen said...

Felix - that excitement quickly wears off