A Jury of One's Peers

For those of you who aren't part of the Secret Society, I'm going to give up some of the dark doctrine. Here's how scientific and medical journals work.

Imagine that you are a hard working person of scientificness. You do some research and you want to get it published, possibly for the good of humanity, possibly for the good of your tenure application, and possibly for the good of the drug company that sponsored your work, or any combination of the above. It's not like applying to college. Rule #1 is that you are allowed to send your manuscript to one, and only one journal. You have to wait for them to reject it before you apply to your back up. It can take anywhere from a month to three months or longer to get a decision, so you have to make a decision. Do I go for the big prestigious journal that would get my work noticed and cited, and get me tenure, but probably get rejected and not get my work published until much later (when somebody else probably will have published similar results, and/or the committee on tenure and promotions has already met)? Or do I go for a relatively obscure journal that is just grateful to get a MS written in complete sentences?

Whatever you decide, the editor who receives your MS, who is overworked and underpaid, will give it a quick look and either immediately send you a form letter saying no thanks, or send it out to peer review. The editor's decision will have a lot to do with where you work and whether she or (far more likely) he recognizes your name.

The so-called peer reviewers, of which there are usually two but conceivably three, will also have this information, but you will never know anything about them. They have an absolute shield of anonymity. I have been a peer reviewer for some well known journals -- including Public Health Reports, Social Science and Medicine, and the Journal of General Internal Medicine. That doesn't mean I'm any great shakes in the world of public health and medicine, however. They're pretty desperate for volunteers, so my main qualification was that they had my e-mail address because I had submitted papers to them myself. (And been published in two out of three of them.) I have also been asked to review because an author suggested me, which some journals allow.

So this is a task for which I am not paid, for which I receive no credit, and for which I have absolutely no accountability. The worst that can happen is that the editor won't ask me to do it again, which is not exactly punishment. Nevertheless, because of how my mama raised me, I take it very seriously. I always start by doing a medline search on the subject, so that I have an idea of the state of the relevant art and science. (Once I discovered that the author was attempting redundant publication, which is a major no no.) Then I read the article very carefully, think about its overall logic and coherence, then I write thorough criticism, paragraph by paragraph.

Unfortunately, I do not have access to the data, but only the tabulations the authors choose to present. Sometimes I can do some calculations of my own to make sure everything adds up and is consistent (which it occasionally is not!) but I have to take it on faith that they did the analysis accurately. I usually have the expertise to assess some aspects of the study, analysis and conclusions, but not necessarily all. I have even gone so far as to ask for help from a statistician or medical doctor on occasion. Then I make my (secret) recommendations to the editors. The authors get to read the critique I wrote for them, but I continue to hide behind my cloak of absolute anonymity. After the editors make their decision, they inform me, and send me a copy of the other reviewer's critique, which is inevitably, in my experience, cursory and frequently highly credulous.

So, I take this very seriously and I put in time, thought and extra effort. But most peer reviewers don't work at it nearly as hard as I do. I have also had the experience -- and I managed to ferret out the truth later, by means I cannot reveal -- of receiving a negative review because the reviewer was about to submit similar findings for publication. I had to argue with the editor very hard to get our study accepted as a brief report. Then, when the reviewer finally got his study published a 18 months later -- with essentially identical findings -- he did not cite our work. I ended up shaming him into writing a letter to the editor of the journal acknowledging our priority.

I am certain that in the case of studies of potentially high importance, the editors make more of an effort to obtain thorough, competent peer review. However, under ordinary circumstances even the most conscientious peer review can only assure that the methodology as stated is sound, that the analysis as stated is plausible, and that the findings as stated follow from the analysis. We simply have no information to assess the rigorousness and integrity of the data collection process, the accuracy of the data processing and analysis, or for that matter the truthfulness of the authors. At the same time, for any number of possible ulterior motives, we can encourage publication of studies that should not be published, and discourage publication of studies that should be published. Nobody is checking up on us and we face no consequences for our actions.

You just have to trust us.