An interesting, though probably doomed, experiment

One e-mail list I'm glad to be on is that of Nicholas Zamiska, the WSJ's most excellent health and science correspondent based in Hong Kong. (And best of all, he never sends me Chinese pornography or offers to enlarge any of my body parts.) He tips us off to an experiment by Nature Magazine* to open up the peer review process to any competent lector, not just the two or three reviewers assigned by the journal.

Authors who agree to it have their submitted manuscripts placed on this website, where anybody can download the PDF and where qualified people can then leave their own comments. (The comments are moderated so it ain't Eschaton. Sorry, Adrian Spidle.**)

While Nick's article (WSJ, Sept. 14, page B1) emphasizes some recent failures of the peer review system to weed out fraud, this new procedure isn't likely to help a whole lot with that problem and it really isn't intended to do that. The possible virtues of this experiment, in my view, include:

1. It contributes to the democratization of science. Interested lay people, or experts in fields other than those covered by the articles on the site, can see what scientific work in progress looks like and how fellow scientists in the same field critique it. The discussions are bound to be quite technical and hard for non-experts to follow, but you can still glean a good bit of insight, I think. I would certainly consider using this as a resource in science classes, at the college level and even for bright high school students.
   
2. It can speed up discovery by giving the community of science early access to research. Publication can take months to a year or more, and meanwhile other scientists may be busy reinventing the wheel, or losing time by not having the results to build on.
   
3. It can certainly improve the quality of published research by providing a broader range of critiques ahead of time. Again though, even a thousand readers can't necessarily detect fraud. Lies don't always give themselves away.

A downside is that jealousy or competitiveness could lead people to trash each other's work, but that's actually less of a worry than it is with the traditional peer review process, since commenters must identify themselves, whereas peer reviewers are anonymous. Another problem is that people could steal ideas, I suppose, though it's hard to see how they'd get away with that -- if the results are valid, they'll be published long before somebody else can repeat them and get them printed. Some scientists might prefer to hoard their own findings while they work on the next step, but I don't find that honorable -- the realm of the unknown is infinite, it's silly to try to keep some of it for yourself.

A more serious downside is that journalists or activists of one kind or another might seize on these unpublished findings and publicize them or try to use them for advocacy in some way, and it will turn out that they are rejected by the peer review process and are not truly valid. Once a wrong idea gets embedded in the culture, it's awfully hard to get rid of it. (No, taste buds for sweetness are not concentrated on the tip of the tongue.) Well, we'll see.


*and if you check out their web site, you can get free access to the much-ballyhooed report on the Neanderthal's purported last stand near Gibraltar.
** Adrian won the Rising Hegemon award for "most psychotic commenter" in 2004.