On the Road Again . . .

I'll be heading to the DC area tomorrow for a National Cancer Institute review panel. That's a bunch of people selected by an arcane process to review and score proposals for funding. That this panel exists, and that I am on it (for those who know or care to ferret out my secret identity as a mild-mannered reporter), is a matter of public record, but beyond that I'm not supposed to say anything specific about the proposals or our deliberations.

But this is a good occasion to talk about the peer review process at NIH. Congress appropriates money for what is usually collectively called biomedical research, but NIH also supports a certain amount of social science and public health research which is not strictly biomedical. In so doing, Congress conventionally establishes broad outlines of how the money is to be allocated among the various institutes (e.g., NCI, National Institute of Allergies and Infectious Diseases, National Institute of Child Health and Human Development, etc. -- you can see the entire list here) and may direct some amount of funding toward specific diseases or research problems. But the scientific community generally speaking does not like it when Congress tries to direct money to a specific study or investigator, and with few exceptions Congress has honored this convention.

Instead, NIH staff issue requests for proposals in various categories, using various mechanisms. Program Announcements are fairly open-ended, occasionally even catch-all funding streams that allow investigators to develop studies on their own initiative within broadly defined areas. Usually they include opportunities to apply for exploratory or pilot studies, with comparatively small amounts of funding, or large scale, fully developed studies using the so-called R01 mechanism. These remain open for years, and if you aren't funded the first time, you can revise your proposal and try again. Requests for Applications are one-time announcements seeking proposals to address more narrowly specified questions.

The review panels don't officially make the final funding decisions, but they are pretty much dispositive. Each member reads a few of the dozens of proposals that may be before the panel. Normally each proposal is read by three members. Then we write extensive critiques, and assign a score. The bottom half don't get discussed at the meeting, and don't get their score reported, but the applicants do get the comments. The rest get discussed, the critiques are revised, and final scores assigned. Then the relevant NIH national advisory panel makes the official funding awards. They seldom overrule a panel.

As you can see, this means that policy makers steer the funding only in general directions. In a meaningful sense, it is representatives of the community of scientists who rule on the merits of specific studies. This system is definitely imperfect, as I have observed as both an applicant and a colleague of applicants. One reason I wanted to do this is to get an inside look at it, because I would like to find ways to improve it. On the other hand, there are a lot of reasons why it's hard to improve upon. In the coming days, I'll take a look at the good, the bad and the ugly -- not with reference to this panel, as I am sworn to secrecy, but more generally.