PEMS revisited

This is really inside baseball, in other words it's more specialized than the grand world historical and philosophical questions we usually concentrate on here, but it's also kind of interesting and illustrates a more general problem. Some people may remember (though I doubt it) that I wrote last autumn about PEMS, which is not a version of female trouble, but the Program Evaluation and Monitoring System that CDC is installing for all of its HIV prevention programs.

PEMS requires states and community based organizations that provide HIV prevention services under contract to CDC to collect standard sets of data about their programs and clients, and to enter these into databases maintained on federally owned computers in Atlanta over secure Internet connections.

Now, it is normal and expected and completely proper for government agencies that purchase services to require reports on what is done with your money. I have previously discussed the Government Performance and Results Act, GPRA, which is Al Gore's fault, and as it happens the Center for Substance Abuse treatment fulfills its GPRA obligations by having its contractors submit data about clients using a web-based application. The HIV/AIDS Bureau of the Health Services and Resources Administration, which administers the Ryan White CARE Act Funds for services for people with HIV, provides an application called CAREWare which grantees install at their own sites, to collect information about clients and services. They then submit data reports by e-mail.

These systems can actually be useful to grantees, who need to be able to keep track of their clients, program outcomes, staff productivity, etc. However, there is considerable grumbling about some of the data requirements. CSAT, in particular, requires 80% follow-up after 6 and 12 months. In other words, programs are expected to be able to track down 80% of their clients 12 months after intake and give them the GPRA interview. If you know anything about addiction and treatment, you know that it requires a huge investment to even have a chance of doing that for most types of programs -- resources we could be spending on providing treatment. On the other hand, there is a legitimate public interest in knowing what happens to people after they leave treatment, how many relapse or end up in jail, how many are living more or less successfully, how many are still in treatment, etc. The good news is that CSAT and HRSA put enough funding for evaluation into their contracts that programs can generally meet these requirements or at least put up a respectable effort.

Since I first got involved in public health program evaluation some, ahh, years ago, my career in the field has consisted largely of a battle with direct service providers who see collecting data and filling out forms as an imposition and a waste of time, when they have needy people to help and lives to save. And the answer is yes you do, but if you don't collect this data and fill out these forms, we will never know if you are really helping people and saving lives, we won't be able to figure out how to do it better, and we won't be able to persuade anybody to keep giving us money to go on doing it.

So it's a tradeoff, and there is an appropriate balance somewhere, but there's no magic formula that tells you where it is. The PEMS initiative, it turns out, has ignited a firestorm of resistance. According to a confidential e-mail from a source to be named after the statute of limitations has expired, community based agencies are complaining that, among other complaints:

1. PEMS takes 25 minutes to complete, when you may only have 20 minutes to connect with a person on their first visit to your agency.
   
2. The PEMS questionnaire forces the most experienced and dedicated counselors to follow a script that drills people with a pre-set order of questions, and that stops them from establishing rapport.
   
3. PEMS marginalizes staff people from hard-hit communities who may not use computers.
   
4. PEMS drills people with hundreds of questions, but never lets clients self-identify as gay, which means no data on crucial questions about interventions and their impact on men who have sex with men vs. men who identify as gay.
   
5. PEMS questioning means you miss the window of opportunity to reach someone before they nod out in the middle of the now-lengthy intake.

Then there is the question of putting potentially identifiable information about people who sometimes do things that are illegal, may not be in the country legally, and who offend the morality of the present administration, onto federal computers. No matter what the legal safeguards, under the present regime, according to its own repeated and affirmative assertions, legal safeguards for privacy are meaningless and can be ignored by the president.

So this is a tough one. I know why CDC is doing this. They want to know everything they can about their HIV prevention programs and their results, so they can do a better job of preventing HIV. But the people who are out there trying to prevent HIV are afraid the data collection requirements are getting in the way. I don't think there are any evildoers here, I just think we have people down in Atlanta who have spent too much time in school and not enough time actually doing stuff.