GPRA -- or , it's all Al Gore's fault

Many thanks to Speechless for the bumper stickers. Some of them are fanciful, a couple of them definitely have potential. I’m afraid I can’t give her the extra doctor appointments she wanted, but I can say something about an issue she is interested in: how can we show the effectiveness of social service and behavioral health programs, especially for the most troubled and least fortunate?

It is indeed very difficult. These sorts of programs don’t lend themselves very well to that gold standard, the randomized controlled trial. (See archives.) You can’t randomly assign homeless, mentally ill addicts to either be sheltered and fed, receive counseling and substance abuse treatment, and vocational rehabilitation; or be left out on the street and periodically interviewed by a graduate student to get comparison data.

That is not to say that over time and by dint of much qualitative and quantitative investigation, and trial and error, we haven’t gotten some good evidence. Substance abuse treatment along with case management, peer mentorship, support groups, and vocational services, are quite effective in preventing released prisoners from re-offending and going back to jail. Unfortunately, very few ex prisoners are fortunate enough to receive such services. Mentally ill people and substance abusers who receive intensive community based treatment do have a better chance of being stably housed, holding jobs, and maintaining natural support networks – i.e. having relationships with family and friends. For people who can’t get more well or more functional, obviously having a roof over their heads and some kindness and encouragement make them better off..

Even though your program is effective, it is not necessarily the case that most of your clients are going to be better off after six months than they were at the beginning. People with multiple problems including unstable lives, chronic mental illness and addiction, if left to themselves, tend to get worse over time, not better. Of newly released prisoners, many will re-offend. If you record at intake whether they have used illegal drugs within the past 30 days, most of them will say no because they were in jail. After a few months, many of them will relapse; your program will prevent only some of those relapses. So if you do longitudinal surveys of your clients, you will see many of them looking worse off after six months than they did at intake. But it is likely that fewer of them are worse off than would have been the case had your program not been there for them. Furthermore, addiction is a relapsing/remitting condition. It usually takes multiple attempts to stop, and some people never really do although they may maintain longer periods of sobriety in between episodes of abuse, and so achieve better social functioning.

The federal government, as we all know, has largely gotten out of the business of attacking social problems on a scale that might really make a difference. However, thanks to the efforts of the Congressional Black and Latino Caucuses, and their friends, we do have a few “Targeted Capacity Expansion” and demonstration projects. One of these is the Center for Substance Abuse Treatment’s (CSAT’s) Targeted Capacity Expansion program which funds community based organizations to develop and expand substance abuse treatment aimed at people at high risk for HIV. The program targets underserved minorities, particularly African Americans, Latinos, and Native Americans, all of which population groups have a disproportionately high risk of acquiring HIV and Hepatitis C infection due to substance abuse.

To give credit, or blame, where it is due, thanks to Al Gore’s initiative as Vice President to make government programs more “accountable,” we have to live with something called the Government Performance and Results Act (GPRA). Thanks to GPRA, the grantees of these programs have to report on their clients at intake, and 6 month intervals thereafter, including recent illegal drug use and alcohol abuse, their housing situation, their employment situation, etc. CSAT gets mad if the programs don’t get 6 and 12 month follow-up interviews with 80% of their clients – which is extraordinarily challenging with drug addicts. So the programs have to devote a large portion of their resources trying to keep track of people who drop in and out of the program, may be in jail, back in the crack house, in the hospital, or dead. Or, for all we know, working and living happily and soberly, but in Puerto Rico or Oregon.

What is worse, these numbers just don’t tell us whether the programs are effective or not. The average person could be worse off or better off after 12 months, but without anything to compare them to, what do the numbers really mean? Maybe half your clients are worse off, 25% are better off, and 25% are doing about the same. Does that mean your program is a failure? Certainly not, it might be that without it, 75% of your clients would be worse off. And maybe it really takes 2 years for some of them to make it. But when the Republicans in Congress, and the Bush appointees in the Office of Management and Budget, see these numbers, what do you think they will do?

(And sorry for the long post. Here I am asking for bumper stickers . . .)