Old death in the new Iraq

Okay, the Lancet doesn't come out until tomorrow but you can already get the manuscript of the Johns Hopkins/Al Mustansiriya/MIT study on excess deaths in Iraq here. (PDF) Since we're already hearing that it's just a political stunt, not reliable, etc., I figured I ought to offer my take on it.

As readers of Today in Iraq know, there are daily death tolls reported by the Baghdad morgue, the police, and stringers around the country who feed information to the major news services who have offices in Iraq -- AP, AFP, Reuters, KUNA (the Kuwait news agency) and Xinhua (the Chinese agency) provide the bulk of the available information. Many people assume that the reports they read in the newspaper, usually from AP, or the daily Reuters "fact box" report that many bloggers (including Atrios and Juan Cole) often repost, are more or less complete descriptions of the day's violence. In fact, they don't even come close.

By combining information from all of the available sources, we always come up with at least two or three times as many violent incidents as you will find in any one source. Even so, it's pretty obvious that most deaths by violence in Iraq never get noted by the police, the morgue, or the news services. Out of deference to their masters, the Iraqi authorities generally don't try to count people killed by the occupation forces. Much of the country is off limits to journalists. Many people who are shot dead never end up in the morgue and there certainly isn't any reason for Iraqis to make police reports. (Police or other security forces, or people dressed like them, are responsible for most of the murders in the first place.) Muslims bury their dead quickly and relatively unceremoniously, and, while the health ministry issues death certificates, it has no system for aggregating and reporting vital records data.

So, the researchers set out to estimate deaths by means of a household survey using area probability sampling methods. This is a method used all the time in health surveys. It's a method I have used myself, in fact. To begin, you just need census data -- it actually doesn't even have to be highly accurate as long as any errors are essentially random, or unrelated to your study questions. Then, you pick geographic areas based on probability proportionate to the population they contain. This is usually done in stages. In the Iraq study, they first determined the number of clusters they would select in each province based on population size (Baghdad, with its population of over 6 million, got 12; Muthanna, with a population of 570,000, happened to get none.) Then, towns, blocks, and starting households were selected at random. For each household selected, the 39 nearest houses were also included. This survey had a total of 47 clusters, including 12,801 persons.

The researchers interiewed adult household members between May and June, 2006, to learn about births, deaths, and migration since January 1, 2002. They also asked people to report if an entire neighboring household had been wiped out, to account for households with no-one left to speak for them. They report that for 92% of reported deaths, the respondents were able to produce a death certificate. A substantial omission in the report, I must say, is the failure to state the response rate. The investigators also refer to procedures for substituting areas which were too unsafe to visit. They do not say how often this happened, but if anything, it would tend to bias the results downward.

To arrive at an estimate of total deaths for the country, they simply multiply the deaths in the study population by the appopriate weights for the number of people each cluster represents (i.e., the inverse of the probability that a person living in that province would have been selected). The clustering does not directly affect the estimates, but it does affect the so-called confidence interval. Since people living in a specific area are at greater or lesser risk of violent death than average, the statistical power of the study is less than it would be for a single stage probability sample of 12,801 persons, because of the possibility that the selection of clusters introduced sampling error. Although the manuscript does not discuss the specific calculations that were done to adjust for this, I am willing to give investigators from these institutions the benefit of the doubt that they did it correctly.

It is conventional to report 95% confidence intervals. The researchers find that there is a 95% probability, assuming no systematic biases in their data, that there have been between 426,369 and 793,663 excess deaths from violence among Iraqis since the invasion -- i.e., deaths that would not have occurred had the death rate continued as before. (There were very few violent deaths in Iraq prior to the invasion. The famous mass graves date from the era of the Iran-Iraq war, and suppression of Kurdish and Shiite rebellions associated with that era.) The investigators also estimate that there have been about an additional 54,000 deaths from non-violent causes, mostly in 2005-2006, as Iraq's health care and public health infrastructure severely deteriorated.

The steady increase in violent death rates is quite appalling, from 3.2/1,000/year in March 2003-April 2004; to 12/1,000/year from June 2005-June 2006. Not surprisingly, the deaths are concentrated in Baghdad and the predominantly Sunni Arab areas of the country. The three provinces of autonomous Kurdistan have been peaceful. 31% of violent deaths were caused by coalition forces, 24% by other actors, and in 45% of cases the perpetrators were unknown to the respondents. Even if none of these were caused by coalition forces, it results that U.S. troops have killed about 200,000 Iraqis, with perhaps a modest contribution from the British.

Are these results reliable? They are in fact the most reliable information we have about this subject. Particularly powerful confirmation comes from the very close match in this survey between deaths reported to have occurred in 2003-2004; and the results from a similar study conducted by the team in 2004. That of course had an entirely different sample of households, but used the same methods. People often misunderstand the concept of the confidence interval. It is far more likely that the true number of violent deaths is close to 600,000, than that it is close to 427,000. People also do not understand how a sample consisting of such a small percentage of the population can give us confidence in saying something about the entire population. But that results from the laws of probability, which assure that state lotteries and casinos will always win.

Was the release of this report politically motivated? Possibly the authors made a special effort to get it out before the election, but that has no relation to its truth.

Finally, as I have said many times, Iraq Body Count should go out of business. They are doing positive harm to the reality based community by giving the perpetrators of this world historical crime cover for saying that the death toll is only 10% of what it really is. That is not helping the Iraqi people.

UPDATE: Thanks to a tip from Whisker, here's an article that shows that innumerable violent deaths in Iraq go unreported.