How not to question research

What bothers me most about this controversy is that the whole thing began when the authors announced that they were about to release their raw data. Where is the discussion about the evidence in question, rather than historical questions without the numbers.

(I'll admit that the numbers boggle me - I'm not sure how one source could differ from one another by one third to one million deaths, without other groups noticing.

However, I can't help wonder how even 5 teams could interview 38 families during the violence that those teams were reporting.)


Last October, the journal Lancet published a report (available by subscription only and I don't have access) by Les Roberts and Gilbert Burnham claiming that Iraqi citizens suffered hundreds of thousands of "excess" deaths due to the war. Last month (Feb 28th online and in the March 1 issue), Nature published a news article (available by subscription only, excerpts below) critiquing the study:

On paper, the study seems simple enough. Eight interviewers questioned more than 1,800 households throughout Iraq. After comparing the mortality rate before and after the invasion, and extrapolating to the total population, they concluded that the conflict had caused 390,000–940,000 excess deaths (G. Burnham, R. Lafta, S. Doocy and L. Roberts Lancet 368, 1421–1428; 2006). This estimate was much higher than those based on media reports or Iraqi government data, which put the death toll at tens of thousands, and the authors, based at Johns Hopkins University in Baltimore, Maryland, and Al Mustansiriya University in Baghdad, have found their methods under intense scrutiny.

Much of the debate has centred on exactly how the survey was run, and finding out exactly what happened in Iraq has not been straightforward. The Johns Hopkins team, which dealt with enquiries from other scientists and the media, was not able to go to the country to supervise the interviews. And accounts of the method given by the US researchers and the Iraqi team do not always match up.

The authors of the original study have answered and Nature has published it:

In our opinion, your News story about our Lancet paper "Death toll in Iraq: survey team takes on new critics" (Nature 446, 6–7; 2007) has confused the matter rather than clarified it. You outline three criticisms of our work: that there was not enough time to have conducted the survey; that the sampling method suffered from a 'main-street bias'; and that the study team fabricated the data (the last being attributed to anonymous "researchers"). These criticisms have been previously addressed, and have little merit.

On the first point, the 1,849 interviews in 49 days described in our study suggest that 38 interviews had to be conducted each day by our eight interviewers. Although introducing themselves and explaining the confidentiality agreement might have taken interviewers several minutes, the five-question interview would take only a couple of minutes for most households that reported no deaths. The idea that eight interviewers could not conduct a total of 38 interviews in a day is not credible.

Second, we dismiss the suggestion that our sampling over-represented main streets, where car bombs are more likely. As stated in our paper (G. Burnham, R. Lafta, S. Doocy and L. Roberts Lancet 368, 1421–1428; 2006), when excluding the statistically outlying cluster of Falluja from the first report, we estimated 98,000 (95% c.i.: 8,000–194,000) excess deaths versus 112,000 (95% c.i.: 69,000–155,000) over the same period with the second survey. The first survey was done selecting random starting points with a Global Positioning System unit. The second used the random street-selection process, which is being criticized as biased. It rarely occurs in the field that two sampling methods are used allowing for comparison, and here the results are nearly identical. Moreover, there is no plausible mechanism for a significant main-street bias to operate, because only 15% of all deaths are from car bombs and other ordnance, and because most violent deaths are believed to occur away from the home.

Third, as for the accusation that researchers fabricated the data, we are ready, willing and eager to have an established international authority take a sample of the cluster forms and go to the field with our interviewers to verify the findings. Until that time, the Coalition and Iraqi governments' statements that during the first three years of occupation, Iraq's violent-death rate was lower than those of Russia, Estonia, Latvia, South Africa and Kazakhstan remain an implausible contrast with our findings.

When Nature called one of our study members in Iraq and asked if local officials joined them during the survey, that individual later clarified to Nature by e-mail that 'local officials' did not mean local clinicians and colleagues. This was inaccurately reported in the Nature summary along with a statement by our co-author that interviewers often worked alone. These points were wrongly cited as contradictions between the study team members in your News story.

All reports will eventually have "criticisms that dogged the study", if previously addressed criticisms with so little merit are given a voice in the press.