exchange between Les Roberts and John Sloboda on Iraq Body Count and the Lancet

June 19, 2006

Dear Dr. Sloboda,

I was sorry to receive the e-mail below. In dozens of interviews where reporters have delved into the discrepancies of our death tolls over the first 18 months of the war in Iraq, I have attempted (sometimes unsuccessfully) to emphasize my gratitude to IBC. IBC kept the issue of civilian deaths in the public eye for months when insecurity kept me from going to Iraq. I suspected we had the same motives and goals and had hoped that our efforts might complement one another. I believe that I emphasized this in my note to Hamit Dardagan on 10/28/04 and my unanswered e-mail seeking cooperation with you 3 months later. But, given the widely broadcast and personal nature of your comments below, I am compelled to respond.

1) In the HPN report and the MIT Audits of Conventional Wisdom, I do cite Dr. Hoge's fine article on the mental health of returning soldiers. In that article it is reported that interviewed Army soldiers returning had been there 8 months and that (in Table 2) 14% of them reported "Being responsible for the death of a non-combatant". The Marine column of this table is more complex, while it says 28% felt they were responsible for killing a non-combatant, in the methods section their duration of service is less clear. There were two Divisions sampled and one was there for 6 months and one was there for 3 months. Thus, we need to make some assumptions to acquire a daily death rate. These include:
a) the interviewed Marines were in Iraq 4.5 months on average;
b) there were US 135,000 soldiers serving in Iraq during this period;
c) 70% were Army, and 30% were Marines. We felt that this was conservative because other forces (Air Force) would probably have reported higher rates and because it is believed the Marine fraction averaged a little higher;
d) there were 30.4 days in a month.

The 133 number came from Richard Garfield almost 18 months ago. I have not discussed this note or his assumptions with him since then and I just today calculated an almost identical 137 deaths per day based on the assumptions above. Of course Dr. Hoge did not say that this is reported in his article. He is a highly regarded mental health expert and I suspect that tallying death tolls is outside of the scope of his ethical review for the research and perhaps his comfort level as well. Moreover, if he said, "Yes, we reported a civilian death toll six times higher than our Commander in Chief," he would probably lose his job.

The text of the Hoge article implies that an individual being interviewed felt responsible for a death. We shared Dr. Hoge's concern that it is in the best nature of humans to feel individual guilt for something that was done by a group. Given that the bulk of civilian deaths were from artillery and air-strikes according to our study and the NCCI dataset, and that those soldiers would be unlikely to fully witness their damage, this estimate was probably a gross under-estimate. Also note that the Hoge article would record a soldier who thought they killed 5 civilians the same way it would record a single killing. Thus, attributing that 14% of Army soldiers accidentally killed "a" civilian would err toward a low estimate.

2) Please note the Audit article says that the Hoge reference "...suggests an unintentional non-combatant death toll of 133 deaths per day." In the HPN paper the table (6) citing the death toll estimates is headed in bold as, "Violent deaths per day implied". We have never said "reported" or "recorded." We used words such as "suggests" and "implied" to convey the notion that we had some role in interpreting these data.

Both of the HPN and Audit articles were meant to give an overview of the issue of civilian death tolls. All of the time, public health data is collected for one purpose and is later used for another. Most newspaper reports cited in your data set were not collected to show a tally of deaths nationwide. Given that a large sample of returning soldiers reported accidentally killing civilians, and that this rate was quantified from a sample, it would have been irresponsible to have not mentioned the Hoge article in these overviews. Your assertion that we "... have misused the authority of the New England Journal of Medicine and the authors of this July 2004 paper..." is perplexing.

3) Please note that all of the mortality sources cited in the HPN paper did not provide a daily violent death rate. Death rates were not provided by IBC. In each case, assumptions needed to be made to get the provided estimate. The assumptions regarding the Lancet estimate are far, far greater than the Hoge article estimate. The Lancet estimate, for example, assumes that no violent deaths have occurred in Anbar Province; that it is fair to subtract out the pre-invasion violence rate; and that the 5 deaths in our data induced by a US military vehicles are not "violent deaths." It seems strange that of the 5 other references cited, you did not reach out to the authors of the highest estimate, or the articles which required the biggest assumptions, but instead reached out to a US military researcher who is the only one not free to publish or speculate about the number of civilian deaths in Iraq.

4) Most epidemiologists are reluctant to speculate about completeness of their surveillance system before going out and trying to evaluate the system sensitivity. It is easily done! You have publicly speculated about IBC's completeness rates, rates which are unheard of in times of war. According to my colleague Riyadh Lafta (cc:ed above), only about 1/3rd of deaths were captured by the Government's surveillance system in the years before the Coalition invasion (that was based on a conversation with Riyadh and the assumption that there were 24,000,000 people in Iraq dying at a rate of 5/1000/year, or that 120,000 people were dying per year. In 2002, the Gov. of Iraq documented less than 40,000 from all sources).

First, given that there should be 120,000 deaths per year as a baseline and every estimate I have seen implies that there are more violent deaths than peaceful deaths, the IBC estimate seems very low.

Second, Jim Krane from AP News wrote me on March 2, 2006 regarding the appearance that the IBC count might not be complete. (I have cc:ed him above.) He points out that 24,000 violent death corpses have entered the Baghdad morgue since the invasion when it used to be 2-3000 per year before the invasion (thus~17,000 excess deaths as of 3 months ago). I suspect that his suspicion was raised because less than a quarter of the population lives in Baghdad, the violent death rate is lower there than the rest of the country according to our data and NCCI's data, and the morgues traditionally capture a small fraction of deaths. Thus, twice as many deaths in all of Iraq as recorded in the Baghdad morgue suggest IBC is very incomplete.

Finally, we measured the sensitivity of your surveillance system during the first 18 months, we found it was <5%. This is what we generously referred to in the HPN paper as "cannot be more than 20% complete." The Falluja deaths in our data set were recorded by month and in IBC Falluja deaths were not as distinct as elsewhere so that we could not match them. However, among the other 21 violent deaths we encountered in our random sample of 988 households, one was in the IBC data set. Thus, unless you have evaluated the sensitivity of your system from some independent data source, I hope you will temper the statements you make about the complete nature of the IBC dataset.

5% completeness is the norm of newspaper reporting in times of war. (See Patrick Ball's work in Guatemala online with the AAAS) I suspect and hope that the sensitivity has increased over time as systems have improved and the role of major battles with airpower has diminished. But, the speculation in the press that the real number might only be twice the IBC tally is preposterous.

Last October 11th, I was invited at 2 hours notice before a flight left to appear on the BBC program Newsnight with Jack Straw. I called you at that time, hoping to hear that IBC had calibrated the system and to give you the chance to defend the IBC sensitivity before saying on-air that I had found it to be ~5% complete. Because we did not speak, I did not then, and up until now, report our evaluation of your sensitivity in public. I thought I was doing you a favor by calling.

5) As for your "Speculation is no substitute" paper, I discussed it with some of my coauthors when it arrived. We decided that it was so devoid of credibility, and so laden with self-interest rather than the interest of the Iraqis, it did not merit a response.

6) It is unfortunate that the HPN paper had the wrong dates for the 17/death per day estimate of IBC. I believe that rate was from 2003. That was a mistake on my part. If we revise the dates and rate to cover the first 18 months of the war, the change from 17 to 27 deaths per day does not change any conclusions ever presented in any paper or talk. Surveillance is almost always incomplete in times of war compared to surveys, and the conclusion remains the same, the three lowest estimates of the death rate in Iraq are surveillance-based.

There will come a day when the foreign armies will diminish in Iraq and a full census and accounting of this war will take place. I look forward to that day.

I have been asked dozens of times, mostly by epidemiologists, what is IBC's motive to overstate the completeness of their estimate. I have always been able to respond that I do not know. However, the focus of your note below will only serve to bolster the universally unflattering speculation.

Sincerely,

Les Roberts


**************************************************************************


From: John Sloboda
Sent: Sunday, June 18, 2006 5:10 PM
Subject: your mis-citation of New England Journal of Medicine

Dear Dr Roberts,

This email concerns your mis-citation of a New England Journal of Medicine article as well as a related error.

In your paper entitled "The Iraq War: Do Civilians Matter?", published in July 2005 by the MIT Center for International Studies in its series "Audits of Conventional Wisdom", you state that:

"A report in the New England Journal of Medicine in July 2004, based on interviews with returning U.S. soldiers, suggests an unintentional non-combatant death toll of 133 deaths per day."
( http://web.mit.edu/cis/pdf/Audit_6_05_Roberts.pdf page 2)

This statement was repeated in the version of this paper republished on the Alternet web site ( http://www.alternet.org/story/31508/ ) and the estimate of 133 violent deaths per day also appeared in Table 6 of a publication entitled "Intepreting and using mortality data in humanitarian emergencies" authored by Francesco Checchi and yourself, and published by the Overseas Development Institute (ODI) as "Humanitarian Practice Network Paper no 52" (2005) ( http://odihpn.org/documents/networkpaper052.pdf page 30 ). The table is entitled "Estimates of violent deaths per day in occupied Iraq" and the source of the "133 deaths per day" estimate is described in that table as "Mental Health Study, 2004".

In all cases, the sole reference you provide for this figure of 133 is:

Hoge CW, Castro CA, Messer SC et al. "Combat duty in Iraq and Afghanistam, mental health problems and barriers to care. "New England Journal of Medicine. 351 (July 1. 2004): 13-21." (NEJM)

As we pointed out in our paper "Speculation is no substitute: a defence of Iraq Body Count" published in Arpil 2006 ( http://www.iraqbodycount.org/editorial/defended ), and which was copied to you at the time, nowhere in this cited NEJM paper is there any reference to an estimated per-day rate of violent deaths, whether 133 or any other number, and there is nothing in any of your publications to explain how this 133 per-day rate is derived (our discussion of this matter is contained in section 3.4.2 (page 14 of pdf version) and in greater depth in the related Appendix 3.4.2.a (page 38 of pdf version). You simply cite the undoubtedly prestigious Journal as if it contained the facts you claim for it, which it does not, and nor do you explain how you take the data in that article and transform it into the "133 violent deaths per day", which is surely the crucial calculation and which would allow critical appraisal of your "133" number.

We recently contacted the first author of the NEJM paper, Dr Charles Hoge, who replied as follows:

"In no way can our data be used to estimate civilian deaths. We ask two questions related to killing of enemy combatants and civilians on the our anonymous questionnaire that we ask U.S soldiers, but neither can be used to estimate casualty rates. We ask if at any time in the deployment the soldier perceived that he was responsible for the death of an enemy combatant and another similar question pertaining to the death of civilians. Since all members of a team may in some way be responsible during a combat operation these questions can in no way be used to estimate actual civilian casualty numbers. " (email to John Sloboda, dated 8th May 2006)

In summary, you have published a claim, on the basis of the Hoge et al paper, which the lead author of that paper says is unsustainable (just as we had independently argued).

There are two matters of serious concern:

1. You have misused the authority of the New England Journal of Medicine and the authors of this July 2004 paper to promote a claim which has no basis in that study and which is explicitly rejected by the authors of that study.

2. The supposed 133 per-day-rate of civilian deaths is one of several "estimates" used by you and many of your readers to make unwarranted claims about the relative value of different studies of Iraqi mortality, and the likely overall death toll. Your use of this figure, and the use made of it by others, has thus helped to spread confusion and misinformation on a matter which is of the utmost gravity, and where therefore the highest standards of rigour and professionalism are needed from those claiming academic expertise and authority.

We note that you have made no attempt to correct these and a series of other errors since receiving our "Speculation is no substitute" document referenced above, wherein we subjected some of your claims about Iraqi deaths and the sources of information on them to rational and critical analysis. Since the critiques of "amateurs" apparently leave you unimpressed, whether correct or not, we hope that you will show greater regard for the opinion of Dr Hoge, whose paper you have misused.

We also note that, whereas after the intercession of co-author Francesco Checchi you agreed to correct a blatant factor of two arithmetical error about the daily rate of deaths recorded by Iraq Body Count, you have made no discernible attempt to correct the exact same error in online articles authored solely by yourself. Thus even the simple errors accepted by yourself and corrected in ODI 2005 remain unchanged in MIT 2005 as well as in the (probably more widely read) version of the MIT paper reproduced at Alternet.org. We expect that professional standards would require you and other responsible parties to ensure that these errors are corrected forthwith, without further delay.

In the light of the seriousness of this matter, we are copying this to Dr Hoge, as lead author of the mis-cited NEJM study, your co-author Francesco Checchi, the Humanitarian Practice Network of the ODI hpn@odi.org.uk , the MIT Centre for International Studies cis_info@mit.edu , and to the Managing Editor of Alternet tai@alternet.org

Yours sincerely,

John Sloboda (on behalf of the Iraq Body Count Team)

***

UPDATE: A NEW EMAIL FROM IBC'S JOHN SLOBODA

From: John Sloboda john@oxfordresearchgroup.org.uk
Date: Tue, 20 Jun 2006 09:01:33 +0100

Dear Dr Roberts,

Thank you for your email of Monday 19th June 2006, copied below. We only wish to make three comments here:

1. Our letter to you of Sunday 18th June was not "widely broadcast". It
was sent to you, and to those who would be most directly concerned (i.e.
Dr Hoge, your co-author Francesco Checchi, and the three publishers of
the articles in question). We did not copy it to Media Lens or
journalists. That was your choice.

We will therefore continue to correspond with you, along with those we
originally CC'ed, but beyond one-time inclusion as a courtesy in the
present email, we will copy our future emails only to those recipients
on your CC list who explicitly request to be included in further
correspondence on this matter. We fail to see what their involvement is
in this. (Note that we have only added Amy Tarr because
the other email address for CIS at MIT was non-functional.)

2. You have conceded that no mortality estimate, either per-day or
otherwise, was present in NEJM. Any reasonable person reading the MIT
or ODI papers would have gained the false impression that the estimate
you published was provided by the NEJM authors, and endorsed by the
authority of the NEJM. We continue to believe that there is a
responsibility to correct this false impression.

3. To be consistent, if you have corrected the incorrect
17-per-day-rate attributed to IBC in the ODI paper, you should correct
it in the MIT/Alternet article as well.

The other claims you make in your letter are not relevant to the issue
at hand.

Yours sincerely,

John Sloboda (for Iraq Body Count)

**********************************************************************

Gabriele Zamparini's (public) reply to Dr. Sloboda's last e-mail:

I have received this last e-mail from IBC' John Sloboda:

From: John Sloboda
Date: Tue, 20 Jun 2006 09:01:33 +0100
To: Les Roberts
Cc: (...) "'The Cat's Dream'" info@thecatsdream.com, (...)
Subject: Re: mis-citation of New England Journal of Medicine

In this e-mail, Dr. Sloboda also writes:

"We will therefore continue to correspond with you, along with those we originally CC'ed, but beyond one-time inclusion as a courtesy in the present email, we will copy our future emails only to those recipients on your CC list who explicitly request to be included in further correspondence on this matter. We fail to see what their involvement is in this. (Note that we have only added Amy Tarr because the other email address for CIS at MIT was non-functional.)"

Dr. Sloboda should know my "involvement" since he wrote his first email to Les Roberts after my article was published:

Darkness and Light
When international law and science are silenced, the only outcome is darkness

So I fail to understand Dr. Sloboda's words ["We fail to see what their involvement is in this."] and above all I fail to see the reasons why Dr. Sloboda can't see why the public is very concerned on all this story and demand to know. Is not this the same reason behind Iraq Body Count?

Best wishes,
Gabriele Zamparini

***************************************************************************

NEW REPLY FROM LES ROBERTS TO JOHN SLOBODA:

Dear Dr. Sloboda,

I hope that this will be the last of these exchanges as I get depressed at the world debating two year-old data rather than focusing energies on the problems of the present.

In response to your comments:
1) Fine. Those other journalists had contacted me about your "Speculation" paper and I had refused to provide my opinion at the time. They deserved a response given that you had brought other media sources into the discussion.
2) You seem to have misunderstood. The information exists in the NEJM to reasonably calculate a violent death rate with fewer assumptions than are needed to interpret the IBC data set. What I said is that no daily violent death rate was provided by any of the sources cited in my various articles and op-eds. Assumptions and calculations were needed for all. Your focus on the single article written by the US military (and would be most predisposed to objecting to these interpretations) is deceptive.
3) I have written many articles and all have displayed errors in the months or years after publication. Virtually all were grammatical or trivial including this one. The date error I discussed does not alter the conclusions or points made and is not worthy of pestering an editor. I will correct the minor errors in that table before using the information in the future.

Finally, I would ask you to think of the obscenity of your comment three. You have been given evidence that you have been publicly and repeatedly speculating about the number of deaths in Iraq and your speculation is perhaps 1/10th of reality. You have expressed not the slightest contrition or interest in correcting your errors. And...you are worried about correcting the dates in a table to show instead of
being the lowest of 6 estimates for the first 3 years you were only the lowest for the first year! Is this an attempt to serve the people of
Iraq or the people of IBC?

Sincerely,

Les Roberts