BURYING THE LANCET - Update

In our Media Alert, Burying The Lancet - Parts 1 And 2 (September 5 and 6), we focused on the media response to a November 2004 report in The Lancet which estimated nearly 100,000 excess civilians deaths in Iraq since the March 2003 US-UK invasion.

In Part Two, we cited professor of mathematics John Allen Paulos, who wrote in the Guardian:

"Given the conditions in Iraq, the sample clusters were not only small, but sometimes not random either... So what's the real number? My personal assessment, and it's only that, is that the number is somewhat more than the IBC's confirmed total, but considerably less than the Lancet figure of 100,000." (John Allen Paulos, 'The vital statistics of war,' The Guardian, December 16, 2004)

We noted that we had not found a single example anywhere in the British or US press of a commentator rejecting estimates of 1.7 million deaths in Congo produced by the same lead researcher (Les Roberts) and offering their own “personal assessment” in this way.

We were surprised to receive this reply from John Allen Paulos the following day:

“I liked your piece, MEDIA ALERT: BURYING THE LANCET - PARTS 1 AND 2. I regret making the comment in my Guardian piece that you cite: ‘My personal assessment, and it's only that, is that the number is somewhat more than the IBC's confirmed total, but considerably less than the Lancet figure of 100,000.’ I still have a few questions about the study (moot now), but mentioning a largely baseless ‘personal assessment’ was cavalier. I should simply have stated my doubts about the study's scientific neutrality given what seemed at the time like an expedient rush to publish it.
John Allen Paulos Math Dept, Temple Univ” (Email to Media Lens, September 7, 2005)

We forwarded this email to Les Roberts - lead author of the Lancet report - who replied directly to Paulos:

“Dear Dr. Paulos,
I read your note below with some sadness. FYI, there was a rush to publish as I have said in every major interview I have given.
"A) I have done over 20 mortality surveys in recent years and have never taken more than a week to produce and release a report (because people dying is important) until this article. Thus, this was the least rushed mortality result I have ever produced.

"B) We finished the survey on the 20 Sept. If this had not come out until mid-Nov. or later, in the politicized lens of Baghdad (where the chief of police does not allow his name to be made public and where all the newly trained Iraqi soldiers I saw had bandanas to hide their faces to avoid their families being murdered…) this would have been seen as the researchers covering up for the Bush White House until after the election and I am convinced my Iraqi co-investigators would have been killed. Given that Kerry and Bush had the same attitude about invading and similar plans for how to proceed, I never thought it would influence the election and the investigators never discussed it with each other or briefed any political player.

"C) if you have information about how and why people in New Orleans were dying today, would you rush to release it? The Falluja downfall happened just one week after the study came out and whether you believe the 500 or the 1600 or the 3600 estimates of associated Iraqi deaths, that alone was probably more than will occur from this moment on due to Katrina.

"So, we rushed to get it out, I do not understand why the ‘study's scientific neutrality’ is influenced or the likelihood that the sample was valid, the analysis fair… What does neutrality mean? Do people who publish about malaria deaths need to be neutral about malaria? Yours in confusion and disgust,
Les Roberts (Copied to Media Lens, September 8, 2005)

We also wrote to Paulos:

“Dear John Allen Paulos

Thanks for the kind words, we appreciate them. We also very much respect your willingness to express regret over some of what you wrote in the Guardian. It really is remarkable that so many people have been so cavalier in considering our responsibility for the mass death of completely innocent and completely defenceless civilians.

“But even now you continue to suggest that the rush to publish casts doubt on the accuracy of the report. Surely, again, on such a desperately important issue, this inaccuracy has to be +demonstrated+ through reasoned debate rather than merely suggested in passing. Isn't this also a cavalier comment? According to Les Roberts, the science wasn't rushed at all, but there were compelling moral reasons to publish the report before the US elections.

Best wishes

David Edwards and David Cromwell
The Editors - Media Lens” (September 8, 2005)

Paulos replied to both emails, first to Roberts:

“Dear LR,
You start your note with sadness, which I share, and end it with disgust, which I don't. I wish your study had met a better reception and regret that my column, although by no means dismissive, was more skeptical than it should have been. If your response to well-intentioned critics is disgust, however, then your public relations skills may have been part of the problem.
Best, JAP” (Copied to Media Lens, September 8, 2005)

Paulos then replied to us:

“A suggestion: use Katrina as a news hook and have LR write an OpEd for the NY Times (or Newsweek or some plublication with a huge circulation) explaining sampling, clusters, and the problems associated with counting the dead. Next explain that this was done in the Congo and finally revisit the Lancet findings. The sympathy that Katrina arouses might enable him to get past the political resistance to the Lancet findings (resistance that probably won't disappear until this abominable war ends). He certainly has the standing to write such a piece, and the issue is still very important. I understand now the situation surrounding the studyy's original publication. I also understand LP's anger, but he should lose the vitriol to get such an OpEd published. Good luck. Best, JAP” (Email to Media Lens, September 8, 2005)

Paulos appears to agree that it was inappropriate to offer his “personal estimate” and to suggest that the science behind the Lancet report might have been somehow “rushed” for political reasons.

Roberts Replies To Rentoul

Also in Part Two of our Media Alert, we noted that John Rentoul, the Independent on Sunday's chief political commentator, wrote of the Lancet's 100,000 figure:

"However, this number is only the central point of a range that extends from 8,000 to 194,000. This huge disparity was mocked ignorantly by one American commentator as 'not an estimate, it's a dartboard'. It was also defended, equally ignorantly, by the editor of The Lancet, who said: 'It's highly probable the figure is 98,000. Anything more or less is much less probable.' Both wrong. What the figures say is that there is a 95 per cent chance that the true figure lies between 8,000 and 194,000... It is statistically respectable, which is why The Lancet article passed its peer reviews, but it produces estimates hedged about with great uncertainty.

"And there are good reasons for thinking that the true figure is towards the lower end of The Lancet's range." (Rentoul, 'We should be counting the dead in Iraq, but let's not get the figures out of proportion like this,' December 10, 2004)

Rentoul responded to our alert:

“Dear Mr Edwards and Mr Cromwell
Thank you for your belated interest in the article I wrote in December. I am grateful to Les Roberts for acknowledging that I invited him to reply to the Foreign Secretary in the pages of The Independent on Sunday, which he did. I would invite you to consider the possibility that the 100,000 figure produced by the Lancet study has not been widely accepted because other evidence suggests that it is too high, rather than because of a media conspiracy [sic]. Given The Independent on Sunday’s editorial position on the invasion of Iraq, it seems unlikely that its editors would want to minimise evidence of casualties.
The fact that no one has challenged the Congo study is no more evidence of conspiracy than [former New Statesman editor] Peter Wilby’s say-so is that my book on Blair is ‘reverential‘. The Congo death toll is not controversial. The Iraq one is.
I draw your attention to a survey of Iraq carried out on behalf of the UNDP and published in May this year. The Iraq Living Conditions Survey 2004 covered 21,668 households and was the first in recent years to cover all governorates. Most of the survey took place in April and May 2004, while fieldwork in Erbil and Dahouk was carried out in August 2004. It suggested 24,000 ‘war-related deaths’ of civilians and military personnel in Iraq, with a 95 percent confidence interval from 18,000 to 29,000 deaths. This was in the period of two years before the survey date, which covered the 2003 invasion and one year of the insurgency. Children aged below 18 years comprised 12 per cent of the deaths due to warfare.
I do not think that commentary on surveys such as these should be confined to qualified statisticians. I hope I understand probability well enough to understand that the Lancet study suggested a range of estimates, and that the actual figure could be at the lower end of the range.

Yours sincerely
John Rentoul” (Email to Media Lens, September 8, 2005)

We forwarded Rentoul’s email to Les Roberts, who replied the following day:

“Dear Mr. Rentoul,
Thank you so much for the note below which is the first coherent and appropriate explanation for the rejection of the Lancet results that David Edwards has forwarded.
I believe the study to which you refer was actually done by the Iraqi Government with help from a Norwegian Institute called Fafo. Jon Pederson, who I have only met once but who I think highly of and has a great reputation, led the design and analysis support. I have cc:ed him above to make sure that any errors in my statements here are corrected. The goal of their survey was very broad and it looked at many issues related to housing and development. I believe the survey took on average 88 minutes with only a couple of the questions being related to deaths. Jon expressed to my colleague Richard Garfield that he felt that their survey did not capture all deaths. He repeated this sentiment in an Wall Street Journal Online article last month. In fact, I gather repeat visits to the interviewed houses after the study was over revealed additional deaths not reported during the survey.
In contrast, please note that our survey was totally focused on mortality, that it was conducted almost exclusively by physicians with photo ID’s and who were from a University which is seen as somewhat different than the Government. Also note that I believe the Fafo study was mostly carried out in the Spring of 2004 but was delayed when near completion due to insecurity. Thus, their recall period during the occupation was a few months shorter than ours and missed the most violent months according to our data.
Thus, given that:
a) we looked at excess deaths from all causes and over 1/3rd of those we reported were from increased rates of auto accidents and other aspects of the social disorder,
b) one would have to increase the Fafo number by perhaps 20% or more to have a comparable time period of occupation, and
c) given that the investigators did not set out to measure mortality and believe their survey markedly underestimated deaths,
I think the two surveys are not really very inconsistent. I also realize there was a study that came out with a 128,000 violent death estimate last July. I have only seen the press summaries and would be keen to read the actual report if you have it.
Again, thanks for your apparent logic and open mind (as implied by your engaging Mr. Edwards, and seeking our response to Jack Straw).
Gratefully,
Les Roberts” (Copied to Media Lens, September 9, 2005)

On September 10 this perceptive email to the Independent’s leader writer Mary Dejevsky was forwarded to us from a reader:

“I have been following the ‘debate’ over the Lancet report with some interest. A group of respected statistical experts publish a report based on extensive research in Iraq concluding that some 100,000 civilians have died since the Iraq invasion.

“This figure is then reported in the media (including the Independent) and is then astonishingly statistically criticised by non-experts. ‘Gut feelings’ and hand-waving arguments are not sufficient methods of evaluating a piece of statistical research. Imagine that Nature published a paper suggesting it was now believed that electrons orbit protons in slightly eccentric orbits for such and such a reason. Would you as a journalist feel qualified to attack such a report because it just does not sound right?

“The July 20th article by Kirby and Davies stated that the methodology had been criticised but failed to mention that the critics in mind were actually the government, rather than some independent experts. The source of the criticism should have been made clear

“Either the methods used by the Lancet report authors are sound or they are not. Clearly it is important to know which is the case. The only people really qualified to determine this are statisticians, preferably those with experience in this area. I would strongly suggest that the Independent picks some leading statisticians at random and invites them to review the Lancet article and publishes a summary of their results.

Regards,
Steven Martin MSci (Cantab)” (Copied to Media Lens, September 10, 2005)

The last suggestion was excellent. Another reader wrote to Dejevsky suggesting the Independent might cover these latest exchanges. She responded:

"the people you have to convince are the specialist reporters in this case - not the comment writers,, as it is they who would put the subject back on the agenda. regards, mary dejevsky" (Forwarded to Media Lens, September 6, 2005)

So we wrote to one of the “specialist writers”, Washington correspondent Andrew Buncombe. He suggested we write to Dejevsky!

Conclusion - You May Well Ask!

In the last three Media Alerts, Media Lens has allowed the Lancet authors the first substantive right of reply to criticism - much of it embarrassingly ill-informed - produced by politicians and journalists.

Readers might well ask how on earth it can be that a website operating on a shoestring is left to provide such a vital and basic service to the public. How can it be that powerful, high-tech media corporations with vast financial and human resources have not managed this feat in almost a year since the Lancet report was published? How can they somehow not muster the time, not muster the manpower, not muster the space among the endless tittle tattle, gossip and blather?

It could not be clearer that Les Roberts (like Gilbert Burnham and others involved in the production of the report) is willing and able to produce clear, concise refutations of this criticism almost instantly. As one of the world’s top epidemiologists, Roberts knows what he is talking about, whereas journalists and politicians manifestly do not. The debates are predictably non-contests, and therefore allowing Roberts and colleagues to respond +should+ be a no-brainer.

It speaks volumes about the honesty and independence of our mass media that it has almost never been allowed to happen.

SUGGESTED ACTION

The goal of Media Lens is to promote rationality, compassion and respect for others. When writing emails to journalists, we strongly urge readers to maintain a polite, non-aggressive and non-abusive tone.

Write to Mary Dejevsky
Email: M.Dejevsky@independent.co.uk

Write to Terry Kirby
Email: T.Kirby@independent.co.uk

Write to John Rentoul
Email: j.rentoul@independent.co.uk

Write to Jamie Doward
Email: jamie.doward@observer.co.uk

Write to Alan Rusbridger, Guardian editor
Email: alan.rusbridger@guardian.co.uk

Write to Roger Alton, Observer editor
Email: roger.alton@observer.co.uk

Write to Simon Kelner, Independent editor
Email: s.kelner@independent.co.uk

Write to Tristan Davies, Independent on Sunday editor
Email: t.davies@independent.co.uk

Please send copies of all emails to us at: editor@medialens.org

Part 1 available here
Part 2 available here

This is a free service. However, financial support is vital. Please consider giving less to the corporate media and donating more to Media Lens: www.medialens.org/donate

A printer-friendly version of this alert can be found here for approximately one week after the date at the top: www.medialens.org/alerts
and then, thereafter, in our archive at: www.medialens.org/alerts/archive.php