Paul and me
Back in mid-March, the BBC noted the 3 year anniversary of the ousting of Saddam from Iraq with an article by Paul Reynolds. A couple of TAE readers directed me to it at the time, noting the fact that Reynolds highlighted Juan Cole's dismal portrayal of the current situation in Iraq. (Regular TAE readers will be familiar with both Cole and the respect the BBC seems to afford him.) I confess that I did not pay particular attention to the article at the time, and only had the occassion to focus on it yesterday. Wholly apart from the use of Cole, TAE noticed that Reynolds also publicized the infamous, and discredited, Lancet claim that the invasion had resulted in the deaths of 100,000 Iraqis. Wrote Reynolds:
It was in fact an extrapolation based on an extremely small sample of deaths, a loose and unverifiable method of gathering data, and indeed the headline 100,000 figure is not even an accurate characterization of the study's results. In fact that is simply the approximate mean between an upper limit and a lower limit which bracketed the study's 95% confidence level. Big deal, you might say. If the lower limit was 90,000 and the upper limit was 110,000, then 100,000 is an accurate enough figure to give a meaningful picture, isn't it? Perhaps. But in fact, owing in part to the small sample size as well as other statistical short cuts used because of the difficulties and dangers involved in gathering data, the 95% confidence interval was absurdly wide...8,000 to 194,000. This is so wide as to render the numbers useless. As Fred Kaplan put it in his critique of the study in Slate:
Wondering how such discredited data could not only find its way into his article, but be presented with nary a word about its deficiencies and the controversy surrounding it, TAE put some questions to Paul Reynolds on this issue via e-mail, and, to his credit, he was more than willing to respond. Note, however, that I say he "responded", because, as you will see, to characterize most of his responses as "answers" wouldn't really do them justice.
Mr. Reynold's responses are in blue.
FROM: TAE
TO: PR
Were you aware of the methodology used by the Lancet study and the details of it when you included it in your piece?
Do you think the fact that the study had a 95% confidence interval of 8,000 to 194,000 is irrelevant when trying to judge the usefulness and meaning of the 100,000 figure?
If not, why didn't you include that information for your audience?
Do you think you have a responsibility as a journalist to weigh the credibility of claims made by others before reporting them, and to present information that bears on that credibility if you decide to promote the claims? Or is your responsibility fulfilled by simply reporting factually that, for example, claim A was made by person X, with no regard for the credibility of either A or X?
FROM: PR
TO: TAE
It was a reference only. It was not 'promoting claims'. The arguments about the Lancet report are well known.
FROM: TAE
TO: PR
I take it from your response (ie the arguments were well known) that your answer to my first question is that you were aware of the methodology and details of the study prior to including the reference to the study in your article.
Am I to understand it to also mean that you didn't include the relevant information about the study because you assumed it was already well known by your audience? If so, why would you include even the 100,000 figure, as that must also have been just as well known? If not, then, again, why didn't you include the information?
Finally, will you be answering my last question, or should I assume your on-the-record response is "no comment"?
FROM: PR
TO: TAE
You can use it all [on TAE], including my suggestion that you fight it out with MediaLens!
(The reference to Media Lens referred to the fact that Media Lens, an apparently left-wing critic of the media, had taken Reynolds to task for a different aspect of the same article. He had previously advised me to engage them in debate.)
FROM: TAE
TO:PR
I didn't ask if I could use it. I asked a) if my understanding was correct, and b) if I should assume your answer to the questions you haven't answered is an official "no comment". If you'd like, I can repeat the questions that I think I haven't yet gotten a response to.
FROM: PR
TO: TAE
Yes, I understand the arguments about methodology. There is in fact a long correspondence involving Lancet author Les Roberts on MediaLens. It would be hard not to know the arguments. I simply used the Lancet report as illustrative of the problem over figures. I was not in this piece dealing with the row over its findings which is well known, as I said. This is always a problem with blogs and you are not alone in this. We get it from the left as well. If we do not pause in an article about something else and deal fully with the rows over one particular point, all hell breaks loose and we are accused of ignorance or bias or both! You can quote me on that.
FROM: TAE
TO: PR
Do you think that, as a journalist, you have a responsibility to weigh the credibility of claims before you present them in your articles? Or does it suffice simply to report factually that person A claimed X, with no thought or reference as to the credibility of either A or X?
If the former, isn't it fair to assume that you have made the judgment that the claim that the war in Iraq has cost 100,000 Iraqi lives is credible? If not, why not?
If you believed that the controversy over the 100,000 figure was so widely known that you needn't offer it as context, why did you feel the need to discuss the problem of counting Iraqi deaths at all, since logically that must be at least as widely known?
FROM: PR
TO: TAE
In fact Scott, the 100,000 is no longer the upper limit. There are claims that it should be much higher! See Les Roberts http://www.alternet.org/story/31508/
Am I supposed to include his now higher estimates as well? I should do so, according to your rules.
Of course I weigh credibility. I am currently engaged in a hot debate with The Cat's Dream about why I reported on the Iraqi documents at all, the suggestion being that they were unreliable. In the case of the Lancet article, the arguments are well known. You ignore the point I made.
FROM: TAE
TO:PR
Since you do weigh credibility, and since you passed on the 100,000 Lancet claim without mentioning any caveats, is it fair to assume that you find that figure credible? If not, why not?
Far from ignoring your point about the arguments being well known, I have taken it on board and realized it leads to a further (and so far unanswered) question: Again, if you assumed that your audience already knew about the Lancet controversy, why did you feel the need to inform your audience about the Lancet figures themselves, for surely they must also have known those, too?
FROM: PR
TO: TAE
I neither found the figure credible nor incredible It was simply a figure. I reported it. . What's the problem?
As for not mentioning the figure at all, that is a bizarre suggestion.
So, what have we learned from this exchange? We've learned that "of course" Mr. Reynolds weighs the credibility of information before passing it on to his readers...except, that is, when he doesn't, as in the case at hand.
We've learned that he thinks there is no problem with reporting a highly disputed claim without either a) establishing in his own mind that the claim is credible or b) making any mention whatsoever that the claim has been greatly disputed.
We've learned that, although he believes the controversy surrounding 100,000 figure was so widely known that noting it for the sake of his audience was unnecessary, he also thinks it would have been "bizarre" to assume that the very same audience was equally aware of, and therefore in no more need of reminding about, the 100,000 figure itself.
Finally, while Paul should be commended for taking the time to engage his critics rather than taking the easy course and simply ignoring them (as many of his colleagues do), I think we've also learned that Michael Howard has nothing on BBC reporters when it comes to avoiding direct answers to direct questions.
Thousands of people have died. The true number of Iraqi deaths is not knownIgnore, for the moment, the bizarre use of language which has it that an anti-war organization established for the precise purpose of trumpeting the number of civilian deaths is "admitting" to something which it proclaims in no uncertain terms in its FAQ. At least the IBC figures can be said to be based at least to some extent on hard, verifiable data. The Lancet figure of 100,000, on the other hand, has been well-debunked.
and even the Iraqi Body Count figure -- compiled largely from news reports -- of somewhere in the mid 30,000s is criticised as a possible underestimate and admitted by IBC to be a baseline. The British medical journal The Lancet suggested a figure of about 100,000 back in October 2004.
It was in fact an extrapolation based on an extremely small sample of deaths, a loose and unverifiable method of gathering data, and indeed the headline 100,000 figure is not even an accurate characterization of the study's results. In fact that is simply the approximate mean between an upper limit and a lower limit which bracketed the study's 95% confidence level. Big deal, you might say. If the lower limit was 90,000 and the upper limit was 110,000, then 100,000 is an accurate enough figure to give a meaningful picture, isn't it? Perhaps. But in fact, owing in part to the small sample size as well as other statistical short cuts used because of the difficulties and dangers involved in gathering data, the 95% confidence interval was absurdly wide...8,000 to 194,000. This is so wide as to render the numbers useless. As Fred Kaplan put it in his critique of the study in Slate:
Imagine reading a poll reporting that George W. Bush will win somewhere between 4 percent and 96 percent of the votes in this Tuesday's election. You would say that this is a useless poll and that something must have gone terribly wrong with the sampling.Unless, apparently, you are a reporter for the BBC, in which case you think it essential knowledge for your audience.
Wondering how such discredited data could not only find its way into his article, but be presented with nary a word about its deficiencies and the controversy surrounding it, TAE put some questions to Paul Reynolds on this issue via e-mail, and, to his credit, he was more than willing to respond. Note, however, that I say he "responded", because, as you will see, to characterize most of his responses as "answers" wouldn't really do them justice.
Mr. Reynold's responses are in blue.
FROM: TAE
TO: PR
Were you aware of the methodology used by the Lancet study and the details of it when you included it in your piece?
Do you think the fact that the study had a 95% confidence interval of 8,000 to 194,000 is irrelevant when trying to judge the usefulness and meaning of the 100,000 figure?
If not, why didn't you include that information for your audience?
Do you think you have a responsibility as a journalist to weigh the credibility of claims made by others before reporting them, and to present information that bears on that credibility if you decide to promote the claims? Or is your responsibility fulfilled by simply reporting factually that, for example, claim A was made by person X, with no regard for the credibility of either A or X?
FROM: PR
TO: TAE
It was a reference only. It was not 'promoting claims'. The arguments about the Lancet report are well known.
FROM: TAE
TO: PR
I take it from your response (ie the arguments were well known) that your answer to my first question is that you were aware of the methodology and details of the study prior to including the reference to the study in your article.
Am I to understand it to also mean that you didn't include the relevant information about the study because you assumed it was already well known by your audience? If so, why would you include even the 100,000 figure, as that must also have been just as well known? If not, then, again, why didn't you include the information?
Finally, will you be answering my last question, or should I assume your on-the-record response is "no comment"?
FROM: PR
TO: TAE
You can use it all [on TAE], including my suggestion that you fight it out with MediaLens!
(The reference to Media Lens referred to the fact that Media Lens, an apparently left-wing critic of the media, had taken Reynolds to task for a different aspect of the same article. He had previously advised me to engage them in debate.)
FROM: TAE
TO:PR
I didn't ask if I could use it. I asked a) if my understanding was correct, and b) if I should assume your answer to the questions you haven't answered is an official "no comment". If you'd like, I can repeat the questions that I think I haven't yet gotten a response to.
FROM: PR
TO: TAE
Yes, I understand the arguments about methodology. There is in fact a long correspondence involving Lancet author Les Roberts on MediaLens. It would be hard not to know the arguments. I simply used the Lancet report as illustrative of the problem over figures. I was not in this piece dealing with the row over its findings which is well known, as I said. This is always a problem with blogs and you are not alone in this. We get it from the left as well. If we do not pause in an article about something else and deal fully with the rows over one particular point, all hell breaks loose and we are accused of ignorance or bias or both! You can quote me on that.
FROM: TAE
TO: PR
Do you think that, as a journalist, you have a responsibility to weigh the credibility of claims before you present them in your articles? Or does it suffice simply to report factually that person A claimed X, with no thought or reference as to the credibility of either A or X?
If the former, isn't it fair to assume that you have made the judgment that the claim that the war in Iraq has cost 100,000 Iraqi lives is credible? If not, why not?
If you believed that the controversy over the 100,000 figure was so widely known that you needn't offer it as context, why did you feel the need to discuss the problem of counting Iraqi deaths at all, since logically that must be at least as widely known?
FROM: PR
TO: TAE
In fact Scott, the 100,000 is no longer the upper limit. There are claims that it should be much higher! See Les Roberts http://www.alternet.org/story/31508/
Am I supposed to include his now higher estimates as well? I should do so, according to your rules.
Of course I weigh credibility. I am currently engaged in a hot debate with The Cat's Dream about why I reported on the Iraqi documents at all, the suggestion being that they were unreliable. In the case of the Lancet article, the arguments are well known. You ignore the point I made.
FROM: TAE
TO:PR
Since you do weigh credibility, and since you passed on the 100,000 Lancet claim without mentioning any caveats, is it fair to assume that you find that figure credible? If not, why not?
Far from ignoring your point about the arguments being well known, I have taken it on board and realized it leads to a further (and so far unanswered) question: Again, if you assumed that your audience already knew about the Lancet controversy, why did you feel the need to inform your audience about the Lancet figures themselves, for surely they must also have known those, too?
FROM: PR
TO: TAE
I neither found the figure credible nor incredible It was simply a figure. I reported it. . What's the problem?
As for not mentioning the figure at all, that is a bizarre suggestion.
So, what have we learned from this exchange? We've learned that "of course" Mr. Reynolds weighs the credibility of information before passing it on to his readers...except, that is, when he doesn't, as in the case at hand.
We've learned that he thinks there is no problem with reporting a highly disputed claim without either a) establishing in his own mind that the claim is credible or b) making any mention whatsoever that the claim has been greatly disputed.
We've learned that, although he believes the controversy surrounding 100,000 figure was so widely known that noting it for the sake of his audience was unnecessary, he also thinks it would have been "bizarre" to assume that the very same audience was equally aware of, and therefore in no more need of reminding about, the 100,000 figure itself.
Finally, while Paul should be commended for taking the time to engage his critics rather than taking the easy course and simply ignoring them (as many of his colleagues do), I think we've also learned that Michael Howard has nothing on BBC reporters when it comes to avoiding direct answers to direct questions.
16 Comments:
Let me say first (ignoring the mispelling of the possessive of Reynolds as Reynold's -- I have suffered from this all my life and am used to it, though Scott should know better) that Scott has left out the balancing comment on Iraq to that of Juan Cole whom he highlights. I also quoted an ex US officer; in fact I quoted him first. This is part of what he said: "During a recent visit to Baghdad, I saw an enormous failure. On the part of our media. The reality in the streets, day after day, bore little resemblance to the sensational claims of civil war and disaster in the headlines."
In a classic piece of partial selection by a blogger, Scott has simply left out this reference!
Am I surprised? Not at all.
Instead he choses to concentrate on my reference to the Lancet report. I leave you to judge. This is what I said: "Thousands of people have died. The true number of Iraqi deaths is not known and even the Iraqi Body Count figure -- compiled largely from news reports -- of somewhere in the mid 30,000s is criticised as a possible underestimate and admitted by IBC to be a baseline. The British medical journal The Lancet suggested a figure of about 100,000 back in October 2004."
I suggested that he join the leftwing blog MediaLens in the debate it is running on tnis (blog on blog) but he appears he have declined.. I wonder why.
Paul Reynolds
BBC ONline
Paul,
Your criticism is really quite bizarre. Why would I include the "balance" you provided to Juan Cole's comments when I didn't even include what Juan Cole himself said?
I only mentioned Cole in order to credit the TAE readers who brought your article to my attention, and to point out that Cole seems to get a lot of respect at the BBC. If the ex US soldier whom you quote gets an equal amount of respect and attention, I am unaware of it. Does he?
Quite why you find it questionable that I focused on the part of your article I found worthy of criticism, and not something else, is quite beyond me.
Also strange is your decision to quote back the preicise extract which I had already quoted in the body of my post. Do you suppose people will find it easier to judge your words when you replicate them as opposed to when I do? Very odd indeed.
Lastly, regarding your insinuating "wonder" at my failure to take on your suggestion to engage with Media Lens, as you well know from our private correspondence, I have quite a lot going on in my life away from TAE at the moment. Indeed, as the decrease in posts recently attests, I am finding it difficult to maintain TAE with the same output as in the past. So the fact that I did not immediately pounce on your suggestion to enter yet more debate elsewhere ought not be surprising.
SC
PS - In light of my agreement to "tidy up" your self-proclaimed "often messy typing" when posting your e-mail responses, your decision to highlight my own typo (Reynold's/Reynolds') is remarkably ungracious. It has been noted.
Well Scott I apologise for mentioning the wrong spelling of a name ending in -s. I did so because normally your typing is excellent - far better than mine - so I assumed that this spelling was deliberate and not a typo. I am sorry. Thank you for tidying up my errors in our previously private exchanges.
I do think that if you comment on the use of Juan Cole, you should have also said that I quoted an ex US officer as balance. Balance applies to critics as well as to the original articles in my view.
Paul
The fact that the BBC uses numbers that have been completely debunked show how desperate they remain in bringing down the Blair government and, by extension, the Bush administration.
It is plainly sickening.
Simon Lazarus
"Joobo"
www.joobo.blogspot.com
Any media outlet that relies on Juan Cole and the Lancet for input to its Iraq stories makes itself a joke. I am surprised Mr Reynolds (no apostrophe) has stooped to this level. Cole is a busted flush, and so is the Lancet method of estimating. Mr Reynolds knows this, but still prints them.
Maybe Mr Reynolds or someone at the BBC will get round to FACTUAL reporting - eg that a million or two million Shi'ites made a long pilgramage last week, without the dozens of deaths that have disfigured the pilgramages of 2004 and 2005. Where is that civil war the BBC keeps trumpeting ?
Maybe not, but I on the other hand do understand the statistics of random sampling, and while I don't have time or space here to explain, I regard the Lancet report as both flawed and blatantly political.
Here's one that's a bit easier to understand. The paper says:
"The fact that more than half the deaths reportedly caused by the occupying forces were women and children is cause for concern."
Deaths from violence post invasion are given:
Children 28
Men 38
Women 5
11 were not attributed to coalition forces. Assuming they're all men, that gives women+children = 33, men = 27. Fair enough, you might think, until we look at the number of men, women, and children in the sample.
Men 2220
Women 2233
Children 3084
So they sampled 5317 women and children, and 2220 men!
If you exclude Falluja, the deaths are 4 children and 2 women to 13 men. If you again subtract 11 from the men you get a 6:2 ratio, but the same objection applies, plus you're now trying to project from 8 deaths to draw conclusions about the entire coalition operation.
Thus, while it is technically true the "deaths reportedly caused" are > 1/2 women and children, it's a somewhat dishonest statement to make. The following (spurious) discussion of Geneva convention obligations makes its political intentions obvious.
All the stuff about bootstrapping not being an appropriate technique for heavily correlated undersampled data is a lot harder to explain to the uninitiated, but basically means that as wide as their confidence interval was, it probably should have been much wider.
But I don't expect to persuade anybody, so I won't waste my time.
PS. I like the blog, by the way.
At least the IBC figures can be said to be based at least to some extent on hard, verifiable data
But does anyone seek to verify the data?
It is compiled from media reports, perhaps getting their information from dubious sources.
The media will not return to double check the initial information. And we know how unreliable initial reports can be - even if they become the accepted wisdom.
e.g. Jenin massascre, Bush lied when denying he was warned on breach of levees. Even the 45 minute debacle, where it would appear that initial reports of doom resulted from the media having only read the foreword & looked at the pictures in the WMD dossier (the greatest pity being that the government accepted the overblown media reaction rather than seeking to correct it).
I have this question for Mr. Reynolds:
You seem quite content in relying on the completely unreliable Lancet number of "100,000 dead" in Iraq.
However, where is your reporting on the alleged 250,000 murdered by Saddam Hussein from 1979 to 2003? Do they not count?
Or, could it be that the left is not as upset that an Islamic leader kills his own people than when Islamic terrorists do it under Western occupation?
Simon Lazarus
"Joobo"
www.joobo.blogspot.com
when Islamic terrorists do it under Western occupation?
The left do not even seem to recognise that the killing in Iraq is done by Muslims, let alone "Islamic" or "terrorists".
All the blood is on Bush'n'Blair's (or today Rice's) hands.
Reading the BBC from across the pond here in Canada, I must say that any unbiased observer could not help but notice the constant anti-Bush theme oozing from most BBC reports. The BBC, in it's political reporting, has long ceased to report news, free of added opinion content. Like most Main Stream Media outlets, its journalists are a cross between socialist politician and reporter; with a good deal of Noam Chomsky thrown in. Everything they do is colored by their personal socialist views, and the thesis of the above post proves this once again. The ducking and dodging of Mr. Reynolds is wonderful sport to behold, but he was not able to stop the full body blows. He should run for political office instead of being a journalist, because he obviously can't keep his own personal views from poisoning his product.
One of my favorite debunkings of the Iraq Civilian death myth is done by Logic Times. For a wonderful debunking of the 100,000 ... 30,000 body count, see:
http://www.logictimes.com/civilian.htm
Paul, I have to say I am very disappointed in your journalism that you mention the Lancet figure without even a passing mention of the fact that it has been widely discredited.
I seriously doubt you would (nor should!) do the same if a US govt report quoted only 1000 deaths!
As a reporter, you not only *present* the facts, but you question their sources!
This kind of sloppy reporting is what leads to repugnant people like George Galloway repeating them as established fact in Senate Committee hearings. If it appears on the BBC, then it must be right! Right?
This kind of reportage goes a long way to undermining the journalistic credibility that the BBC has established over the past decade. There is only so much credit in the bank Mr Reynolds.
-Nigel
futhermore, the fact that you omit criticism of the 100,000 deaths figure says a lot about what imagine the death toll to be.
Avoiding this criticism by saying "Some people think its higher!" is a cop-out to say the least!
-Nigel
Kudos to you, Scott, for standing up to Tim Lambert on his dishonest attempt to paint your blog as something it is not and also for holding his feet to the fire on the Lancet study. We need more honesty in the debate.
Paul Reynolds' point about all hell breaking loose if journalists fail to deal exhaustively with particular points about which bloggers are obsessing seems to me an excellent one.
I'm by no means in awe of the BBC -like any large news organisation they make plenty of mistakes and sometimes display poor judgement. But compared with the endless tail-chasing and petty point-scoring on display in blogs such as this, the Beeb's generally honest and balanced news coverage is a breath of fresh air. Guys like Reynolds are out there doing real and often dangerous news gathering, and I tend to trust their judgement much more than that of people like Scott, who prefer to expend their energies sniping from the sidelines. By the way, what would you have us poor benighted Brits listen to instead? Fox News perhaps?
Scott Dirty Harry" Callahan, if you don't like the news in the UK why don't you just go back to the good ol', home baked, apple pie US of A then?
I'm sure there's lot's of opportunities there to have a go at the fair, balanced, excellent reporting...
Simple solution isn't there? Bye, bye!
Scott,
That's a real unhealthy obsession you have with the BBC. What's wrong did they overlook your employment application to clean out the toilets at BBC HQ?
As a mater of interest do you think SKY News, here in the UK or FOX News back in the States are fair, balanced and are 100% corrct in all they print and say?
Of course they are! They spew about Marvel Superheroes Bush & Blair 24/7. You'll love that won't you?
If you stay in the UK a little longer you'll be able to receive a Knighthood or an OBE from Tony Bliar.
Post a Comment
<< Home