News Flash: General Petraeus is a liar.

Why don't they just count people who die from violence? If those numbers goes down, the surge is surging. If the numbers stay the same or go up, the surge ain't surging.

Trying to distinguish violent deaths is like to trying to peel a tomato.

"MNF-I analysts consider the location of the attack -- whether it took place in a mixed area or a homogeneous one -- and the type of attack in order to determine ethnic or sectarian violence."

This is a problem. A real problem, because ethno-sectarian violence is altering the composition of neighborhoods by driving some families out and bringing others in of a different sect. So the violence is altering the criteria by which the violence is measured.

Assuming that this poorly spelled, occasionally ungrammatical document is the real thing, it really isn't terribly useful. It's full of statements that various kinds of death should be considered sectarian, or are likely to be sectarian, or some other weasel wording, but it appears to leave the final determination up to whoever is compiling the stats in a way that could easily lead to cherrypicking. It also has seems to exclude mixed neighborhoods where one sect or another doesn't predominate strongly (even those areas would be prime targets for ethnic cleansing).

Why would anybody be surprised that this administration would make sure the "facts" fit its own goals ... it has been doing just that since Bush was inaugurated.

The most telling facts is that the people who really understand the situation in Iraq indicate that the improved situation in al-Anbar province, the administration's shining star of success, have little if anything to do with the surge. They are a result of diplomatic efforts (read "arming them") with the warlords of the local sunni tribes who are trying to prepare themselves for the inevitable conflict with Iran once the U.S. loses or leaves.

Defining sectarian or dividing out any other type of death is only a way of obscuring the real numbers of casualties. It may be useful for the Military's purposes to understand how/why deaths are happening, so that they can apply the appropriate method/troops/force in each area. But for the purposes of determining whether the surge is working, or even helping, we need to know only the total numbers. Are more Iraqis dying now, for whatever reason, than were dying before the surge? Are more Iraqis dying now than before the war? The difference in those numbers are the result , directly or indirectly, of the US invasion and occupation.

Does anyone else find it odd that the document is undated? That it has no authorship? That it has no Document Number? That it references no specific practices/procedures for which it should be used?

"Which came first, the deaths or the criteria?" The desired conclusion.

"Interestingly, attacks against "same-sect civilians," U.S. forces, the Iraqi government or Iraqi security forces "are excluded and not defined as sectarian attacks." So even though Sunni insurgent groups loathe the Shiite-controlled government, insurgent attacks on it aren't considered sectarian violence."

This is simply astounding. Even with the "placement wounds" story proving to be bogus, this makes up for it in spades.

It seems the point was for them to classify every death in terms that allowed for the maximum number to be discounted.

"There are three types of lies - lies, damn lies, and statistics." (Mark Twain, maybe)

This seems to be a rather transparent attempt to "cook the books" and make something appear to be factual when in reality, it's nothing more than illusion. Dead is dead! Violence is violence! "How" I get to be dead doesn't change the fact that I am dead! The statistics that matter are: 1) How many Iraqis (including civilians, women and children) were killed/wounded today? 2) How many Americans (military, private contractors, civilians) were killed/wounded today? 3) How many foreign fighters were killed/wounded today? Anything less is nothing more than "lies, damn lies, and statistics"

To follow up further on M M’s comment above.

Take a look at page / slide 5 labeled “Ethno-Sectarian Violence” in the above “Petraeus testified” link. It contains four maps with neighborhood shading which shows Majority Sunni, Majority Shia, or Mixed populations between December of 2006 through August 2007. The granularity seems fairly precise.

So what, you may be thinking. Well, these maps have been expertly produced to support a hypothesis. That being that the escalation in troops over the last six months has caused the number of etho-sectarian violence to go down.

Well, what changes, if any, have there been in the make up of the Majority Sunni, Majority Shia, or Mixed population groups during this same period? What about the preceding three, six, twenty or twenty four months?

Could it be that the high level of violence prior to the ‘obvious’ escalation induced victory over etho-sectarian violence shown in “Aug 07” map is due to a significant change in population toward ethnically cleansed neighborhoods rather than the escalation?

Where are the timeline, population tracking maps to show otherwise?

"There is no consideration given to the placement of an entry wound on a murder suspect's head in the tabulation of sectarian violence"

I think spackerman means the placement of an entry wound on a suspected murder victim's head, not on the head of the suspected murderer.

(I'm not used to "murder suspect" referring to a suspected victim instead of a suspected murderer, but I may just be missing a usage of it.)

I second sanityclause's applause.

TPM is the first blog I check into everyday. BEFORE I read the corporate controlled fishwraps.

Can we say Petraeus betrayed us again? It was kind of refreshing to see the Senate actually pass a bill the President wasn't threatening to veto.

sanityclause wrote on September 21, 2007 8:57 PM:

Thanks goodness for TPM and that FOIA still exists.

No disrespect meant to TPM, but why does it take such a little, independent, news team to file a FOIA request on this?

-------

Sanity, what makes you think the other news organizations are interested?

No, seriously...

With this garbly gook insolvent subjective math it is breathtaking in that we were able to land on the moon, harness the atom, or cured diseases.

These retards gene pools should not have evolved. Only in Bush world does failure thrive.

So, how many angels can dance on the head of a pin?

At least we have better information upon which to base that discussion.

Trying to distinguish violent deaths is like to trying to peel a tomato.

Just to let you know, it isn't particularly difficult to peel a tomato. Blanch the tomato for about 25 seconds in simmering water, let it cool, and the peel comes right off.

I agree with sanityclause, are the other media outlets too lazy to file a FOIA. TV outlets never do much actual investigating, but what about the papers?

Good work Spencer Ackerman and TPM. And don’t shy away from reporting about Democratic muck. Muck is muck.

Just more funny numbers.

Here's another whopper.

Oh the Fox Republican officials and politicians made a name for themselves by bashing, bashing and bashing some more the terrible liberal big government tendencies of the Democrats. Here's the frops' dirty little secret, they are twice as much big government big spenders!
Yes. It's the truth. More massive hypocrisy from the frops, who are twice as guilty of one of their "worst" accusations towards Democrats. And to make matters worse the frops are spending all this money on our collective credit. The frops have run our debt into the sky. I'm sick of these fools. Not just their outrageously bad and incompetent performance - also the frops' arrogance and hypocrisy like they walk on water. It's time for them to go. The cold hard truth is that many of them belong in jail, in fact.

http://www.nytimes.com/2007/09/22/washington/22memo.html?_r=1&ref=business&oref=slogin

Greets Liberal Socialists!

Again and again, you libs cannot see the forest for the trees! Yes, war is detestable. However, when I consider your cut-and-run option, all I see is Islamofascists reforming and spreading their cancer further. Don't you get it!They want to KILL YOU and our whole American way of life. Killing them first is the only option we have now. WE must stay the course, root out Al-qaeda and radical Islam with all of its tentacles wherever they exist.

Also, as I have stated before, IF one of your liberal socialist candidates should win the presidency, you are fools if you think he/she is going to pull us out of Iraq and/or defund the war on terrorism...not gonna happen.

And,...get off General Petraeous' back. None of you miscreant writers can hold a candle to him in terms of Honor, Intelligence, Leadership, and Sacrifice that he has made to our country.

Hmmmmm....

Guess there's important stuff on this thread.

To "Troll Patrol":

Not really. I am just a mainsteam conservative red blooded american who hates to see the liberalization of our country and the mush that is being feed to our youth. Your website and lib socialist propaganda reminds me of Marxism that's all.

posted this on Howard Kurtz's Post article about the Post newsroom coming to a stop when Brad Pitt stopped by yesterday; Spencer can thank me when the Post hires him!:

M__M wrote:
While The Post newsroom was hanging with Brad, Spencer Ackerman of TPM was scooping you with an exclusive on the military's tortured and highly subjective definition of sectarian violence (via FOIA he filed two weeks ago). Do you think you can get back to reporting now and FOIA the underlying data for the Petreus report/charts (which would have been helpful about two weeks ago but still important for the record and any future claims of "success"? Maybe Howard could explain why TPM is hustling more than The Post on this. Thanks
9/22/2007 12:38:49 PM

I think that what most people miss is that even if the methodology stinks, and this methodology certainly isn't very scientific, what really matters is that the same methodology is used on both the before and after.

I think that the military started using this method last Spring. If they didn't go back and use the new criteria on the data collected before last Spring, then they are comparing apples to oranges. Depending on the nature of their record keeping, it's probably not possible for them to recompile their older statistics using the new criteria.

At 5:00 pm, Sept. 17 the Pentagon released new data that was clearly at odds with the good Gen. Only Count down reported it. Did you guys take a look at THAT data??

"The actual methodology MNF-I employs has remained unknown. Until now." Unfortunately, it's still unknown. The document, "MNF=I Ethno-Sectarian Violence Methodology," by itself, offers no reassurance that the data CAN BE or HAS BEEN reliably and accurately coded or that it CAN BE or HAS BEEN accurately interpreted.

Let me summarize some of the insightful comments already made here by Mitch, markg8, Lulciftias, paul, MM, TheraP, joeyess, Jim Pharo, Don Holmgren, and Slippery Slope and add a few of my own.

There are two broad issues: 1. the accuracy and reliability of the data Petraeus used and 2. how he interprets it.

A. Issues involving the accuracy and reliability of coding Ethno-Sectarian Violence (ESV) incidents

1. What you've been given is an outline description of various factors involved in categorizing murders into ESVs, but not the actual detailed coding instructions. If this document IS the only document used to guide coding of incident descriptions, we're dealing with a pretty sloppy research crew. It is unlikely anyone using only this document would be able to easily code all of the raw incident descriptions (e.g., what do you do if some of the information is missing? what do you do if there is ambiguous or conflicting information in the incident description? how do you code it if it's a "draw?" etc. etc.). Because of this, it is unlikely that two independent judges--relying solely on the MNF-I document--would code all of the descriptions in the same way. Unless you have a coding system that produces high agreement between independent judges, the data is not considered to be useful for analysis. There must be more detailed coding instructions and we need to see them.

2. Another major piece of missing information keeping us from really assessing this methodology is the "raw data." The raw data in this case would be the incident descriptions gathered from "Coalition reports (SIGACTS)" or--in the case of murders, executions, and kidnappings--from “coalition and host nation” reporting. Either a written incident description or a combination of incident description and other attributes (codes for location, type of violence, etc.) In order to double check the coding of each incident as being an ESV, not-an-ESV, or "indeterminate" (...did they even use this third coding category?) you would need both clear coding instructions AND these incident descriptions. Without examples of actual incident descriptions, we can't know or
assess potential problems and issues in coding this data.

3. We are also missing the actual coded data file (which is probably in some spreadsheet-like format). The "coded data" would be a list of incidents (probably with some sort of numerical identifier), coded characteristics of these incidents (whether it was an SVIBD, IED, or murder, where the incident occurred, the gender and age of the victim, etc.) and the crucial ESV coding (e.g., 1="yes," 2="no," 3=?, etc.). This coded data is probably what the charts and analyses Petraeus used were generated from.

We would need all three types of information--detailed coding instructions (surely more than what they have provided), the incident descriptions, and the coded incident descriptions to
thoroughly assess the accuracy and reliability of this research methodology. What you have received with this MNF-I document is a very general and inadequate description of only one of these three types of information.

4. The primary suspicion when any researcher will not release their raw data (or at least a sufficient sample of it) to be scrutinized by other, independent parties is that the data, will not, in fact, withstand the scrutiny. If the MNF-I really has legitimate reasons to classify the set of raw event descriptions on which their public charts are based, they could at least provide clearance to several reputable, trusted outside researchers to double-check the coding and tabulation of the incident descriptions (without having to release the data to a wider audience). This independent verification could be done quickly.

5. The MNF-I document does not tell us whether this "methodology"--the procedures for coding the incident descriptions and the methods used to create and gather the incident descriptions--was reasonably consistent across the pre-surge and surge-periods. If it wasn't, then we shouldn't be trying to use this data at all.

6. As mentioned in #1 above, inter-rater reliability is one of the most crucial things we need to know about this data (i.e., if two coders look at the same violence description and apply the same MNF-I definitions, how often will they arrive at the same judgment or "coding"--often enough to qualify the data as being reliable?). If I were on an academic panel reviewing this research, the inter-rater coding reliability would be one of the most critical pieces of information I would need to know before we can even proceed with using this data or not. The MNF-I document presents no information about the reliability of the procedures used to code this data.

B. Issues involving the meaning or interpretation of the data

Even if the data coding procedures passed the accuracy and reliability hurdles described above, we would then have the matter of determining what the data really represents...what it means, how we should interpret it, what conclusions we can draw.

7. Excluding Sunni attacks against Shi'a-dominated Iraqi Police and Iraqi Security Forces was a major decision in defining the data to be analyzed. No reason or rationale is given for this
exclusion in the MNF-I document. As several commenters have pointed out, this surely excludes many legitimate ESV incidents from being counted. If I were on a committee reviewing this as an academic research thesis, I would ask to see what the data looks like when ESVs are defined in various different ways. This is how we build an understanding of and confidence in this type of data. Does it look the same from a lot of different but similar perspectives, and if there are differences, what can these
differences tell us about the phenomena were looking at. Looking at ESV data that includes incidents targeted at Iraqi Police and Iraqi Security Forces would provide for a richer analysis and greater confidence in making conclusions. I suspect that these various other analyses using other, broader definitions of ESVs
were performed...but have not been presented because they don't reinforce the results from the restrictive civilian-only incident data set. As a normal procedure in analyzing this kind of data, you would start with a broad definition of the phenomenon and work your way down to a very conservative and restrictive analysis of the data. What would these other analyses look like and what would they tell us?

8. As other commenters have noted, another factor potentially undermining our ability to make clear conclusions from Petraeus's research is the problem of population shifts due to ethnic routing and refugee flight over time. As fewer and fewer Sunni's remain in Shi'a neighborhoods (and vice versa), the absolute number of potential ESV victims in that neighborhood inherently decreases. This alone could account for some or all of the reported decrease in ESVs over the last year. There are various ways to take this population shifting factor into account and there is no evidence in this document that any attempts were made to do so. One way to
make the data more comparable across time and population shifts is to look at and compare incident counts to the per capita ethnic population of that neighborhood at the time of the incident. For example, if an ethnic neighborhood of 75,000 Shi'as and 25,000 Sunnis has 40 Sunni ESVs during one month and then later this same neighborhood with 80,000 Shi'as and 5,000 Sunnis has 20 Sunni ESVs, Pretraeus' research would report that as a 50% drop in ESVs for that neighborhood. The actual population proportionate rate of ESV violence, however, would have increased from 1.6 per thousand (40/25,000) to 4 per thousand (20/5,000)--a 250% increase in the rate of ESV violence. This is just one example of how adjusting this data by other population factors (and other variables) may dramatically change the interpretation and conclusions. The fact that neighborhood populations have shifted (and declined across Baghdad and Iraq as refugees flee the capital and the country) was apparently ignored in the analysis Petraeus choose to present. For simplicity's sake he presents raw numbers (death counts) but this is too simple. I believe that the omission of taking the population shifts into account would be viewed as being a serious or even fatal flaw by an academic research review committee.

On page 4 of the charts accompanying Petraeus' testimony to Congress is a set of maps purporting to show the change in "density of violence" in Baghdad across four intervals from December 2006 to August 2007. There is no indication that these maps are based on ESV data that has been adjusted by changes in population/mix, making the use of the word "density" misleading. New maps based on population-adjusted data may look quite different--e.g., the yellow-to-red violence areas in the December 06 map may shrink and the areas in the August 07 map may expand when adjusted for true population. We need to see these maps, too.

9. Even if the data passed all of the accuracy and reliability hurdles, was adjusted for population, etc., and still showed a marked decrease in ESVs (as narrowly defined by the MNF-I) over the last six months, it wouldn't automatically lead to the conclusion that it was the military surge that was responsible. It might very well be due to our efforts to arm the local Sunni
warlords in places like al-Anbar province—an action that might actually increase ethno-sectarian violence in the long run.