« previous | MUCK HOME | next »
Today's Must Read
The long-story-short on measuring sectarian violence in Iraq? Don't expect consistent data across U.S. agencies.
The Washington Post's Karen DeYoung gets an in-depth explanation of how the U.S. military's Iraq command, known as MNF-I, tallies sectarian statistics, and notes the discrepancies between how the Pentagon and U.S. intelligence characterize the same data. On Friday, we published MNF-I's explanation of what it does -- and doesn't -- consider sectarian violence.
The analyst in charge of MNF-I's sectarianism database is a chief warrant officer named Dan Macomber, and he freely concedes that there's room for debate over whether or not his methodology is proper or consistent.
In recent months, most of the military's indicators have pointed in a favorable direction. As with all statistics, however, their meaning depends on how they are gathered and analyzed. "Everybody has their own way of doing it," Macomber said of his sectarian analyses. "If you and I . . . pulled from the same database, and I pulled one day and you pulled the next, we would have totally different numbers."Apparent contradictions are relatively easy to find in the flood of bar charts and trend lines the military produces. Civilian casualty numbers in the Pentagon's latest quarterly report on Iraq last week, for example, differ significantly from those presented by the top commander in Iraq, Gen. David H. Petraeus, in his recent congressional testimony. Petraeus's chart was limited to numbers of dead, while the Pentagon combined the numbers of dead and wounded -- a figure that should be greater. Yet Petraeus's numbers were higher than the Pentagon's for the months preceding this year's increase of U.S. troops to Iraq, and lower since U.S. operations escalated this summer.
That's not to say the Pentagon quarterly report's tabulation is consistent, either. As we've noted, the report's tabulation of sectarian killing has fluctuated between editions. In the September edition, for instance, the Pentagon suddenly decided to include vehicle- and suicide-bombings in its sectarian total. According to Macomber, whenever MNF-I sent the Pentagon its sectarian-incident numbers, it always included car bombs and suicide bombs in the count, but the Pentagon, for unexplained reasons, plucked those incidents out. Why the change? "We regularly review our metrics to determine the most informative way to report what is happening in Iraq," a DOD spokesman e-mailed DeYoung.
DeYoung doesn't report that the statistics are being manipulated. Indeed, MNF-I tells her that there's a "current effort to consolidate multiple databases" in Baghdad underway right now. But if, even for admirable reasons of improving flawed methodology, figures previously reported change between reports, or definitions subtly shift, it becomes difficult to independently assess the course of the war.
Intelligence officials, on the other hand, are attempting to be more cautious in their assessments than the military to guard against self-deception:
The U.S. intelligence community considers more than numbers in making its war assessments. "What the Iraqis perceive" about their country and their daily lives "may be more important than what the numbers are," said a senior intelligence official, who discussed the subject on the condition of anonymity. Even so, he said, intelligence officials found contradictions in the available statistics as they wrote last month's National Intelligence Estimate on Iraq, whose conclusions were somewhat less optimistic than the military's."It's not anybody trying to make it come out one way or another way," said the official, who sympathized with the military's need to quantify. But it is important, he said, to determine "what the numbers meant. Who collected them? Why do numbers that come in from this piece of the U.S. government differ from those coming in from another part of the government?"
While both Petraeus and the recent Pentagon report emphasized improved statistics over the past three months, the intelligence community generally declines to declare trends based on data measured in periods shorter than six months to a year. Several senior intelligence officials said last week that most numerical indicators appear to be moving in a uniformly positive direction in the nearly two months since the intelligence estimate's data cutoff -- although they said it is too early to determine definitive trends.
Ultimately, Macomber tells DeYoung, "it's an analyst making an analyst's call." It remains an open question if analysts across the U.S. government can make the same call on the same piece of data.
Comments (11)
M M wrote on September 25, 2007 10:48 AM:Spencer, also not that Macomber started his post in Iraq this February so you don't even have the same person supervising or coding incidents during the two comparison periods.
DallasNe wrote on September 25, 2007 10:57 AM:It is standard practice in business that whenever you change your reporting methodology you restate prior reporting periods so that you are reporting on a consistent basis. One would expect no less from the military and federal government.
Obviously, there is a reason that the government keeps providing apples and oranges reports. All of their reports cherrypick the data to show predetermined results so they would have to not only restate current reports but also all of those that preceeded the current reports.
The object is not facts; the object is politics. Facts simply get in the way.
Peter Stone wrote on September 25, 2007 11:47 AM:Lately I've started to notice how the war promoters use language to confuse and befuddle and a typical method is the incessant use of the "cliche of the day. I don't know know who got this one going (Tony Snow loved it) but it is now required to be used ad nauseum by any official spokesperson when talking about Iraq: "metrics." I make my living as a writer and I have never in my entire life used this word. What the hell does it mean? Traditionally related to meter in poetry, if you want to talk about methods why not just say "methods." Every time I hear some pompous spokesperson say "metrics" my brain seizes up. It's such an aggressively cold, heartless term when you're talking about people who got blown up. Are they trying to make themselves sound oh so precise, like "we're really on top of this. Not only do we have statistics, we've got metrics! We're the insiders and you're not, so back off!"
dogfacegeorge wrote on September 25, 2007 12:22 PM:Let's suppose it's true that the total of violent deaths has remained relatively constant while one subset of that total, sectarian deaths, has trended down. That implies that the other subset, non-sectarian deaths, has trended up.
Now why would that be? Is sectarian violence slowly morphing into criminal violence? And if so, is that a good sign, a sign of the surge's success? Or is it a sign of a devolution to a Hobbesian world?
philosopherking wrote on September 25, 2007 2:30 PM:I find the timing of this change in methodology troubling. The idea that car bombings were excluded from calculations of sectarian violence was always preposterous. Now that Petraeus has given his testimony, the parlor trick has served its purpose. When nosy reporters (if there are any left) ask questions about it, MFN-I can say of course it includes car bombings, and where did it get the idea that they don't (probably from those left wing bloggers whose minds are addled from their hatred of Bush).
from PK (whose mind is addled from his hatred of Bush)
Dadmanly wrote on September 25, 2007 3:00 PM:The Analysts described here are Army All Source Intelligence Analysts, MOS 96B. They staff Intel sections from BN sized units up through Bridge and Divisions and higher HQ commands. In this case, at MNF-I, these soldiers are led by a Warrant Officer.
What they are doing is a form of what the Army calls Battle Damage Assessment (BDA). In prior conflicts, BDA could be as simple as trying to determine the number of enemy killed or wounded based on combat SITREPS, open source reporting, really anything that gave any hint of enemy casualties.
Now, in the glassbowl that is the modern battlefield, in a Counterinsurgency effort, the Army is being pressed by military and civilian leadership to determine civilian casualties, and attempt to assess the nature/cause of those casualties. There are obvious reasons to do this, not least the extreme level of criticism and politicization that is driving both sides of the debate about Iraq. Complaints and criticisms from war opponents is a big reason the Army bothers to derive this kind of detail. That, and the new COIN doctrine suggests that protecting civilian populations can be more vital to success than force protection. (And that’s right.)
Note to Peter Stone. The term "metrics" may seem alien to a writer, but is very familiar to anyone in business or industry (all the rest of us non-artist types). Metrics are those data elements that any organization can use to show results, measure progress against goals and objectives, and suggest trends. Metrics lead to better decision-making.
The Army is actually a little late to the game in focusing on metrics.
You can imagine that in a very fluid, complex situation with sectarian conflict, general lawlessness in places, foreign terrorists, and great differences in security and stability between neighborhoods and provinces in Iraq, finding significant and meaningful metrics to show trends or monitor progress can be daunting.
But it provides great value to military and civilian commanders, suggests some reasonable conclusions, and helps guide future decision-making. As such, it's probably as or more sophisticated than the process major corporations use for new product introductions and other investment decisions.
Despite how it will be parodied and misrepresented in traditional and alternate media.
palm pilot wrote on September 25, 2007 3:39 PM:Right on DadM. Thank you for setting these folks straight. Not being in our line of work they don't know what they don't know and you can't fault them for that. They just have no apparent problem talking about it anyway....oh well. My only problem with the current BDA and MNF-I (aside from the fact that is a dog & pony as noted above) is that we're taking credit for changes not of our doing (whether accurate or not). The trend of violence up or down is not because of the surge, Pretreaus tactical doctrine of insurgent warfare (which by the way we are ignoring), new armor for the troops, more money for the INC, contractors, the ING or anyone else. It is happening because your every day Iraqi is waking up to the fact that all the retoric and posturing by the foreingn insurgency is ultimately a load of BS. It has taken three hard years but it is happening. These average Achmeds are taking it back to the insurgency. The insurgency is admitting as much by stepping up attacks on civilians in totally new regions of the country previsously free of violence. Now AQ is pushing revolt in Pakistan, what happened to Iraq? That's where the great satan is, what gives? The truth is that we have stayed in Iraq long enough to convince the people to grow a testicle or two and take their neighbothoods back for themselves. Sectarian payback will continue, but at least they're getting rid of the forgein element, albeit slowly. Look at the Iranian commandos and their weapon shipment we just scored...that was the result of an Iraqi tip-off. Now if the INC can just follow suit and take a clue from the people. I'm not saying "problem fixed," but there is a light very far down the end of this tunnel.
Mari wrote on September 25, 2007 3:42 PM:Look, if one could just trust these guys your explanation could be helpful. As it is, you know "tons of WMDs, a bomb fixin to blow" as they say in East Texas and the big sign declaring, "Mission Accomplished" on the naval ship, all have wised us up. There was a time we could all go along with your idea. As it is, these shifts in a group who consistently inflate the success when all we have to do is watch TV or get a letter from some GI over there to know the facts, well the old adage, fool me once, shame on you, fool me twice, shame on me applies to all of us.
palm pilot wrote on September 25, 2007 7:51 PM:I thought the old addage was "fool me twice won't get folled again" do the Pete Townsend guitar wind mill as you say that last part...
Rudee Giulani wrote on September 25, 2007 9:17 PM:9/11. 9/11. 9/11. 9/11. 9/11. 9/11. 9/11.
9/11. 9/11. 9/11. 9/11. 9/11. 9/11. 9/11.
9/11. 9/11. 9/11. 9/11. 9/11. 9/11. 9/11.
9/11. 9/11. 9/11. 9/11. 9/11. 9/11. 9/11.
9/11. 9/11. 9/11. 9/11. 9/11. 9/11. 9/11.
9/11. 9/11. 9/11. 9/11. 9/11. 9/11. 9/11.
9/11. 9/11. 9/11. 9/11. 9/11. 9/11. 9/11.
9/11. 9/11. 9/11. 9/11. 9/11. 9/11. 9/11.
9/11. 9/11. 9/11. 9/11. 9/11. 9/11. 9/11.
9/11. 9/11. 9/11. 9/11. 9/11. 9/11. 9/11.
9/11. 9/11. 9/11. 9/11. 9/11. 9/11. 9/11.
9/11. 9/11. 9/11. 9/11. 9/11. 9/11. 9/11.
9/11. 9/11. 9/11. 9/11. 9/11. 9/11. 9/11.
9/11. 9/11. 9/11. 9/11. 9/11. 9/11. 9/11.
9/11. 9/11. 9/11. 9/11. 9/11. 9/11. 9/11.
9/11. 9/11. 9/11. 9/11. 9/11. 9/11. 9/11.
9/11. 9/11. 9/11. 9/11. 9/11. 9/11. 9/11.
9/11. 9/11. 9/11. 9/11. 9/11. 9/11. 9/11.
9/11. 9/11. 9/11. 9/11. 9/11. 9/11. 9/11.
9/11. 9/11. 9/11. 9/11. 9/11. 9/11. 9/11.
9/11. 9/11. 9/11. 9/11. 9/11. 9/11. 9/11.
9/11. 9/11. 9/11. 9/11. 9/11. 9/11. 9/11.
Got it?
aesthete wrote on October 3, 2007 3:01 PM:Almost got it. You should have left out the center column. You would have been left with two towers.