Larry Johnson and the Numbers Game (Updated)

by Demosophist

I had a brief debate with Larry Johnson on the Jawa Report last week, that was recently referenced by an article in Mother Jones Blog. I expanded on the issue briefly here. To summarize, my position has basically been that:

1. The inference that State is somehow avoiding empirical evidence related to its policies is not justified by what we know and what Larry has reported. It's more likely that they are just tired of confounding policy with a report whose only real value is in a fruitless advocacy conflict; and

2. That the increase in attacks, even if specified and counted correctly, is not a meaningful indicator of whether we are "making progress" in the War on Terror, though Larry insists the data demand some negative inference.

In addition, I also suggested:

3. That if the method used to identify a terrorist event were mis-specified, and that portion affected by the mis-specification was the one that increased during the period, or that more incidents are simply being reported (which is a commonplace issue with crime statistics) then we may not actually be looking at an increase. (It's unlikely that this has seriously distorted the numbers, but we haven't checked very carefully.)

But even if we concede that the numbers are largely correct and consistent with previous counts, there are numerous reasons why it may not be qualitatively a very good metric. In fact it appears to even include thwarted attacks, if they happen to kill or injure a small number of people. Should these thwarted attacks really be counted towards a lack of progress?

But regardless of whether we have a qualitatively good metric (and we almost certainly don't), we have probably mis-specified terrorist attacks relevant to the real status of the war, and that's the central issue.

If Armed Liberal is correct (and it appears he is) that 30% of the attacks were in Iraq, and Andrew Cochran is correct that almost half occurred in Kashmir (and that's not even in dispute), then what we have are two localized conflicts accounting for more than 75% of the attacks in 2004, both of which are being lost by the terrorists. (Pakistan and India are close to agreement over Kashmir.)

Just how does this translate into a "lack of progress?" Moreover, is Larry's conclusion that "... the news is not good for U.S. efforts to contain and reduce the threat of international terrorism" even relevant? I submit that the fact that he's a specialist may be clouding his perspective. The real issue is whether we're winning. Wouldn't that be the relevant "good news?" And what reason do we really have to doubt that we're winning?

Futhermore, consider that if we are indeed winning against a terrorist movement, that's just about the first time it has ever happened! Up until recently "asymmetric warfare" of the sort that targets civilians as a deliberate strategy, has been an enormously successful strategy. Thus, it's hardly surprising that a winning strategy may not look much like a walk in the park.

If we'd used the same sort of metric in WWII to gauge success or failure we'd have observed higher and higher losses with each passing year and a big jump at the Battle of the Bulge and Iwo Jima, which were when we actually began to close on the enemy. It was certainly "not good news" that we lost a lot of fine men on Iwo Jima, but what does that have to do with the crux of the matter? Would it have been better to leave the strategically important island in the hands of the Japanese?

Incidentally Princeton has a series of webcasts of its ongoing Colloquium on Public and International Affairs. If readers go to that link and scroll about halfway down the page there's a synopsis and several links to the video of a panel discussion entitled Measuring Success in Combating Terrorism. The speakers in that discussion include Larry Johnson, but they also include Raphael Perl from the Congressional Research Service and Peter Probst, a former CIA officer. It's an excellent panel, and well worth the time spent to view it.

My feeling is that if Larry Johnson had paid more attention to the other speakers on that panel, and especially Raphael Perl, we wouldn't be having this debate. Perl is especially eloquent about the fact that we don't currently have any metrics that can tell us not only whether, but the degree to which, we're winning. And we need them. To that extent I agree with Johnson, that the numbers are important. But I don't agree that these numbers tell us much.

One of the biggest deficiencies in our recording and analysis is that we have no measure of the "quality" of the attacks, nor do we even bother to sort them by scale. In fact, we often don't even distinguish between attacks that are carried out successfully and those that were pre-empted! But the greatest deficiency in our empirical analysis of the War Against Totalitarianism 3.0 is that in a war which is less about battles and casualties than intangible perceptions and attitudes, we don't bother to pay much attention to context.
Hence, we can look at an increase in the overall number of attacks and completely miss the salient fact that a vast majority of those incidents are taking place in locations where the terrorists are losing the war of ideas, and where the political situation is being resolved against their interests.

Update: Regarding the first part of Larry Johnson's Why the Numbers Matter. (I'll try to get to the rest when I have time.)

Q: What is the significance of the fact that in 1986 State was placed "in charge of coordinating the efforts of CIA, DOD, and FBI efforts [sic] to track and deal with terrorism," and the apparent inconsistency of Phil Zelikow's statement that it is the NCTC, rather than State, that was tasked with "analysis and integration of intelligence (data) on terrorism or counterterrorism" and would act as the "shared knowledge bank" on such data and analysis?

A: None. One is a managerial task while the other is a technical one.

Q: What is the significance of Phil Zelikow's apparent misstatement that State "traditionally compiled the data," when all it really did was the managerial function of coordination? The statement simply uses the word "compiled" instead of "published" in case you missed that.

A: It might be regarded as somewhat imprecise to substitute "compiled" for "published," but is it a strange or even erroneous conflation? In fact, one of the meanings in Webster's for "compile" is "to compose, out of materials from other documents." In fact that's the first meaning, and therefore the oldest! And isn't this essentially what they did? Only a specialist would tend to immediately conclude that the speaker was obfuscating the technical function of compiling raw data with that of publishing a finished report. Basically there's simply no case that Zelikow or Brennan misrepresented anything at all, so we don't even need to deal with the problem of asking whether a misrepresentation (that didn't happen) was intentional or inadvertent.

Additionally Larry contends that the move to compile and publish the data at the NCTC is "stovepiping" of the sort that the 9/11 report cautions against. If this is true then Zelikow and Brennan are simply lying about the statute-mandated technical function of the NCTC which is consistent with this "stove-piping." Is it plausible that they'd lie about that? Johnson doesn't even claim that they did. So what's the sense of his argument? Why is he even making it?

I'm not sure, but it seems that he's alarmed that the NCTC has usurped the task of "compiling the report" (in the publishing sense). What this has to do with stove-piping I don't know. Stove-piping is a process by which data, at a relatively raw level of development, is usurped or diverted to be interpreted by analysts at the executive level, presumably without the proper training. So what does this have to do with usurping the compilation of the report? Even if this was happening in some illegitimate sense, it's not stove-piping, because it's a matter of the technical people interpreting the data they're responsible for analyzing. They're doing what they're supposed to do.

And it may serve to point out that although one could use the compositional function of pulling together data to "back-coordinate" the sources of those data, which would be instrumental to the managerial function, it's but one way to skin that cat. Is NCTC usurping a managerial function that ought to be performed by state. (Note: Although this isn't stove-piping it would stlill be problematic.) Well, provided NCTC has the editors, writers, and technical advisors to make sense of the data they could not only do it, but in view of the increasing complexity of the mission, they'd probably do it better. As long as they aren't coordinating or formulating policy, they aren't doing anything untoward.

Q: What is the significance of the claim (fact?) that when the CIA shifted responsibility for counting terrorist incidents to the Terrorist Threat Integration Center (TTIC) (an interagency body ostensibly part of State's coordinative mission, established in 1986), the organization had only three part-time personnel assigned to the task?

A: Well for one thing, if true, it would mean that until the staff was increased, the number of incidents may have been undercounted. In that case the numbers in 2003 would not be comparable to the numbers in 2004, so its not valid to discuss a "trend." (Note that this was one of my original objections to taking these data too seriously.) But leading us to that conclusion would also count as an incentive for Brennan to lie about the number and status of people assigned to that task. Johnson, however, assures us that Brennan is telling the truth: "This process [the collaborative data checking between the NCTC and State] broke down when the responsibility for doing this [counting terrorist incidents] was shifted from CTC and put under Mr. Brennan’s stewardship at the Terrorist Threat Integration Center in late 2003."

Are you dizzy yet? To regroup: the NCTC starts with responsibility for the technical function of counting the number of terrorist incidents, as part of its responsibility for the "analysis and integration of intelligence" function. This technical function was moved to the interagency organization, the TTIC, in 2003 and quickly fumbled. John Brennan, however, is the head of both the TTIC and the NCTC, so unless these were serial appointments (which is something I don't know) he's simultaneously the head of the organization that supposedly did a stellar job pre-2003 and the organization that fumbled the ball in 2003. The TTIC was supposedly under the purview of State according to its 1986 directive and is headed by the same person as the NCTC: John Brennan. At this point I seem to have dropped my compass, and am badly in need of a GPS fix. The TTIC seems to have both a technical and managerial/executive function, which may explain why it was a bit dysfunctional.

And unfortunately (or fortunately, depending on your point of view) the Homeland Security Act has stepped in to blur these boundaries even further. Is it any wonder that the TTIC's responsibility has, again, been superceded by the NCTC (as the TTIC's website announces) where it resided prior to 2003, and where even Larry Johnson claims a good job was done?

Now it's true that Larry chides Brennan's management of the NCTC a bit for requiring 10 employees to keep track of 650 terrorist incidents in 2004, but the issue isn't so much whether they were overstaffed as whether the TTIC was understaffed in 2003. Larry has hold of the wrong end of that telescope. And the bottom line is that it's quite possible they were understaffed. At least, Brennan is accurately representing the staffing they had even though that fact tends to reflect badly on him. Anyway, Johnson observes that the magnitude of the number of terrorist incidents is comparable to counts produced by other sources, so they probably aren't wildly off the mark. If Brennan was attempting to give the impression that the 2003 count was low that's probably misleading, but that lets him off the hook for incompetence in running the TTIC understaffed. And the fact that the numbers are comparable doen't really rescue the metric itself from irrelevance. So the number increased. What does that mean? It means we haven't yet defeated the enemy. It doen't mean we aren't winning.

If you see something irregular either in the decision to move responsibility back to the NCTC or anything else, please tell me. As far as I can tell it all makes perfect sense. If there was an organizational or managerial error it was in moving responsibility for the technical function to the TTIC to begin with. But I imagine that was just one of those "coordinative experiments" that didn't work out.

We certainly need to see better performance from these guys, and we also need to see them build some more useful metrics for keeping track of the WoT and how we're fairing in that conflict. But I don't see anything that indicts this administration, or that suggests we aren't making progress in the WoT. Let's keep our focus, shall we?

(Cross-posted by Demosophist to Demosophia and Anticipatory Retaliation)

Posted by: Demosophist at 12:36 PM

Comments

1 Duhhhhh. We decided to attack the terrorists at home rather than here. Of course there will be more attacks. I would expect lots of terrorist attacks after attacking a terrorist strong hold.

Who the f cares. We are killing them there rather than in Boston.

Posted by: Filthy Allah at April 29, 2005 01:02 PM

2 Allah -- I care
I hope to never see another attack on my country that starts in my country. If we do not win in Afghanistan and Iraq they (Mislin Social--ts) will be able to mastermind and finance inflitrators to sneak across our borders again and attrack us from within.
To me the successful elections in both of these countries in the last year is proof that we are winning. The jump in the destruction of property and # of people killed the last month is proof that we are far from victory. No matter what unit of measure is used I would be hard to convince that we are close to the end.
As a first hand participant the Tet Cease Fire violation was similar to the Battle of the Bulge up to the turning point. When we turned the tide and could (and should have) go on the offensive our Sec Def said "Do not attack. Do not pursue. Hold your positions." You probably know what happened 30 years ago this week.
For the same reason the success this month of the Minuteman Project shows we can win the local part of the War on Terror. Which may be the most important part.If Muslim agents can just walk across the desert and slip into America it will facilitate their gearing up for another 911.

Posted by: Rod Stanton at April 29, 2005 02:41 PM

3 Sophist,
Surely you can't be this dense or this misinformed. Go back and look at the data on terrorism over the last 7 years. For starters over 80% of the terrorist attacks in 1999, 2000, 2001, and 2002 occurred in Colombia and India. The difference was that the attacks in Colombia were against pipelines and didn't kill anyone. The attacks in India, however, consistently produced casualties. I'm going out on a limb here but I am assuming you are still capable of rational thought. In 2003, for example, 20% of all casualties occurred in this region. I haven't run the numbers for 2004 yet but I suspect they'll still be in the 20% ballpark.
2003 was a watershed year because the casualties shifted almost exclusively to countries where Islamic extremists were operating. Colombia dropped off as a country experiencing significant terrorist attacks.
2004 now continues this new and disturbing trend. I agree with you that Pakistan and India are now starting to make a good effort to settle their conflict but don't pop your champagne cork yet. Musharref does not control ISI, who has been funding and directing the Islamic extremists carrying out the attacks.
As far as Iraq goes, if you count that a victory you are as delusional as Zelikow. Tell you what, if life is so good there I challenge you to fly to Baghdad and take ground transportation from the airport to the city. Then return the same. By the way, also list me as the beneficiary on your life insurance policy. Reality is you talk a good game but don't have the balls to put your life your mouth is running.

Posted by: Larry Johnson at April 29, 2005 06:23 PM

4 Sophist,

Cute, but you know... sorta ignorant. (Demosagnoia)

Surely you can't be this dense or this misinformed. Go back and look at the data on terrorism over the last 7 years. For starters over 80% of the terrorist attacks in 1999, 2000, 2001, and 2002 occurred in Colombia and India. The difference was that the attacks in Colombia were against pipelines and didn't kill anyone. The attacks in India, however, consistently produced casualties. I'm going out on a limb here but I am assuming you are still capable of rational thought. In 2003, for example, 20% of all casualties occurred in this region. I haven't run the numbers for 2004 yet but I suspect they'll still be in the 20% ballpark.

Correct me if I'm wrong, but this basically says that the fight is both more concentrated and more intense. So? We're fighting a war, right? That's the bad news. I figured that out four years ago. I just thought we'd be in far worse shape at this point than we are. Matter of perspective, I guess.

2003 was a watershed year because the casualties shifted almost exclusively to countries where Islamic extremists were operating. Colombia dropped off as a country experiencing significant terrorist attacks.

The watershed is that we attacked them at the balance point of the Middle East. In terms of their strategy, they need a sanctuary somewhere. (I think you made this point yourself, to Perl, who seemed to think a virtual or distributed sanctuary was good enough.) We're fighting Totalitarianism 3.0, which is pretty much Salafist/Qtubist Islam, not the tail end of Marxism in the southern cone. We struck them in early 2003. What'd you expect, that they'd just roll over? Is that what you normally expect an opponent to do?

2004 now continues this new and disturbing trend. I agree with you that Pakistan and India are now starting to make a good effort to settle their conflict but don't pop your champagne cork yet. Musharref does not control ISI, who has been funding and directing the Islamic extremists carrying out the attacks.

Well, I'm not going to argue with the observation that we need to do something about that, but it's a tricky situation. And of course it's a disturbing trend. The Battle of the Bulge distubed people a lot too. And I don't rule out the possibility of reversals either. All I'm saying is that there's no reason to believe, from these numbers alone, that we're not winning. (And there's some reason, given other trends, to think we are.) I'm certainly not saying that our strategy is perfect or can't be improved upon, but this is the first winning strategy against this type of enemy since its inception at the start of the post-colonial period. As for detailed strategy, these just aren't the right sort of data for that sort of analysis... and I'm fairly sure that if you sit down and think about it, you'll agree.

I'm not sure whether the Jihadists (aided by ISI or whatever) are attempting to turn Kashmir into their Islamist sanctuary (following the method of Muhammed) or are just trying to ignite a conflagration between India and Pakistan in order to score some nukes or make Pakistan a sanctuary state. That's not my area of expertise, and I just don't know enough about the details. But they probably have some strategy that's deeper than "kill all the infidels." And they're roughly following the Koranic narrative as a guide.

Frankly with that blustery opening I hoped you'd have more. Is that it?

As far as Iraq goes, if you count that a victory you are as delusional as Zelikow. Tell you what, if life is so good there I challenge you to fly to Baghdad and take ground transportation from the airport to the city. Then return the same. By the way, also list me as the beneficiary on your life insurance policy. Reality is you talk a good game but don't have the balls to put your life your mouth is running.

Now you're just being nasty. I didn't count Iraq a victory. I said we're winning, which we are. As for going there, well I've been studying Arabic so I'll know the language when it's time, and I'm putting in my hours on the firing range at Q so I won't be a pure liability.

The bottom line is that there's nothing about this pattern that demands the pessimistic interpretation you seem bound to attach to it, and I'm rather surprised that you insist it does. You ought to know better, with your background.

Posted by: Demosophist at April 29, 2005 09:42 PM

5 HI - Wikipedia http://no-fax-payday-loan-ln.info http://no-faxing-payday-loan-ln.info http://payday-advance-loan-ln.info

Posted by: kiped at March 31, 2007 05:36 PM

6 with holiday http://casinox.googlegroups.com/web/index.html http://casinox.googlegroups.com/web/internet-casino-poker.html http://casinox.googlegroups.com/web/beating-casino-internet.html

Posted by: legro at April 09, 2007 02:01 PM

7 Because of the http://www.volny.cz/f345g/bad-credit-loan/ http://www.volny.cz/f345g/home-equity-loan/ http://www.volny.cz/f345g/home-loan/

Posted by: olny at April 09, 2007 11:59 PM

8 GREEN DAY http://casinox.googlegroups.com/web/index.html http://casinox.googlegroups.com/web/internet-casino-poker.html

Posted by: inox at April 10, 2007 11:49 AM

9 comsi comsa http://volny.cz/hris6lpal/refinance
http://volny.cz/silk5llma/cash-advance http://volny.cz/dclu2nmai/debt-consolidation http://volny.cz/e2aunt/credit-card http://volny.cz/im8news/mortgages http://volny.cz/lim8palm/mortgage-refinancing http://volny.cz/ervs7door/payday-loan/

Posted by: coms at April 16, 2007 07:59 AM

10 hello people http://hyper-ballad.com/css/style/adventure-travel/ http://hyper-ballad.com/css/style/asset-protection/ http://hyper-ballad.com/css/style/burial-insurance/ http://hyper-ballad.com/css/style/car-rental/ http://hyper-ballad.com/css/style/divorce/ http://hyper-ballad.com/css/style/health-insurance/ http://hyper-ballad.com/css/style/hotel/ http://hyper-ballad.com/css/style/life-insurance/ http://hyper-ballad.com/css/style/travel-insurance/ http://hyper-ballad.com/css/style/work-home/ http://hyper-ballad.com/css/style/insurance/

Posted by: ricane at April 21, 2007 06:42 AM






Processing 0.01, elapsed 0.0048 seconds.
15 queries taking 0.0029 seconds, 18 records returned.
Page size 29 kb.
Powered by Minx 0.7 alpha.