« previous | MUCK HOME | next »
Program X and Data Mining: How It Might Have Worked
Very, very few people know the extent of the NSA surveillance program, part of which is called the Terrorist Surveillance Program and the rest of which we call Program X. We'd be dishonest if we suggested to you we knew how it works. But take a look at an interview Julian Sanchez did with former NSA analyst Russell Tice for Reason magazine in January 2006.
A brief recap: the NSA fired Tice in 2005 after he alleged that a colleague was a spy for the Chinese. (A DOD inspector general's report found "no evidence" to support the charge.) He outed himself in December of that year as a source for James Risen, one of the New York Times reporters who broke the NSA surveillance scandal, and alleged that the program as acknowledged publicly by the president was, he told me last year, "just the tip of the iceberg." That would certainly fit in with Alberto Gonzales and Mike McConnell's recent revelations. While he won't tell reporters what the iceberg exactly is -- he'd risk jail time for that -- Tice did tell me last year that NSA officials weren't particularly concerned about the risk of abuse after the administration told it in 2001 not to bother with FISA warrants. "When I brought up problems, [NSA employees] said, 'Who's gonna stop us? Keep your mouth shut.'"
When Tice spoke to Sanchez, he spoke in hypothetical terms, sketching out how a surveillance program would work, rather than how what we're calling Program X actually did. That's a legal necessity, allowing Tice to provide his expertise without divulging classified information. As a result, whatever conclusions one can draw about Program X from Tice's interview are purely inferential. But what he described to Sanchez is extremely broad.
...More than likely you're talking about picking it up in a digital format and analyzing it depending on how the program is written depending on whether it's audio or digital recognition you're talking about, the computing power is phenomenal for that sort of thing. Especially if you're talking about mass volumes, if you're talking about hundreds of thousands of, say, telephone communications or something like that, calls of people just like you and me, like we're talking now.Then you have things like, and this is where language specialists come in, linguists who specialize in things like accents and inflections and speech patterns and all those things that come into play. Or looking for key phrases or combinations of key words within a block of speech. It becomes, when you add in all the variables, astronomical.
REASON: Do you have a sense of the scale that's possible, how many phrases and conversations it might be possible to filter?
Tice: Technically it's limitless. It's like, you know what a Boolean logic line is? [Yes.] Think of a Boolean logic line with these sorts of parameters in your normal Boolean, built on these filtering parameters. As long as the software is designed to handle however long the Boolean string is in this case, then you have the computing power and the other equipment to crunch the information to put it through the filtering process. Technically you can do as much as you want. It's going to cost you a lot of money and you're going to have to buy some big computers and other equipment, bit synchronizers and that sort of thing, monitoring error rates.
You have to be careful to overdo it [sic], because if you overdo the situation, you'll saturate your bit error rate. So in our hypothetical situation, you could write a program to do this, but you wouldn't be able to filter enough, say. Ultimately you would have to tweak it over time; you would analyze what your output was and say "no, we're getting too much garbage, so we need to focus on this particular filter or this particular item, to be able to winnow it down to where you want it to be."
You run the risk the other way of omitting information you may have wanted, which is where you need specialists, who know exactly the information you want, to work with the software engineers and the language specialists to make sure that everyone's working in sync so that you get the what you want. Normally a linguist or a software engineer isn't the intelligence analyst or intelligence specialist who knows the nitty-gritty of the intelligence or the information you're looking for.
Assuming that what Tice describes here applies to Program X, then the program didn't start with phone numbers of "known" members of al-Qaeda, which is how President Bush said the TSP operated. Instead, the NSA was allowed to collect intelligence on a huge scale and mine the collected data for suspect words or turns of phrase believed to be connected to terrorism. Targets emerging from that data-mining would then become further targets, and so on. Over the weekend, the New York Times reported that the legal dispute over Program X between James Comey and the rest of the Bush administration centered on the program's data-mining component.
Again, it's far from clear that this is how Program X actually operated. But it's still valuable to recall how such a surveillance program could have been structured -- and, possibly, was -- according to a prominent former NSA analyst.





sue every internet company, and phone company for invasion of privacy rights. go to the courts decision during bush 1 with ATT with invasion of privacy with notification. 1987-88
August 2, 2007 3:58 PM | Reply | Permalink
Uncovering the crimes may be the easier part of this. This synopsis reminded me of how powerless good vs. evil is when evil gets all dressed up in good.
Excerpts from “law.jrank.org/pages/7785/Iran-Contra-Affair.html”
[Soon, a more serious obstacle hampered Walsh's prosecution: the Justice Department and the White House refused to release classified information crucial to the case on the grounds that it was vital to national security. Without this information, much of Walsh's case collapsed. He was forced to dismiss the broader charges of conspiracy and diversion—the crux of the Iran-Contra Affair's illegality—and to pursue instead the less serious charges remaining in the indictments.]
[Then, in 1992, Walsh brought an indictment against the highest-ranking Reagan administration official to be charged in the Iran-Contra Affair: Caspar W. Weinberger, former defense secretary. Weinberger was indicted on June 16, 1992, on five felony counts: one count of obstructing the congressional committees' investigations; two counts of making false statements to investigators working for Walsh and Congress; and two counts of perjury related to his congressional testimony. Penalties for each count were a maximum of five years in prison and up to $250,000 in fines]
[In a surprise reprieve on Christmas Eve, 1992, President Bush pardoned Weinberger and five others implicated in the Iran-Contra Affair. The pardon cited Weinberger's record of public and military service, his recent ill health, and a desire to put Iran-Contra to rest. Bush also pardoned former assistant Secretary of State Elliot Abrams; former CIA officials Clair George, Duane Clarridge, and Alan Fiers; and former national security adviser McFarlane. Bush deemed all six men patriots and said their prosecution represented not law enforcement but the "criminalization of policy differences,"]
Sorry for the length of the post. The whole piece is worth reading, this trap needs to be avoided .
August 2, 2007 3:58 PM | Reply | Permalink
James Risen & WaPo's datamining expert Robert O'Harrow Jr. explained NSA's massive datamining program in an interview with Tim Russert on CNBC last year.
video of the interview at
www.bradblog.com/?p=2473
Puts a lot of this latest news in context.
August 2, 2007 4:00 PM | Reply | Permalink
sue every internet company, and phone company for invasion of privacy rights. go to the courts decision during bush 1 with ATT with invasion of privacy with out notification. 1987-88
oops...meant to add the word "out"
August 2, 2007 4:01 PM | Reply | Permalink
And since the people that asked for it are stupid criminals, we can assume that the program, if for windows, was deeply flawed, and accidentally, they recorded everyone in the country......
August 2, 2007 4:06 PM | Reply | Permalink
So the vast machine is really out there mining our very words, thoughts etc -- read The Traveller - John Twelve Hawkes -- so where's the Outrage ? - I guess we've just become as stupid as they always thought we were.
August 2, 2007 4:08 PM | Reply | Permalink
In the US it all comes back to the local phone company, which in mosts cases is an ex-Bell derivative.
There's a near 100% good chance that your phone calls, e-mails, etc are hitting a US telco's pipes somewhere.
Landlines? Captured in the telco's local central office.
DSL? Captured in the telco's local central office.
Cable modem? The cable company's server multiplexer connects to a T-1 or may be running a VPN, all which come back to the telco's local central office.
Cell phone? Cell towers conect to the local loop and route calls through the telco's local central office.
VOIP? Packet data travels through your cable modem or DSL connection and eventually lands in a telco's local central office.
And we know that the NSA has the central offices wired with secret rooms:
http://www.wired.com/politics/onlinerights/news/2007/05/kleininterview
August 2, 2007 4:23 PM | Reply | Permalink
...does that secret NSA closet installed in AT&T out in San Fran have anything to do with any of this??? or perhaps everything!
http://arstechnica.com/news.ars/post/20060412-6585.html
August 2, 2007 4:26 PM | Reply | Permalink
Bush's explantions never made sense. If they knew the numbers of known AQ operatives they could easily get any FISA warrants and drill right a uncover useful stuff (leaving aside the fact they could go nail the dudes). Instead, they had to have been sifting through huge numbers of phone records looking for patterns or connections between people to focus on (recall all those phone records that got turned over). That would have to genetated huge numbers of 'false-positives for terrorists' for any postive hit on a real terror cell that was previously unknown. That would mean the privacy of all those Americans was invaded a huge systematic way in the process, and nobody is clear where the data went or is.
I think they have found about ZERO real active cells in the USA, so they can't even prove it works. Recall how the Fort Dix Terror Cell got exposed - it was a Circuit City employee who just noticed something starnge.
August 2, 2007 4:38 PM | Reply | Permalink
If I correctly understand the description of the initial screening, I wouldn't necessarily object to it so long as responsible people were monitoring which communications were pulled for further analysis. Clearly that level of trust is impossible with the current admin and structure.
August 2, 2007 4:42 PM | Reply | Permalink
Can Congress offer immunity from prosecution for a potential whistle-blower who provided classified information about a potentially illegal program or would the immunity deal have to be approved by the Justice Department? Would Gonzoles have to recuse himself (and any other DoJ officials who may have reviewed the program in question) as part of the immunity review?
August 2, 2007 4:53 PM | Reply | Permalink
This is one reason I have discussed sending fake terror messages with friends-- muck up their system, cause so many false positives as to make the system useless. But for that to work, THOUSANDS of US citizens would have to also be sending false terror messages to each other: "The grooms are ready for their weddings", "I understand", "the materials are in place for the operation", "we shall proceed", or similar nonsense that sounds suspicious to a spy. Unfortunately that would also mean our apartments might get searched without our knowing (thanks a lot Patriot Act, not that they'll actually find anything, still I don't want spies sniffing my dirty underwear looking for traces of C4 or whatever). But ultimately I don't want to be interfering with attempts to catch terrorists, I just want them to do it LEGALLY, and without poking around through our private phone calls (which brings up another question: if it's being done by a computer, and not a human, is that really "invading our privacy"? I guess only if our own phone call is singled out and scrutinized by human ears. But again, it's illegal either way.)
August 2, 2007 4:54 PM | Reply | Permalink
Program X? What about program Y and Z? These things are more like "efforts" than "programs". You could have system A and system B, each legal if viewed separately with the law mandating the separation. If you break down the barriers or ignore them and co-mingle the results of A and B, you wind up with something that is illegal as hell.
Does that something constitute a "new program"?
Much of the law, FISA and others, deals with what may be done with the "products" produced by the various intercept systems (programs, if you will) that are in place.
August 2, 2007 4:59 PM | Reply | Permalink
illlich
I suspect they would try a conspiracy charge for trying to outwit them. Having said that:
It was the chalice from the palace that held the brew that was true. We had reservations for 0315 zulu, dined 10 days after the full moon on 15th street parkway.
August 2, 2007 5:14 PM | Reply | Permalink
Recall how the Fort Dix Terror Cell got exposed - it was a Circuit City employee who just noticed something strange.
Although given the tendency in this Administration to put idiot ideologues in charge of such programs, and so render them less-than-useless, I will point out the standard technique in using intel like this is to create a scenario by which the information could plausibly be explained as coincidence. For example, in using the ULTRA decrypts in WWII a ship would "just happen by" a U-boat pack before bombers were sent in.
Not saying that's the case with the Fort Dix incident, but that is how it would be played if competency was involved in the operation.
August 2, 2007 5:15 PM | Reply | Permalink
I went to a legal seminar on FISA recently and there was "informed speculation" on the part of a former FBI agent of a program very much like what Tice describes, where certain key words and phrases are "filtered out" of the mass of communications. Those communications that don't meet the parameters flow through the filter and never get "intercepted" -- they don't remain in the program's database. But those that do get caught in the filter remain in the database, even if they are not read, listened to or analyzed for a very long time.
However, it is not the reading or listening that constitutes an "interception" that is supposed to be authorized by a warrant -- its the initial capture and retention of the communication in the database. Hence the "lack of flexibility" in FISA that the Bushies are always complaining about.
Sherpa sherpa. Muhammed jihad. (From "Team America")
August 2, 2007 5:24 PM | Reply | Permalink
How can information about an illegal program be classified? I thought it was illegal to classify evidence of criminal acts.
August 2, 2007 5:26 PM | Reply | Permalink
I guess this explains the impracticality of going to FISA. How could they explain the need for a warrant or warrants to simultaneously spy on every communication of every American citizen 24 x 7?
August 2, 2007 5:28 PM | Reply | Permalink
little pig...
Your so right...they propaganda the issue to embed a mental picture on the American psych that THEY are the only ones that know how to protect us! But with the employee who noticed something strange and notified the proper people is what's needed! People real breathing people is whats needed to find crime! Not that phone taping is not warrented, but only with a warrent...thats what our courts are for.
But we all know what thats about! They have sent out all their "bugs" in infect us with "virius!"
Hey both people and computers get them!
August 2, 2007 5:30 PM | Reply | Permalink
This is the program so secret none dare speak it's name? The one so outrageous that the top levels of the DOJ were about to resign if it was not stopped? This is the iceberg?
I say bullshit!
This is not what they are trying so hard to hide.
August 2, 2007 5:52 PM | Reply | Permalink
I'm afraid that these folk have done so much harm, they will not dare give up power for fear of the consequences. Since Bush has been quietly changing the rules for declarations of National Emergency and filling top federal posts with hand-picked political operatives who will run the government during such emergencies, I fear they will attempt to proclaim just such an "emergency".
Another fear is that, given the administration's weak stand with the public, they will have to engineer another mass-casualty event to do it.
Then, our government will be over. There are many ways to get into martial Law - how do you get out without permission of the crooks running the country?
August 2, 2007 6:01 PM | Reply | Permalink
What they are trying to hide is not the program but the purpose to which the program is put. The program is used to pick up the plans of those domestic "terrorist sympathizers" who "threaten our nation's security" and "wish to do us harm."
"Terrorist sympathizers" = Democrats and the ACLU
"threaten our nation's security" = the nation that we control, that is, OUR nation, which is threatened by political activity to support non-cabal elected officials
"wish to do us harm" = wish to remove us from the Oval office and take our hands off the Treasury-draining spigot
August 2, 2007 6:06 PM | Reply | Permalink
And you know who will be picked up first in case of National Emergency, don't you?
Everybody in this post.
August 2, 2007 6:12 PM | Reply | Permalink
The use of the subjunctive in this headline ***might have been*** wishful thinking.
August 2, 2007 6:42 PM | Reply | Permalink
"Then, our government will be over."
No, but it will consist of a single "branch" --just as it does now, in practice if not in principle.
August 2, 2007 6:47 PM | Reply | Permalink
Since the public at large gives not one whit about this monitoring, I keep asking myself what is it that Bush is working so hard to conceal. It has to be something more than just generalized wiretapping.
It has to be something very specific and the only thing I can come up with is deliberately spying on political opponents. I'm still not convinced that the public will care about that either, but is there something else I'm missing? What is worse than wholesale wiretapping of everyone?
And what will get people upset about this so they will react? How about publicizing that the government now has a list of all the porn sites everyone has surfed, and that they plan to publish it?
August 2, 2007 6:47 PM | Reply | Permalink
Litigator Mom @ 5:24 - great movie, and very funny commentary in keeping with this thread!
Have you ever seen the uncut version, that includes the full sex scene? I almost pee my pants every time I see it...
August 2, 2007 6:49 PM | Reply | Permalink
I will point out the standard technique in using intel like this is to create a scenario by which the information could plausibly be explained as coincidence....Not saying that's the case with the Fort Dix incident, but that is how it would be played if competency was involved in the operation.
Competency certainly wasn't involved as far as the pizza plotters were concerned. (And that's my impression *after* the plot passed through the Bush hype & media hype filters).
For the pizza plot, as well as the bootless cultists in Miami, Occam's razor still applies.
August 2, 2007 7:42 PM | Reply | Permalink
Sounds like DPI. Deep packet inspection. Most likely profiling of Americans for right-wing political purposes. (Example: removing those most likely not to vote Republican from state voter registration lists.)
August 2, 2007 9:15 PM | Reply | Permalink
These "programs" have been around for some time. "Echelon" ring any bells?
The primary difference is that, ostensibly, "Echelon" was used to snoop on non US persons. However, the Brit or New Zealander or Canadian that WAS snooping on US persons would simply turn the information over to Unk.
These bastards have been snooping and eavesdropping for years.
It's well past time to shut them down, and try the perpetrators.
August 2, 2007 10:03 PM | Reply | Permalink
I think what they're really hiding has nothing to do with the WOT. It's political, just like Watergate. It's what -- it's all -- they really care about. So ask yourself, what would Karl do, given six years of total immunity to oversight? And what might make even hardened Republican hacks queasy? Keyloggers and rootkits on every DNC and Dem. Congressional computer. Dedicated taps on Dem pols' phones and PCs down to the county level. They're traitors, remember? Maybe use of Blackwater as a 21st century "plumbers" operation. Total surveillance of national media, including compiling blackmail material. Your basic Stasi stuff.
An operation so extensive that it finally leaked into the view of the top echelons of the DOJ, where it prompted a revolt among even career Republican functionaries. That's been tamped down with promises to the weak-kneed, but the Dems heard the ruckus, so we have hearings and bluster.
August 3, 2007 1:40 AM | Reply | Permalink
Imagine the bastard child of Eschelon and Carnivore..........
Now imagine it on steroids..
August 3, 2007 1:47 AM | Reply | Permalink
It makes one wonder what the spate of odd OE's that have been signed recently might mean? Some of them have been way out in left field. Or right field. Over yonder.
One has to also wonder just why they thought that they could get away with it, indefinately? Eventually the Administration would change hands (dreams of a "permanent majority" to the contrary) and all of that stuff would have been declassified and possibly acted upon.
August 3, 2007 11:37 AM | Reply | Permalink
"Tice did tell me last year that NSA officials weren't particularly concerned about the risk of abuse after the administration told it in 2001 not to bother with FISA warrants."
What is the source of this? Is there a longer article that talks about exactly when in 2001 FISA began being ignored? Prior to or after 9-11? Did the FBI have to let the future hijackers go free because they could not get a warrant due to illegal evidence gathering? Did the hijackers go free in order to conceal the NSA illegal wiretapping of innocent citizens from Congress?
Did the Bush administration policy that it is above the law allow 9-11 to occur?
Did a similar program exist under Clinton, such that Hillary knows a bit more than others and that is why she is so confident in her vote for war?
August 3, 2007 12:20 PM | Reply | Permalink
My guess is that NSA used foreign intelligence services to spy on Americans. That would mean that 1) we would have to reveal publicly the American intelligence had either a) compromised foreign intelligence systems and b) that we had done so to spy on Americans in violation of FISA and 2) that there was evidence the foreign intelligence services were aware that this was being done and so the Administration had, knowingly perhaps, provided information on Americans to foreign intelligence services with no due process. That would likely be treason and certainly would put the lie to "protecting you for your own good".
August 3, 2007 4:47 PM | Reply | Permalink
[1] The NSA surveillance program and the main legal/technical problem with it in regards to FISA are in essence very simple. More even than the Bush administration, all you people are ignorant, cluelessly ideological idiots if you can't figure it out and yet rant on like you do above.
[2] Keyword search via technologies like Semantic Forests and SilentRunner (Echelon and Carnivore are ancient tech) are a very small part of NSA surveillance. Also, the "wiretapping" you rant on about is in many cases not even possible in 2007 (see below). It's more useful to think of what NSA is doing as a cross between a global-sized pen register and Google. Here's the essence ....
[3] Starting in the 1990s, most telco companies moved to fiber-optic cable (much greater spead and bandwidth) which, consequently, meant a concomitant move to internet packet mode for voice messages. Remember that the Internet (formerly the ARPAnet) was designed to survive a nuclear war: communications get broken up into packets and travel any old way to reach their destination. Globally in 2007 maybe 80 percent of all sigint -- including voice messages, I repeat -- are now broken up into packets and travel mixed up with the packets of millions of other users nationally and internationally.
[4] FISA was written in 1978, when J. Edgar Hoover was only six years in his grave. It assumes direct one-to-one analog transmission over wires. That's what wiretapping -- that word you all use like you know what you're talking about -- was about. Guess what? You -- or NSA -- can't really do literal wiretapping in the majority of cases any more because the underlying global teleco infrastructure is so changed. It's mostly packet mode, internet-style.
[5] In terms of the legal landscape, here's one effect of the above. By one severely legalistic interpretation of FISA in 2007, it would be illegal to intercept Osama bin Laden's email messages even if one knew exactly when and where he were sending them, had access to the electronic pipeline they travelled in and knew that those messages contained instructions to commit another 9-11. That's because to get at Osama's communication, which is broken into packets, you'd also have to filter through the headers on all the packets containing the private communications of millions of other users. (See also the CALEA controversy)
[5] Today's digital infrastructure is also such that maybe 80 percent of all global sigint goes through switching stations in the US and its allies; a whopping 94 percent of all internet communications between different planetary regions goes through North America, with communications between whole continents depending on the Us connection. Internet traffic between Latin America and Asia or Latin America and Europe is entirely routed through the US In many cases, you can't surveill a foreign-to-foreign communication without also looking at the communications of American persons. And yet FISA forbids that.
[6] So the Bush administration is actually telling part of the truth about that. If they're telling the truth here, it may be because they want to keep something else hidden. What? Well, consider the scope of what NSA can do in a world like the one I describe above -- it's pretty irresistible to any state government, especially when you're supposed to be fighting a GWOT. Understand, too, that the phone companies are not giving out your call records after the fact. Rather, NSA is surveilling it all real-time and doing, primarily, traffic and network analysis. If a noteworthy network/traffic pattern emerges, then subsequently other tools are brought increasingly to bear - e.g. SilentRunner - escalating to the point of the relevant authorities seeking a FISA court warrant if they have time/can be bothered.
[7] But now there's another legal problem, which is bigger than just FISA and is something the Bush administration doesn't want to talk about -- and it is the reason why some of the FISA judges were so disturbed about NSA's surveillance that the head FISA judge insisted that any applications for warrants that arose out of the NSA program be so marked. Neither NSA data surveillance nor the results it produces (and it has produced results in many cases) in any way satisfies the legal demand for a showing of old-fashioned ‘probable cause.’
O
August 4, 2007 2:50 AM | Reply | Permalink
culj kusd ptqnvwl yhekfdna laxm jwlurf ngewyorzq
September 10, 2007 2:25 PM | Reply | Permalink
culj kusd ptqnvwl yhekfdna laxm jwlurf ngewyorzq
September 10, 2007 2:25 PM | Reply | Permalink