Above: a slide from the 2002 overview of the now-canceled Total Information Awareness program.
This post is inspired by the RAND Corporation’s testimony presented before the Cybersecurity Subcommittee of the Senate Armed Services Committee last Thursday, April 27, 2017.
The RAND Corporation expert who actually delivered the testimony was Rand (yes you read that right, Rand who works at RAND) Waltzman, Senior Information Scientist.
CLICK HERE to watch video of the testimony. Three other guys testified as well: John Inglis, Former Deputy Director of the NSA; Michael D. Lumpkin, Principal At Neptune Computer Incorporated (and Former Undersecretary Of Defense For Policy); and Clint Watts, a fellow at the Foreign Policy Research Institute.
— Rand Waltzman (@CogSec) May 2, 2017
Rand Waltzman is a former DARPA official who ran a program called “Social Media in Strategic Communication” (SMISC for short). When the program was first announced back in 2011, media like Wired magazine and The Guardian came out with articles calling it a “social media propaganda tool” and “ways to covertly use social media for purposes of propaganda and deception.”
The birth of Cognitive Security
SMISC resulted in dozens of peer-reviewed scholarly and technical articles published by universities (USC, Indiana U. and Georgia Tech) as well as private companies (IBM, Sentimetrix and Systems and Technology Research). The articles range from hyper-geeky (“Comparing IndexedHBase and Riak for Serving Truthy; Performance of Data Loading and Query Evaluation“) to sinister-sounding (“Recommending Targeted Strangers from Whom to Solicit Information on Social Media“) to puzzlingly pedestrian (“Demographics, Weather and Online Reviews: A Study of Restaurant Recommendations”) to just plain puzzling (“A Fast Algorithm for Streaming Betweenness Centrality“).
But last Thursday, Waltzman wasn’t on Capitol Hill to talk about SMISC. He was there to pitch an idea he’d first unveiled in 2015: that the U.S. government should set up a “Cognitive Security Center” (originally the “Center for Information Environment Security”) which would defend Americans’ brains, literally, from falling victim to persuasion, propaganda, fake news, viral memes and the like.
After SMISC wrapped up and Waltzman left DARPA, he and his allies, mostly former military men in private industry and government, have been increasingly vocal about the government needing to realize that the new Great Game being played between world powers is an information war, a permanent information war fought on the battlefield of people’s thoughts and opinions. And he wants you to know that so far, America is losing.
All warfare is based on deception.
There is no instance of a country having benefited from prolonged warfare.
– –Sun Tzu, The Art of War, 6th Century B.C.
Waltzman’s timing couldn’t have been better for getting the attention of American lawmakers. We have both Democrats and Republicans calling Russia’s alleged interference an “Act of War,” and the U.S. intelligence community has not only directly accused Russia of having supplied Wikileaks with embarrassing e-mails hacked from the DNC and phished from John Podesta’s Gmail account, but has also “connected the dots” of those criminal acts to Russia’s noncriminal persuasion channels, forming a picture of a comprehensive, whole-of-government influence campaign that included paid online covert trolling operations as well as overt messaging led by Kremlin-controlled international media like Sputnik and RT.
As part of the 2017 National Defense Authorization Act (US military budget), on December 23, 2016 President Obama signed a law called the Countering Disinformation and Propaganda Act which creates an interagency center at the State Department (probably the same as, or combined with, the Global Engagement Center which focuses on anti-terrorism messaging). The new activities will include traditional international aid focused on spreading democracy, combined with the more sinister aim to “develop, integrate, and synchronize whole-of-government initiatives to expose and counter foreign disinformation operations by our enemies and proactively advance fact-based narratives that support U.S. allies and interests.”
Washington has never been more receptive to proposals to censor the press, social media and web news, and to counter lies with truth,* than right now. America is losing the Information War? Unacceptable! There are many people, mostly but not all Democrats, that believe that the alleged Russian interference in the 2016 election is as much of a wake-up call, thrown-down gauntlet–and overall unprecedented threat and existential danger to America–as 9/11.
*Originally I had this styled as “‘lies’ with ‘truth,'” with “lies” and “truth” in quotation marks, but I think this casts unnecessary shade on the intentions of members of Congress. I think that CogSec’s potential danger exists irrespective of the motivations of its backers, since the people who plan and promote CogSec won’t be the people who implement the program and/or use the finished technology.
I’m not going to bother to make the argument that censorship is bad. At best, censorship is a necessary evil in extraordinary circumstances for a limited period of time. But I do argue that how bad censorship is depends on who’s doing the censoring. As much as Congress might hate the idea of America being behind in the Information War and want to do something about it, Congress doesn’t execute the laws–the Executive Branch does. Censorship is troubling when the censors have the best intentions; but as we now know, there’s no guarantee that the Executive Branch won’t be led by people who have a strange and unpredictable relationship to the press, social media and the formation of public opinion.
— Donald J. Trump (@realDonaldTrump) May 5, 2016
And Waltzman’s proposal doesn’t just involve censorship; it involves the U.S. government and intelligence community running covert information operations on the American public aimed at countering the influence of foreign propaganda and influence campaigns.
After the US intelligence community’s January 2017 release of an assessment of “high confidence” that the Russians were behind the DNC and Podesta hacks, the 2016 Wikileaks dumps, “fake news” and trolling operations, Waltzman has rebranded his concept as “Cognitive Security,” which appears to be a term he invented, and has tweaked his pitch to refer more to the Russian threat.
In March 2017, Waltzman registered a new Twitter handle, @CogSec. Together with 4 former military officers (Waltzman’s own military experience is limited to DARPA, which is part of the Department of Defense but is staffed by civilians), he formed an organization called the Information Professionals Association and registered a website with the masthead mission statement Bringing together experts in Cognitive Security. Yes, that’s right, Waltzman invented the field of Cognitive Security and is a founding member of the field’s only professional organization, in which he features prominently as a leading expert. So you can’t really talk about CogSec without talking about Rand Waltzman.
From Waltzman’s testimony before the Cybersecurity Subcommittee:
My question is: couldn’t they have come up with a less Orwellian name than “Cognitive Security”? “Cognitive” is the adjectival form of “cognition,” which is another word for “thought,” and “security” is a synonym for “police,” so semantically “Cognitive Security” is the same thing as “Thought Police” (that’s right, literally just like in Orwell’s 1984) and when translated into other languages I’m pretty sure any distinction between “Cognitive Security” and “Thought Police” will be lost. You would think that experts in persuasion would easily be able to give their new field a less heebie-jeebie-inducing name.
So anyhoo, Waltzman wants the U.S. government to form a Center for Cognitive Security. The problem is, as Waltzman himself points out, his plan is almost certainly illegal under existing law. For example, subsection (f) of 50 U.S. Code § 3093 says “No covert action may be conducted which is intended to influence United States political processes, public opinion, policies, or media.”
Attempts to apply … out-of-date laws and regulations by lawyers throughout the government have led to overly cautious and non-uniform policies and prohibitions resulting in massive confusion and paralysis. The bottom line is that the U.S. government has no ability to defend us against being manipulated by actors—state as well as non-state—that recognize no boundaries on a scale that was previously unimaginable.
—Rand Waltzman, Time.com, “The U.S. Is Losing the Social Media War,” 10/12/2015
Before 2013, U.S. government media like the Voice of America, Radio Free Europe/Radio Liberty, Radio Marti and others were prohibited from targeting U.S. domestic audiences. That prohibition has been lifted, so now U.S. government-produced programming is being broadcast in the United States. Whether or not you believe that’s sinister depends on who you are. Some people who had no problem with the Voice of America broadcasting in the U.S. under Obama have concerns about the direction of American government media under Trump. One of the first things Trump did after inauguration–within the first two days of his presidency–was order a review of the Voice of America, which had reported unfavorably (and probably unwisely) on the size of Trump’s inauguration crowds compared to Obama’s.
As it stands right now, U.S. government media are relatively independent–probably not as independent as the BBC (British state-owned media), but by no means a White House mouthpiece. But if Trump or any future president wants to turn the Voice of America into a White House mouthpiece, there’s really no legal barrier to stop that from happening.
So the pieces are falling into place to give the government a greater presence in the American information environment. But Waltzman says he also needs a change in attitude and application of the laws, complaining, “the interpretations are extremely conservative, to the point where we aren’t able to function. It’s a combination of interpretation, existing laws and policies which are based on those, and attitude.”
Waltzman, as well as other CogSec advocates, say that the Privacy Act of 1974 is also a problem, because it restricts the collection and dissemination of U.S. citizens’ personally identifiable information by the government. The following is a quote by Michael Lumpkin, Special Envoy and Coordinator of the State Department’s Global Engagement Center:
The 1974 act was created to make sure that we aren’t collecting data on U.S. citizens. Well, … by definition the World Wide Web is worldwide. There is no passport that goes with it. If it’s a Tunisian citizen in the United States or a U.S. citizen in Tunisia, I don’t have the ability to discern that. Therefore, I have trouble grabbing that personally identifiable information. If I had more ability to work with that [personally identifiable information] and had access…I could do more targeting, more definitively, to make sure I could hit the right message to the right audience at the right time.
There’s also the little problem of the First Amendment. If the U.S. government is going to be engaged (whether on its own or by enlisting corporate agents) in restricting the dissemination of information and ideas, whether by media outlets or by individuals using social networks, it will almost certainly bring constitutional challenges.
Total Information Awareness, redux
Some of you might be thinking: this sounds familiar. Haven’t I heard of another legally questionable, DARPA-linked idea being proposed as a new solution in the wake of a national emergency?
Yes you have. The year was 2002. The program was called Total Information Awareness (TIA), which was even more sinister than it sounds. The program was so widely reviled it was denounced by both the Cato Institute and the ACLU. It was headed by Admiral John Poindexter, the former Reagan-era National Security Adviser known for his involvement in the Iran-Contra scandal, who famously said that it was his duty to withhold information from Congress.
TIA was reported on ad nauseam in the media in the early 2000s, after 9/11, so I don’t want to waste time and space describing it. Basically, it’s the machine from “Person of Interest.” If you don’t know what it is, it’s more efficient if you watch a few episodes of that Jim Caviezel vehicle before reading further.
The name “Total Information Awareness” was quite literal: the aim was to build a machine that would take every piece of digital information in the world and analyze it in real time using artificial superintelligence, in order to predict terrorist acts before they happen.
In the case of TIA, it wasn’t the First Amendment that was a problem, but the Fourth, the one prohibiting unreasonable searches and seizures. In order for TIA to work, it would require access to every existing database, public and private, which would almost certainly be an unconstitutional search and breach of citizens’ right of privacy.
People freaked out. So in the 2004 Defense Appropriations Act, Congress shut TIA down. Although the program didn’t really actually shut down, what, are you crazy? They just closed the bureaucratic entity known as the “Total Information Awareness Office” (although, yes, by that time they’d renamed it the “Terrorism Information Awareness Office”). The technology and the idea and the goal live on.
There are a lot of parallels between TIA and the proposed Center for Cognitive Security: DARPA; an “unprecedented” intervening event that made an idea once considered anathema to American values suddenly seem reasonable; bandwagoning by government officials and members of Congress; both of them violate existing law; both seem tailor-made for objections by civil liberties advocates…
But the most important similarity in the two cases is this: in both cases, an ancient way of controlling the population and suppressing dissent has been wrapped up in a high-technology package and offered as a new way to address a “paradigm shift,” a solution to novel problems in a changing world. Things have changed, we’re told. The laws must also change. We have the technology to fix this, if only we have the political will to untie the government’s hands.
From Waltzman’s testimony last Thursday:
But of course, there’s nothing new under the sun. Government searches and surveillance, propaganda and control of information are as old as civilization. So are influence campaigns waged by one state on the population of another state. The word “censor” comes to us unchanged from Latin, as a description of an ancient Roman official who was responsible for, among other things, public morality.
And in both cases, both TIA and CogSec, there’s no assurance that the program will actually work for its intended purpose, but there is absolute certainty that it will change the balance of power between the citizens and the government, probably for the long term.
The limits of influence
Would it work though? Another RAND study, quoted in Waltzman’s testimony before the Cybersecurity Subcommittee, gives some reasons why people are vulnerable to propaganda:
OK, got it. But is “propaganda” really different than other forms of persuasion? And how are the government officials and others that would be involved in Cognitive Security supposed to be immune from the cognitive biases that affect, as far as we can tell, all humans (and probably all conscious entities, including artificial intelligence)?
But this is different, you say. DARPA-developed technology will be brought to bear to distinguish truth from untruth. Is that possible? Can technology designed by cognitively biased humans do what a human cannot–conclusively determine what is objectively true? We don’t know.
That brings us to a familiar trope of #TrumpRussia: that Russian propaganda is nothing less than an assault on the very concept of truth. This thesis was first brought to the public’s attention in Peter Pomerantzev’s 2014 book Nothing Is True and Everything is Possible: The Surreal Heart of the New Russia. The idea got immediate traction: Putin isn’t like the propagandists of the Soviet Union, who were trying to convince the world to believe in the wretchedness of capitalism and the righteousness of Communism; he wants people to believe in nothing. And not only to believe in nothing, but to believe that objective truth, the thing that people have been searching for from time immemorial, doesn’t exist.
It raises an interesting philosophical question: the “search for objective truth,” from Diogenes to today, is considered one of the quintessential noble human endeavors, which unites the human race across time and culture. In other words, it’s a good thing. We want people to believe that there is an objective truth and we want people to want to discover it. One could go so far as to say: if everything we think we know is a matter of subjective opinion, what’s the purpose of life?
Diogenes Searching for an Honest Man, attributed to J. H. W. Tischbein (c.1780)
But at the same time, we know now more than ever that, ironically, the one truly objective truth we can be sure of is that there is no such thing as objective truth. There are observer effects and observer bias all the way down to the subatomic level. As strange as it might sound, reality appears to actually, physically respond and change in response to a conscious observer. Then there’s the “Rashomon Effect” (named after Akira Kurosawa’s classic 1950 film Rashomon): rational, honest and reasonable people often experience and remember the same events differently. A famous recent example is the tragic killing of Michael Brown by a police officer in Ferguson, Missouri in 2014: 64 witnesses, some of whom were believed to be lying and some of whom were not, gave contradictory accounts of the same events. And even the people believed to be telling the truth gave statements that contradicted those of other people believed to be telling the truth. This results in a paradox: the more witnesses there are to an event, the harder it is to use witness testimony to prove what happened.
So there’s a major doublethink that can’t be ignored: we want people–in the abstract, as a teeming mass–to believe in the existence of an objective truth (as a value system and motivator more than anything). No one wants to live in a society of cynics. But as individuals, we’re becoming more and more aware of convincing reasons to doubt in the existence of objective reality or, if it exists, to doubt that humans can discover it.
And there’s also a paradox: although the idea that objective truth doesn’t exist is potentially dangerous to society, so is the idea that objective truth does exist, but that the government has a monopoly on it. That’s exactly what the Thought Police in 1984 were all about.
It seems extremely unlikely that increased U.S. government involvement in the marketplace of ideas can resolve anything, and it seems extremely likely that it will backfire. How would it possibly make people more confident in the existence of objective reality if they know that the U.S. government manipulates the information disseminated in social media? What’s to prevent the appearance of new social media based outside the United States whose main selling point is that they don’t cooperate with the U.S. government? How much of a propaganda coup would it be for Putin if in response to RT and Sputnik, America develops its own state media aimed at its own people? How long before a new Snowden gives Wikileaks hundreds of pages of memos and PowerPoint presentations describing the inner workings, hidden aims and occasional abuses of the government’s “truth machine”? Cognitive Security could easily be the coffin in which objective truth is eventually buried.
Busy little fingers
I really don’t think that Rand Waltzman’s intentions are evil, from his subjective perspective. Frankly, same with Admiral Poindexter. In the case of Total Information Awareness, you have to remember that after 9/11 there was public outrage that the government had failed to “connect the dots” and prevent the attack before it happened, like in “Minority Report” but, you know, not using magical telepaths lying in milk.
There is nothing surprising or wrong with the fact that there are, at all times, “idea men” like Waltzman and Poindexter working in and around the government, who aren’t afraid to come up with out-of-the-box solutions that are hard to stomach in “normal” times, but suddenly seem more palatable after something bad happens.
It’s important to remember, when emotions are high, that when the government comes under pressure to do the impossible, government officials will ask for powers that were previously considered unthinkable. And there will always be someone like Rand Waltzman or John Poindexter with a ready-made solution to sell. There’s nothing evil about that, but it means that we have to overcome our emotional response, maintain our skepticism and remember that human nature is not to relinquish power once it’s granted.
Will the Thought Police–sorry, Cognitive Security–sometimes mistakenly (or, sometimes, intentionally) mislabel U.S. citizens’ speech as foreign propaganda? Of course it will. Will it tend to use its influencing powers to portray the U.S. government’s (including the president’s) actions and proposals in a favorable light, and suppress criticism, even conscientious criticism? Sure, it’s human nature. And those are just the incidental effects.
What could a multi-billion-dollar, government-controlled, covertly operated public opinion influencing machine do in the wrong hands? And is there such a thing as the “right hands”* for that kind of power? So far we don’t have a TV series about that, but it’s enough to make one shudder to imagine it.