• US and UK spy agencies piggyback on commercial data
• Details can include age, location and sexual orientation
• Documents also reveal targeted tools against individual phones
GCHQ documents use Angry Birds -- reportedly downloaded more than 1.7bn times -- as a case study for app data collection.
The National Security Agency and its UK counterpart GCHQ have been developing capabilities to take advantage of "leaky" smartphone apps, such as the wildly popular Angry Birds game, that transmit users' private information across the internet, according to top secret documents.
The data pouring onto communication networks from the new generation of iPhone and Android apps ranges from phone model and screen size to personal details such as age, gender and location. Some apps, the documents state, can share users' most sensitive information such as sexual orientation -- and one app recorded in the material even sends specific sexual preferences such as whether or not the user may be a swinger.
Many smartphone owners will be unaware of the full extent this information is being shared across the internet, and even the most sophisticated would be unlikely to realise that all of it is available for the spy agencies to collect.
Dozens of classified documents, provided to the Guardian by whistleblower Edward Snowden and reported in partnership with the New York Times and ProPublica, detail the NSA and GCHQ efforts to piggyback on this commercial data collection for their own purposes.
Scooping up information the apps are sending about their users allows the agencies to collect large quantities of mobile phone data from their existing mass surveillance tools -- such as cable taps, or from international mobile networks -- rather than solely from hacking into individual mobile handsets.
Exploiting phone information and location is a high-priority effort for the intelligence agencies, as terrorists and other intelligence targets make substantial use of phones in planning and carrying out their activities, for example by using phones as triggering devices in conflict zones. The NSA has cumulatively spent more than $1bn in its phone targeting efforts.
The disclosures also reveal how much the shift towards smartphone browsing could benefit spy agencies' collection efforts.
One slide from a May 2010 NSA presentation on getting data from smartphones -- breathlessly titled "Golden Nugget!" -- sets out the agency's "perfect scenario": "Target uploading photo to a social media site taken with a mobile device. What can we get?"
The question is answered in the notes to the slide: from that event alone, the agency said it could obtain a "possible image", email selector, phone, buddy lists, and "a host of other social working data as well as location".
In practice, most major social media sites, such as Facebook and Twitter, strip photos of identifying location metadata (known as EXIF data) before publication. However, depending on when this is done during upload, such data may still, briefly, be available for collection by the agencies as it travels across the networks.
Depending on what profile information a user had supplied, the documents suggested, the agency would be able to collect almost every key detail of a user's life: including home country, current location (through geolocation), age, gender, zip code, marital status -- options included "single", "married", "divorced", "swinger" and more -- income, ethnicity, sexual orientation, education level, and number of children.
The agencies also made use of their mobile interception capabilities to collect location information in bulk, from Google and other mapping apps. One basic effort by GCHQ and the NSA was to build a database geolocating every mobile phone mast in the world -- meaning that just by taking tower ID from a handset, location information could be gleaned.
A more sophisticated effort, though, relied on intercepting Google Maps queries made on smartphones, and using them to collect large volumes of location information.
So successful was this effort that one 2008 document noted that "[i]t effectively means that anyone using Google Maps on a smartphone is working in support of a GCHQ system."
The information generated by each app is chosen by its developers, or by the company that delivers an app's adverts. The documents do not detail whether the agencies actually collect the potentially sensitive details some apps are capable of storing or transmitting, but any such information would likely qualify as content, rather than metadata.
Data collected from smartphone apps is subject to the same laws and minimisation procedures as all other NSA activity -- procedures that the US president, Barack Obama, suggested may be subject to reformin a speech 10 days ago. But the president focused largely on the NSA's collection of the metadata from US phone calls and made no mention in his address of the large amounts of data the agency collects from smartphone apps.
The latest disclosures could also add to mounting public concern about how the technology sector collects and uses information, especially for those outside the US, who enjoy fewer privacy protections than Americans. A January poll for the Washington Post showed 69% of US adults were already concerned about how tech companies such as Google used and stored their information.
The documents do not make it clear how much of the information that can be taken from apps is routinely collected, stored or searched, nor how many users may be affected. The NSA says it does not target Americans and its capabilities are deployed only against "valid foreign intelligence targets".
The documents do set out in great detail exactly how much information can be collected from widely popular apps. One document held on GCHQ's internal Wikipedia-style guide for staff details what can be collected from different apps. Though it uses Android apps for most of its examples, it suggests much of the same data could be taken from equivalent apps on iPhone or other platforms.
The GCHQ documents set out examples of what information can be extracted from different ad platforms, using perhaps the most popular mobile phone game of all time, Angry Birds -- which has reportedly been downloaded more than 1.7bn times -- as a case study.
From some app platforms, relatively limited, but identifying, information such as exact handset model, the unique ID of the handset, software version, and similar details are all that are transmitted.
Other apps choose to transmit much more data, meaning the agency could potentially net far more. One mobile ad platform, Millennial Media, appeared to offer particularly rich information. Millennial Media's website states it has partnered with Rovio on a special edition of Angry Birds; with Farmville maker Zynga; with Call of Duty developer Activision, and many other major franchises.
Rovio, the maker of Angry Birds, said it had no knowledge of any NSA or GCHQ programs looking to extract data from its apps users.
"Rovio doesn't have any previous knowledge of this matter, and have not been aware of such activity in 3rd party advertising networks," said Saara Bergström, Rovio's VP of marketing and communications. "Nor do we have any involvement with the organizations you mentioned [NSA and GCHQ]."
Millennial Media did not respond to a request for comment.
In December, the Washington Post reported on how the NSA could make use of advertising tracking files generated through normal internet browsing -- known as cookies -- from Google and others to get information on potential targets.
However, the richer personal data available to many apps, coupled with real-time geolocation, and the uniquely identifying handset information many apps transmit give the agencies a far richer data source than conventional web-tracking cookies.
"They are gathered in bulk, and are currently our single largest type of events," the document stated.
The ability to obtain targeted intelligence by hacking individual handsets has been well documented, both through several years of hacker conferences and previous NSA disclosures inDer Spiegel, and both the NSA and GCHQ have extensive tools ready to deploy against iPhone, Android and other phone platforms.
GCHQ's targeted tools against individual smartphones are named after characters in the TV series The Smurfs. An ability to make the phone's microphone 'hot', to listen in to conversations, is named "Nosey Smurf". High-precision geolocation is called "Tracker Smurf", power management -- an ability to stealthily activate an a phone that is apparently turned off -- is "Dreamy Smurf", while the spyware's self-hiding capabilities are codenamed "Paranoid Smurf".
Those capability names are set out in a much broader 2010 presentation that sheds light on spy agencies' aspirations for mobile phone interception, and that less-documented mass-collection abilities.
The cover sheet of the document sets out the team's aspirations:
Another slide details weak spots in where data flows from mobile phone network providers to the wider internet, where the agency attempts to intercept communications. These are locations either within a particular network, or international roaming exchanges (known as GRXs), where data from travellers roaming outside their home country is routed.
These are particularly useful to the agency as data is often only weakly encrypted on such networks, and includes extra information such as handset ID or mobile number -- much stronger target identifiers than usual IP addresses or similar information left behind when PCs and laptops browse the internet.
The NSA said its phone interception techniques are only used against valid targets, and are subject to stringent legal safeguards.
"The communications of people who are not valid foreign intelligence targets are not of interest to the National Security Agency," said a spokeswoman in a statement.
"Any implication that NSA's foreign intelligence collection is focused on the smartphone or social media communications of everyday Americans is not true. Moreover, NSA does not profile everyday Americans as it carries out its foreign intelligence mission. We collect only those communications that we are authorized by law to collect for valid foreign intelligence and counterintelligence purposes -- regardless of the technical means used by the targets.
"Because some data of US persons may at times be incidentally collected in NSA's lawful foreign intelligence mission, privacy protections for US persons exist across the entire process concerning the use, handling, retention, and dissemination of data. In addition, NSA actively works to remove extraneous data, to include that of innocent foreign citizens, as early as possible in the process.
"Continuous and selective publication of specific techniques and tools lawfully used by NSA to pursue legitimate foreign intelligence targets is detrimental to the security of the United States and our allies -- and places at risk those we are sworn to protect."
The NSA declined to respond to a series of queries on how routinely capabilities against apps were deployed, or on the specific minimisation procedures used to prevent US citizens' information being stored through such measures.
GCHQ declined to comment on any of its specific programs, but stressed all of its activities were proportional and complied with UK law.
"It is a longstanding policy that we do not comment on intelligence matters," said a spokesman.
"Furthermore, all of GCHQ's work is carried out in accordance with a strict legal and policy framework that ensures that our activities are authorised, necessary and proportionate, and that there is rigorous oversight, including from the Secretary of State, the Interception and Intelligence Services Commissioners and the Parliamentary Intelligence and Security Committee. All our operational processes rigorously support this position."
• A separate disclosure on Wednesday, published byGlenn Greenwald and NBC News, gave examples of how GCHQ was making use of its cable-tapping capabilities to monitor YouTube and social media traffic in real-time.
GCHQ's cable-tapping and internet buffering capabilities , codenamed Tempora, were disclosed by the Guardian in June, but the new documents published by NBC from a GCHQ presentation titled "Psychology: A New Kind of SIGDEV" set out a program codenamed Squeaky Dolphin which gave the British spies "broad real-time monitoring" of "YouTube Video Views", "URLs 'Liked' on Facebook" and "Blogspot/Blogger Visits".
A further slide noted that "passive" -- a term for large-scale surveillance through cable intercepts -- give the agency "scalability".
The means of interception mean GCHQ and NSA could obtain data without any knowledge or co-operation from the technology companies. Spokespeople for the NSA and GCHQ told NBC all programs were carried out in accordance with US and UK law.
• This article was amended on 28 January 2014. It referred to martial status, instead of marital status. This has been corrected.