View from London: Private intelligence companies became the new spymasters in Internet

As society has migrated to the internet, so have its secrets, and, therefore, so has intelligence, notes ‘The Engelsberg Ideas’.

Consider the deep web, a part of the internet that is not indexed by search engines, and the dark web, which requires specialised software to access. They offer a degree of anonymity attractive to a variety of unsavoury people: terrorists, paedophiles, drug dealers, and cyber-criminals. But that anonymity is superficial.

Consider the example of Flashpoint, a so-called threat intelligence firm. It now normally deals in data. By tracing extremist groups’ cryptocurrency ‘wallets’, for instance, you can spot anomalous movements that might hint at an impending attack. This kind of intelligence can be semi-secret: tucked out of sight, accessible but often ephemeral. Joseph Cox, a journalist, notes that administrators of criminal and hacker chat rooms on Telegram, a social media platform, frequently wipe messages in one channel and establish another. ‘It really is like missing a whispered conversation in the bar.’ Collecting those messages requires vigilance or automation.

If one approach is to observe what is happening out there – on the internet, on the deep and dark week, in particular places – then another is to combine that with what is happening inside one’s own networks  – ‘in host’. The firms that build key hardware and software – Google for email, Microsoft for operating systems, and Amazon for cloud computing, to name a few – have unprecedented and unmatched insight into the traffic moving over their networks. The result is that these companies are, in one sense, the largest signals intelligence agencies on the planet.

Microsoft tracks more than 78 trillion ‘signals’ per day.

These companies observe not just the traffic on their own networks but, like counter-intelligence services, map and track the activities and signatures of their adversaries, including state-linked hacking groups known as advanced persistent threats or APTs. The fact that Western cyber-security companies have been involved in the defence of Ukrainian networks from the earliest days of the war means that they also see some Russian cyber threats that Western agencies might not be aware of.

All this is an opportunity for spycraft. For one thing, it expands collection capacity. Take the example of the Falklands War. America found that its spy satellites, designed to watch the Soviet Union, were in the wrong orbit to point at the South Atlantic (‘Nobody ever thought there’d be a damn war in the Falklands for God’s sake’, noted Robert Gates, later the CIA director). The private sector has since solved that problem. The spectacular growth of the commercial satellite industry allows states to enjoy near-blanket coverage. Britain has gone from buying hundreds of thousands of dollars of commercial satellite images every year to multiple millions.

A second advantage is that secrets acquired by non-secret agencies can be shared more widely. In space intelligence, for instance, according to the historian Aaron Bateman, the United States rarely shared satellite images with its NATO allies except Britain. In some cases it did not acknowledge certain sorts of satellites, such as those which collected radio emissions or which used synthetic aperture radar, even existed. That began to change in 1991 during the first Gulf War. But it is now routine for governments to buy and publish high-resolution satellite images to expose malfeasance by an adversary.

The most radical interpretation of all these changes is that Western intelligence is broken and needs to start again from scratch. ‘The UK intelligence community (UKIC) is facing an existential challenge,’ argued Lucy Mason, a former British defence official, and Jason M, a semi-anonymous serving intelligence official, in a paper published by the Alan Turing Institute, a research centre in London that works closely with the intelligence services, in November. ‘It is being out-competed by providers of open-source intelligence and data companies.’ The authors proposed a completely new model ‘away from one where national security is done only by some cleared people in highly centralised, closed, organisations, to one which is open, collaborative, and joined up by design’.

To be sure, non-secret sources are increasingly important. Open source contributed around 20 per cent of British defence intelligence ‘current processes’, noted General Jim Hockenhull, then chief of the service, in late 2022, ‘but the availability and opportunity means that we’ve got to invert this metric.’ The same appetite exists in the non-military intelligence world. ‘If I’d gone and collected all of China’s military procurement records, I’d probably have got an OBE,’ says a former British intelligence officer. ‘The fact that they were, for many years, just sat there in open source just completely bypassed everybody.’ A flourishing trade in personal location data harvested by advertising brokers from apps on mobile phones is a rich seam for state agencies around the world. In April 2024 America’s communications regulator levied $200m in fines on the country’s largest telecoms firms for selling such data without permission to firms who then sold it on again.

‘If we do not find a way to merge the great capabilities of Western governments and the private sector to defend our own values and interests’, argues Duyane Norman, a former CIA officer, ‘these adversaries will continue to close the gap.’

That is easier said than done. Democracies tend to impose stringent requirements on the sort of thing that may or may not be collected. In Britain the intelligence agencies do collect ‘bulk personal data’, but if they want to ‘retain or examine’ it then they must jump through a few hoops: they need to get a warrant and then show that getting, keeping and using it is proportionate to some specific aim. It is not enough to believe that it might prove useful. Some data is thus ‘more easily accessed and used by the private sector than by government organisations’, write Lucy Mason and Mr M, the authors of the paper published by the Alan Turing Institute.

In 2013 the disclosures by Edward Snowden, contractor working for America’s National Security Agency, prompted an intense and unexpected public debate over the activities of intelligence agencies and their ability to collect, if not actively read, vast amounts of phone, internet and other traffic. In the decade since, much has changed. The majority of internet browsing and personal messaging now takes place with the protection of end-to-end encryption, making it harder for spies to read what they might intercept. More data is also being encrypted ‘at rest’ – on devices, and in use. That trend, too, has been driven by the private sector, as large tech companies – Apple, Google and Meta (banned in Russia), above all – have embraced encryption and user privacy in the face of opposition from law enforcement agencies around the world.

At the same time, daily life relies more than ever on digital technology: more things run on software (fridges, cars, phones), those things have a greater array of sensors (GPS receivers and radio transmitters) and they are increasingly connected, often over the internet, allowing data, often embodying our most personal secrets, to flow to and fro. The paradox of the modern world is that, while we have more means to keep our data secret, there is so much more data to contend with and so many more places from where it can seep out into the world, where a sprawling ecosystem of private intelligence can collect, analyse and use it.

read more in our Telegram-channel https://t.me/The_International_Affairs