Want To See Your Dystopian Future? Look at China

(Bloomberg Opinion) -- South Koreans are worried about the surveillance state. At a recent conference that was supposed to be about protests — such as pro-democracy demonstrations and the #MeToo movement — journalists invariably asked me one question: How do we maintain democracy in an age when governments and companies are collecting ever more data on everything we do?

They’re right to be concerned. If people don’t recognize the danger, they might eventually be unable to protest at all.

Thanks to their proximity to China, South Koreans are witnessing up close that country’s efforts to deploy big data technology as a tool of social control. Omnipresent surveillance cameras, trained in facial recognition, record people’s whereabouts. The ubiquitous social credit score monitors how people behave on the train, whether they pay their bills on time, and whether they participate in undesirable organizations.

This is a real threat. The potential accuracy and reach of government surveillance will only increase as the data pile up and the technology improves — particularly in places such as China, where there are few if any privacy protections. Worse, in many cases, it doesn’t even have to work as advertised. The mere knowledge that they are being watched will often be enough to keep people in line. And if mistakes are made — if, for example, the system misidentifies criminals or wrongly dings someone’s social score — repressive governments won’t necessarily be held to account.

China’s heavy-handed approach is spreading. Venezuela is reportedly using Chinese technology to create its own social credit score. The Saudi government has been monitoring Twitter to identify and punish political dissidents — an effort in which the consulting company McKinsey may have (perhaps inadvertently) played a role.

Governments will wage the next cold war with technology, in large part against their own citizens, but the nature and ideology of that conflict are still uncertain. So far, China’s version appears to be in the lead, in part because the world’s great superpower, the U.S., still hasn’t figured out what its strategy will be. Discussions about the use of big data and artificial intelligence in the U.S. and China tend to frame it as a technology race, rather than a clash of values. From that perspective, privacy protection is merely a pesky constraint to the innovation that will bring about the brave new world.

Trying to compete with China in developing social control technology would be a terrible mistake. Europe’s General Data Protection Regulation explicitly disavows such an approach, choosing instead to defend human dignity and freedom. In the U.S., though, people are too distracted by the political circus to take on a creeping surveillance state, starting with big tech firms such as Facebook, which have demonstrated their incapacity to promote democratic values over profit.

We need to do better, and soon.

To contact the editor responsible for this story: Mark Whitehouse at mwhitehouse1@bloomberg.net

This column does not necessarily reflect the opinion of the editorial board or Bloomberg LP and its owners.

Cathy O’Neil is a Bloomberg Opinion columnist. She is a mathematician who has worked as a professor, hedge-fund analyst and data scientist. She founded ORCAA, an algorithmic auditing company, and is the author of “Weapons of Math Destruction.”

©2018 Bloomberg L.P.