A Secret Algorithm Is Deciding Who Will Die in America
Authorities’ reliance on inscrutable algorithms to make important decisions, may end up costing lives and livelihoods.
(Bloomberg Opinion) -- I have written extensively on the dangers that arise when authorities rely on inscrutable algorithms to make important decisions, bypassing public conversations. All too often, their statistically careless models end up costing lives and livelihoods.
Sadly, this appears to be precisely what will happen with the Trump administration’s Covid-19 model.
Arizona Governor Doug Ducey announced plans to allow businesses to start reopening this week after consulting the Federal Emergency Management Agency’s new pandemic-prediction model, which has not been released to the public. The move came as a shock because a team of university experts -- who had developed their own model on the state’s behalf -- had advised waiting until the end of May, lest the hospital system be overwhelmed.
In my book, “Weapons of Math Destruction,” I identified three properties that make a predictive algorithm particularly dangerous: It must be important, secret and destructive. FEMA’s new model has them all. If it can persuade a governor to lift stay-at-home orders in the middle of a pandemic, it’s important. As of this writing, its details remain a secret. And while what will happen in Arizona remains to be seen, it has the potential to destroy many lives by justifying bad decisions.
Such models don’t arise accidentally. They pop up when people want to avoid a difficult conversation, especially one that involves historical struggle, money and complex decisions. Illustrative examples include a model that assessed public school teachers, and the “crime risk score” algorithms that decide, among other things, who gets jailed pre-trial. Both were deeply flawed statistical embarrassments intended to bypass tricky issues, such as what makes a good teacher and who deserves incarceration. Yet governments and companies keep using black-box models, because people -- due to fear of math or lack of information –- so rarely challenge them.
If there was ever a subject that required a difficult public conversation, the response to Covid-19 is it. How many people are we willing to let die in order to keep businesses working? And which people will we let die? By creating a secret model to inform such decisions, the Trump administration is taking these questions out of the public and scientific spheres, replacing data-driven ethical debate with a pseudo-mathematical political tool. This is bad news for science, and potentially terrible news for Arizona residents.
This column does not necessarily reflect the opinion of the editorial board or Bloomberg LP and its owners.
Cathy O’Neil is a Bloomberg Opinion columnist. She is a mathematician who has worked as a professor, hedge-fund analyst and data scientist. She founded ORCAA, an algorithmic auditing company, and is the author of “Weapons of Math Destruction.”
©2020 Bloomberg L.P.