Disarming ‘Weapons of Math Destruction’

In the world of big data, there’s a lot of music to be faced. With all its upside, data science’s deployment risks being prejudicial, predatory, exploitative, buggy, blindly trusted and secretive. And it has the potential to magnify the consumer’s personal economic struggle rather than remedy it.

These risks permeate across the field. The book’s broad coverage includes examples from all the main business application areas to which predictive models commonly apply: marketing, online ads, credit scoring, insurance, workforce analytics, law enforcement and political campaigns.

By providing such a uniquely comprehensive treatment of data’s downside, the book addresses two dire needs: increasing awareness and opening the door to prolific discussion.

When exercising the power of analytics, what could be more important than that? Establishing and managing ethical data science is as important as including the brakes in a state-of-the-art automobile. O’Neil’s book pioneers much-needed first steps. 

Risks to Social Justice

“Weapons of Math Destruction” covers a range of ethical dilemmas. The sociological risks of data science include:

Magnifying the social divide. Predicting a poor credit risk is, to some degree, a self-fulfilling prophecy that can hurl those less financially secure into a vicious cycle. This pitfall applies analogously when prison sentencing may be guided in part by a felon’s neighborhood, and when a university places low on U.S. News & World Report college rankings. More generally, data science may amplify capitalism’s tendency to further disempower the disenfranchised, i.e., punish the poor.

Racial prejudice. Law enforcement models that predict crime and recidivism have been shown to intrinsically enact biases against minority groups. They are swayed in part by who you are rather than what you’ve done.

Predatory micro-targeting. Highly targeted online ads are more adept than ever at exploiting vulnerable consumers and separating them from their money.

Opaque, overly trusted and sometimes buggy. Whether buggy (in some demonstrable cases) or unjust, predictive models often are not disclosed transparently – hidden from scrutiny – and in some cases are overly trusted by those who rely on their predictions.

But Data Science is Largely Good

The book’s main shortcoming to keep in mind while you’re (hopefully) reading it is that – in most places – it appears to sweepingly indict and vilify data science.

Ultimately, that’s going too far. A little extra copy could have easily clarified as much up front in the introduction by saying something like, “The risks are dire, measures must be taken to protect social justice when deploying data science, and here are several examples along with suggested adjustments to implement such protective measures.”

Instead, the author begins the book by regaling her personal story of defecting completely from data science’s commercial practice as a generally unethical endeavor, and the remainder of the book covers a list of examples that will sound to many readers like no less than a diatribe. The book’s opening anecdote – on a flawed teacher evaluation metric – leaves out that, instead of doing away with the entire system, such a bug could in fact be remedied. Most mentions of micro-targeting tacitly imply it always necessarily serves greed or otherwise enacts some forms of injustice. The book focuses almost exclusively on the negative – depicting one train wreck after another – with relatively little to balance that out in the way of constructive advice that could actively mitigate risks to social justice within existing practices. The implicit advice is to basically turn it all off.

But, if you read the full book in detail, you will indeed come across this intelligent author stating that, no, math is not inherently evil, and it can also serve for the greater good. She goes further with the following two quotes, each relatively buried within the book (the latter is literally the last sentence of the last chapter before the Conclusion):

“… mathematical models can sift through data to locate people who are likely to face great challenges, whether from crime, poverty or education. It’s up to society whether to use that intelligence to reject and punish them – or to reach out to them with the resources they need. We can use the scale and efficiency that make ‘weapons of math destructions’ so pernicious in order to help people. It all depends on the objective we choose.”

“… with most ‘weapons of math destruction,’ the heart of the problem is almost always the objective. Change that objective from leeching off people to helping them, and a WMD is disarmed – and can even become a force for good.”

I would go further than O’Neil and claim that many deployments of predictive models, even when designed to pursue profit as the objective rather than social justice, do more good than bad. Profit by way of efficiency is not always a bad thing. Moreover, consumers gain value by way of predictive models: less junk mail (and better for the environment), more relevant ads, better movie, music and books recommendations, effective email spam filters, better Google search results, more engaging Facebook feed content, more robust healthcare, and increased safety by more effectively targeting the inspection of buildings and manholes.

This technology is like a knife: Its power can be used for good or for evil. That means it can be dangerous, but the idea of completely eliminating it – or even just its profit-driven deployment – is not on the table.

It’s important not to go too far and criticize the entire field of data science in absolute terms. By painting it completely black, you compromise your own credibility and weaken your valuable voice in the call for social justice.

Having said that, I strongly support O’Neil’s inspirational motion that you go out and pursue social justice rather than – or at least in addition to – profit.

Conclusion

While “Weapons of Math Destruction” conveys an oversimplifying, “black-and-white” position, that’s just one aspect of what amounts to a broad treatment of a multi-faceted, critical topic. I encourage you to look past that aspect in order to gain from this important book. I anointed it five stars on Amazon, and I strongly recommend you read it as well and pass on the word.