Victimized by WMD

My family and I are spending the second half of my sabbatical in Madrid. We arrived in Spain a couple of weeks ago, and the day we arrived I tried to log in to my bank’s online site to pay my credit card bill.

Quite unexpectedly, I received a message informing me that before logging on I would need to enter a verification code that had just been sent in a text message to my U.S. cell number. Alas, this number is temporarily turned off while I am overseas, so I was unable to log in. I wound up on the phone for more than an hour – at international telecomm rates! – with a customer service representative (who could not successfully help me) and then with her supervisor (who did ultimately help me get on to the online banking site only after having to change my login name). Feeling triumphant, I logged in to my bank’s site, paid my credit card bill and went to bed feeling that the matter had been resolved.

When I woke up the next morning, however, I had received an email from my bank’s fraud detection department saying that it: (a) suspected that the payment I made was fraudulent; (b) was canceling the payment; and (c) was suspending my online access until I called them. After spending another hour on another international phone call, I was informed that the payment could not be re-submitted and that my online access could not be restored due to a technical issue (“a known bug in the system,” the agent told me sheepishly).

Dr. Cathy O’Neil, author of the book “Weapons of Math Destruction” [1], would be quick to say that I had been victimized by classic WMDs, a term that she coined to describe applied mathematical models with the following attributes:

Opacity: It was not at all clear why my access to online banking had been revoked. Yes, I was accessing my account from outside of the country, but I had informed the bank of my travel plans several weeks before leaving. Not only could I not figure it out, but neither could the many bank employees who I spoke with over the phone. Could it have been my foreign last name? Note that my wife had no trouble accessing her accounts at the same bank from Spain.

To O’Neil, this is typical: “… these mathematical models were opaque, their workings invisible to all but the highest priests in their domain,” she writes.

Scale: My bank has tens of millions of customers worldwide. So, the same models that were being used to prevent me from accessing my account online and paying my bills are undoubtedly blocking other customers in other circumstances from accessing their accounts as well. As O’Neil points out, “scale is what turns WMDs from local nuisances to into tsunami forces.”

Damage: The potential cost to me of a late credit card payment, especially on a bill that included international plane flights for my entire family, was substantial. And despite an almost endless stream of apologies from the various bank employees over the phone, there was absolutely nothing either they or I could do about it.

This is just as infuriating to O’Neil as it was to me: “You cannot appeal to a WMD. That part of their fearsome power. They do not listen. Nor do they bend. They’re deaf not only to charm, threats and cajoling but also to logic.”

O’Neil’s book has a special focus on social justice (its subtitle is “How Big Data Increases Inequality and Threatens Democracy”). This focus is evident in her text and in many of the examples of WMDs that she describes, including credit scoring, payday lending, predictive policing [2], criminal recidivism, predatory online marketing, employee selection and staff scheduling. She convincingly argues that many of these models perniciously combine to reduce social mobility for the poor (while also enabling the wealthier classes with increasingly personalized options for all kinds of products and services). She sadly observes that, “Being poor in a world of WMDs is getting more and more dangerous and expensive.”

Two of O’Neil’s key criticisms of WMDs are the narrowness of their objectives and the absence of feedback. For example, when a for-profit university successfully targets students who then take out huge government-guaranteed loans before leaving school with no appreciable improvement in their job prospects, the institution views these models as “successful,” even though the cost to society – and to the unsuspecting students who bought into the sales pitch – are substantial. Moreover, information about situations where models have failed – for example, when an employee screening model filters out an individual who turns out to be extremely successful elsewhere – is too often simply not considered, resulting in models that codify what was reflected in their initial assumptions and inputs. Meanwhile, the book also describes many instances in which data is being captured and aggregated across multiple sources and through various middlemen (all of whom have their own narrow objectives and lack of feedback), propagating data errors and biases across models and industries.

Overall, O’Neil does an excellent job of describing very serious problems that have too often been ignored or discounted by the chorus of analytics cheerleaders, this writer included. Solutions, however, are harder to pin down. In the book’s final chapter, she offers some suggestions, including self-policing, richer models that measure broader metrics, changes in laws surrounding the use of data (including updates to the Fair Credit Reporting Act, the Americans with Disability Act, and the Health Insurance Portability and Accountability Act to prohibit discrimination that is driven by predictive models), calls for businesses to make their models more transparent, and academic research using agent-based simulations to help provide an understanding of the logic underlying WMDs and of the results that they produce.

Ultimately, O’Neil clearly believes that these problems can only be addressed with the active participation of analytics professionals, exhorting us to “come together to police these WMDs, to tame and disarm them.” And my sense is that it will require not only a great deal of intellectual firepower, but also a clear understanding of one’s own values and a great deal of courage. Kudos to O’Neil – a mathematician whose career has included stints as a college professor, Wall Street quant and data scientists – for sounding the alarm. We are all challenged to answer its call.