Weapons
of Math Destruction: How Big Data increases inequality and threatens
democracy.
Cathy
O'Neil (Crown, 2016)
Cathy
O'Neil is a math nerd who, in 2007, left her academic career to do
math for a major hedge fund. She thus had a front row seat for the
financial catastrophe of 2008, which was in large part a product of
the kind of mathematical processes people like her were engaged in.
"The housing crisis, the collapse of major financial
institutions, the rise of unemployment–all had been aided and
abetted by mathematicians wielding magic formulas.What's more, thanks
to the extraordinary powers that I loved so much, math was able to
combine with technology to multiply the chaos and misfortune, adding
efficiency and scale to systems that I now recognized as flawed."
After
leaving D. E. Shaw, O'Neil recast herself as a data scientist,
working on models in the Big Data economy, and once again privy to
the magic formulas that are taking over in so many areas of our
lives, from college admissions to parole recommendations. She started
her blog, MathBabe, "to mobilize fellow mathematicians against
the use of sloppy statistics and biased models that created their own
toxic feedback loops."
In
Weapons of Math Destruction, she expands that warning to
general readers, including the mathematically challenged. In fact,
for the kind of algorithm she describes as a WMD, the math is usually
opaque, anyhow: it's proprietary to the company that is profiting
from it, whether by sorting your resume by the ZIP code you come
from, or selling your search clicks to on-line marketers. The
black-box quality is part of what makes WMDs so dangerous: the
numbers they spit out are nearly impervious to challenges, even
though the numbers that went into them may be biased, false, or
completely spurious.
Another
dangerous aspect of WMDs is its potential to damage people's lives.
O'Neil has a case study about the Washington, D.C., school
department's program of rewarding teachers whose students improve,
and firing those whose students don't. Such a practice can (and did)
lead to the firing of good teachers, if the previous year's teachers
cheated by padding their students' scores. The school hierarchy got
what it wanted, to be seen as weeding out underperforming teachers,
but since they didn't do any external checking to see if that's
actually what they had done, they don't know how many good teachers
they lost in the process. A healthy model would have an independent
way of looking at the results to see if they made sense.
That's
unlikely to happen, however, when the third destructive effect starts
to work: scale. If your credit report were managed by your own bank,
you could speak with them about it face to face, and presumably
establish that you were not the same John Bradshaw who had defaulted
on that electric bill three years ago. But scale it up to the size of
the big three credit bureaus, over billions of data points, and you
are unlikely to find a person who can fix mixups; but you may very
well pay for the errors not only in higher interest rates, but also
in difficulty getting a loan, or a job, at all.
And
that's to mention only the official credit bureaus, which are
governed by requirements that let you see the data they're using, and
challenge it. E-scores generated by studies of internet use, or
assumption about the street you live on, are under no such
constraint, and their feedback loops tend to make unfairness worse.
"There's a very high chance that the e-scoring system will give
the borrower from the rough section of East Oakland a low score. A
lot of people default there. So the credit card offer popping up on
her screen will be targeted to a riskier demographic. That means less
available credit and higher interest rates for those who are already
struggling."
Most
WMDs embody corporate goals such as efficiency and profit; if
corporations are persons, they tend to be sociopathic ones. Human
beings are much better at thinking about justice than computers are,
still, and perhaps always–if we choose to, and if we know what
we're up against. Weapons of Math Destruction is disturbing,
and distressing, but I couldn't put it down. Cathy O'Neil is a
warrior for economic justice, and we ignore her at our peril.