Skip to main content

Weapons of Math Destruction - Cathy O'Neil ****

As a poacher-turned-gamekeeper of the big data world, Cathy O'Neil is ideally placed to take us on a voyage of horrible discovery into the world of systems making decisions based on big data that can have a negative influence on lives - what she refers to as 'Weapons of Math Destruction' or WMDs. After working as a 'quant' in a hedge fund and on big data crunching systems for startups, she has developed a horror for the misuse of the technology and sets out to show us how unfair it can be.
It's not that O'Neil is against big data per se. She points out examples where it can be useful and effective - but this requires the systems to be transparent and to be capable of learning from their mistakes. In the examples we discover, from systems that rate school teachers to those that decide whether or not to issue a payday loan, the system is opaque, secretive and based on a set of rules that aren't tested against reality and regularly updated to produce a fair outcome.
The teacher grading system is probably the most dramatically inaccurate example, where the system is trying to measure how well a teacher has performed, based on data that only has a very vague link to actual outcomes - so, for instance, O'Neil tells of a teacher who scored 6% one year and 96% the next year for doing the same job. The factors being measured are almost entirely outside the teacher's control with no linkage to performance and the interpretation of the data is simply garbage.
Other systems, such as those used to rank universities, are ruthlessly gamed by the participants, making them far more about how good an organisation is at coming up with the right answers to metrics than it is to the quality of that organisation. And all of us will come across targeted advertising and social media messages/search results prioritised according to secret algorithms which we know nothing about and that attempt to control our behaviour.
For O'Neil, the worst aspects of big data misuse are where a system - perhaps with the best intentions - ends up penalising people for being poor of being from certain ethnic backgrounds. This is often a result of an indirect piece of data - for instance the place they live might have implications on their financial state or ethnicity. She vividly portrays the way that systems dealing with everything from police presence in an area to fixing insurance premiums can produce a downward spiral of negative feedback.
Although the book is often very effective, it is heavily US-oriented, which is a shame when many of these issues are as significant, say, in Europe, as they are in the US. There is probably also not enough nuance in the author's binary good/bad opinion of systems. For example, she tells us that someone shouldn't be penalised by having to pay more for insurance because they live in a high risk neighbourhood - but doesn't think about the contrary aspect that if insurance companies don't do this, those of us who live in low risk neighbourhoods are being penalised by paying much higher premiums than we need to in order to cover our insurance. 

O'Neil makes a simplistic linkage between high risk = poor, low risk = rich - yet those of us, for instance, who live in the country are often in quite poor areas that are nonetheless low risk. For O'Neil, fairness means everyone pays the same. But is that truly fair? Here in Europe, we've had car insurance for young female drivers doubled in cost to make it the same as young males - even though the young males are far more likely to have accidents. This is fair by O'Neil's standards, because it doesn't discriminate on gender, but is not fair in the real world away from labels.
There's a lot here that we should be picking up on, and even if you don't agree with all of O'Neil's assessments, it certainly makes you think about the rights and wrongs of decisions based on automated assessment of indirect data.


Review by Brian Clegg


Popular posts from this blog

Conjuring the Universe - Peter Atkins *****

It's rare that I'd use the term 'tour de force' when describing a popular science book, but it sprang to mind when I read Conjuring the Universe. It's not that the book's without flaws, but it does something truly original in a delightful way. What's more, the very British Peter Atkins hasn't fallen into the trap that particularly seems to influence US scientists when writing science books for the public of assuming that more is better. Instead of being an unwieldy brick of a book, this is a compact 168 pages that delivers splendidly on the question of where the natural laws came from.

The most obvious comparison is Richard Feynman's (equally compact) The Character of Physical Law - but despite being a great fan of Feynman's, this is the better book. Atkins begins by envisaging a universe emerging from absolutely nothing. While admitting he can't explain how that happened, his newly created universe still bears many resemblances to  nothing a…

Beyond Weird - Philip Ball *****

It would be easy to think 'Surely we don't need another book on quantum physics.' There are loads of them. Anyone should be happy with The Quantum Age on applications and the basics, Cracking Quantum Physics for an illustrated introduction or In Search of Schrödinger's Cat for classic history of science coverage. Don't be fooled, though - because in Beyond Weird, Philip Ball has done something rare in my experience until Quantum Sense and Nonsense came along. It makes an attempt not to describe quantum physics, but to explain why it is the way it is.

Historically this has rarely happened. It's true that physicists have come up with various interpretations of quantum physics, but these are designed as technical mechanisms to bridge the gap between theory and the world as we see it, rather than explanations that would make sense to the ordinary reader.

Ball does not ignore the interpretations, though he clearly isn't happy with any of them. He seems to come clo…

Big Bang (Ladybird Expert) - Marcus Chown ****

As a starting point in assessing this book it's essential to know the cultural background of Ladybird books in the UK. These were a series of cheap, highly illustrated, very thin hardbacks for children, ranging from storybooks to educational non-fiction. They had become very old-fashioned, until new owners Penguin brought back the format with a series of ironic humorous books for adults, inspired by the idea created by the artist Miriam Elia. Now, the 'Ladybird Expert' series are taking on serious non-fiction topics for an adult audience.

Marcus Chown does a remarkable job at packing in information on the big bang, given only around 25 sides of small format paper to work with. He gives us the concepts, plenty about the cosmic microwave background, plus the likes of dark energy, dark matter, inflation and the multiverse. To be honest, the illustrations were largely pointless, apart from maintaining the format, and it might have been better to have had more text - but I felt …