Model mistakes

Mathematical modelling is an indispensable analytical tool, but not always accurate. Erica Thompson's book takes a nuanced look at the challenges of matching its predictions against reality

Escape from Model Land: How mathematical models can lead us astray and what we can do about it

Author: Erica Thompson

Publisher: Hachette

Pages: 247

Price: Rs 899

This highly original and thought-provoking book is a philosophical treatise on the art and practice of mathematical modelling. It is funny, light-hearted, and engaging even as it examines very deep foundational questions. The “escape” in the title is really about moving out of the rigid artificial constraints of a model back into the real world, to match the model’s predictions against reality.

Erica Thompson is a physicist who moved into data science with research interests in climate change, the spread of Covid-19, humanitarian crises and so on.  She’s a senior fellow at the LSE’s Data Science Institute.

Her interest in models and their escapes grew out of her PhD thesis. She was reviewing peer-reviewed scientific literature on North Atlantic storms. She claims she didn’t learn much about storms because so much of the research was directly contradictory. But she did learn a lot about the ways we use models. Ever since, she’s looked for a mean between taking models literally and risking being dangerously wrong, and chucking away the valuable data and structure models impose.

Our world is full of fat pipes with ginormous data on natural phenomena, and the broad spectrum of human behaviour. We can’t make sense of the world without making sense of the data. This is true across multiple disciplines ranging from sports forecasting, to financial trading, to understanding climate change, making pandemic mitigation policy, budget allocations and what have you.

We make sense of data by imposing structure on it, and removing what we think is  irrelevant. We incorporate certain assumptions when modelling. The assumptions could be the value of Pi, or the freezing point of water, the value of gravity, an interest rate, or the local time. 

The assumptions may even be wrong and known to be wrong, but useful in a specific context. For example, Dr Thompson says you would model the Earth as a flat object if you’re trying to accurately predict the trajectory of a cricket ball a la Hawkeye. You would assume it was a perfect sphere when making a large-scale map, ignoring the bulges at the Equator. However, you have to take even tiny local ups and downs into account, if you’re modelling geo-location data, accurate to within metres.

As our tools have improved, so has our ability to make granular complex models incorporating more variables and crunching more data. Access to more computing power has also led us to develop far more sophisticated modelling techniques and to black box AI where even the programmers don’t know how the algorithm is handling data.

An early chapter quotes Galbraith’s famous statement, “The only function of economic forecasting is to make astrology look respectable.” Dr Thompson examines the casting of horoscopes in some detail. No, she’s not a fan of astrology. But she does point out that even this pseudoscience helped us get into the habit of structuring data, and maintaining internal logical consistency in the artificial world of the horoscope and creating an environment for policy debate.

Astrology fails because it cannot “escape”. Astrological models cannot match against reality to yield practical predictive value. However, a modern mathematical model is no better than a horoscope unless it can escape.

The challenge is working out how much of what we learn from a model remains valid in real life. If we do compare the predictions against new, real world data, and deviations occur, we still have problems diagnosing why: Were the models’ assumptions wrong, or did we fail to make enough assumptions, or do we need more data?

Often too, there simply isn’t enough data to validate a model, and the data is not going to be generated at sufficient speed. Think of the Covid-19 pandemic when policymakers were screaming at epidemiologists for quick answers and policy direction, but nobody knew much about infection rates, hospitalisation rates, and other relevancies.

This problem also exists in very long-term policy planning, of course. Climate policy is based on models that predict changes over the next 50 years and beyond. If those models are wrong, we won’t know about it until decades down the line when the current policymakers are long dead, and it may be too late to change direction.

The astrologer used to be the domain “expert” on heavenly data. In the absence of sufficient new data to test predictions, Dr Thompson points out that appeal to authorities remains the other way of attempting escape. Our models may be far more complex and rigorous but we need domain experts to assure us that a model makes sense if we need to implement policy before relevant data is available. Unfortunately, domain experts can also be catastrophically wrong, especially if they fall in love with beautiful models — the subprime crisis occurred because of this.

As a working data scientist in contentious areas, Dr Thompson offers a nuanced look at an indispensable analytical tool. She ranges far and wide in her choice of examples as she says “it’s up to us to learn from models without being drawn in by their seductive elegance”, and thereby losing touch with “our messy magnificent world”.