Modelling Matters: Are Models the End of the Story?

by Tim Gent

In the last article, we talked about the awkward questions that modellers need to ask their clients – and often themselves – in order to ensure their modelling expedition will be successful.

But what do we mean by ‘successful’ modelling? Is producing some results as promised sufficient, or is it more than that? (Spoiler alert: of course there’s more to it). Let’s take a look at what issues can – and probably will – arise and how they can be dealt with.

What does the client say?

All modellers will have a client somewhere, usually funding the work, or at least a major stakeholder in the model and its outputs. There is no getting away from the fact that how the client responds to the model outputs matters. Are they happy? Are they better informed? Are they confident in the outputs? Will they use the results at all?

Here are a few things clients often say when they see model results:

“It’s the right answer! Job done, thanks. Bye!” – This does happen! Many clients are naturally focused on action and less interested in scrutinising results, particularly if it’s the answer they expected or hoped for. Why waste time digging when you have the answer you wanted? As a modeller, you may be left with the uncomfortable feeling that there is more to discuss. Perhaps the real answer is more complicated?

“It’s the wrong answer! Your model is wrong!” – The model may show the proposed action is infeasible, too costly, or simply does not produce the hoped-for benefits. It’s natural that the client will want a more palatable answer, and they could hold the model – and the modeller! – responsible until they find one. Every modeller will come up against this “blame the model” response, probably frequently.

“Why did you assume that!?” – All models and model tests rely on input data and assumptions. If the results are unpalatable, these are often identified as the weak spot, and the client is likely to have real-world experience with which to challenge assumptions. Modellers should anticipate this, explaining input assumptions clearly and ensuring buy-in before model runs, not after.

“Your model is too complex” / “Your model is too simplistic” – It often seems a modeller cannot win - I have heard both these phrases from the same client in the same meeting! Clients may have a simpler – and not necessarily incorrect! – understanding of the problem. Conversely once they have understood the model, they may identify ‘vital’ aspects overlooked and see simplicity as a weakness.

What should the modeller say?

My previous article discussed how modellers need to ask awkward questions to lay down the groundwork and manage client expectations. Ideally, clients should appreciate at the outset that the model may give inconvenient answers; that model results often need time and collaboration to understand, clarify and refine; and they should share ownership of key assumptions and simplifications.

However, it’s not always possible to achieve or sustain such a perfect working relationship. Pressures of time at the outset, changes in thinking, priorities and personnel will get in the way.

So, it’s always best if the modeller is able to plan and control the way results are released and discussed. In an ideal world, model outputs should never be allowed out into the world unaccompanied. I always try to introduce them gently to the client over a presentation and discussion, preferably in a draft form with time and scope for review and experimentation.

Ironically, the point the client completely ‘understands’ often leads to doubts about the modelling approach. The impression of over-complexity turns to over-simplicity, as awareness rushes in of the many factors left out of the model. I have learned to take this as a good thing: the basics are understood, and the discussion will now be more interesting and productive!

Did we all learn something?

This is my own favourite measure of modelling success, that I always hope my clients buy into. The outcome of modelling is seldom just a number, the things we learned along the way are usually more important. Maybe the problem is not what we thought it was? Possibly the data was not suitable, factors we chose to ignore are key? If the approach we have chosen won’t work, but we know enough to pick a better way.

I think this also provides the perfect answer to the question “Why do we model?”. We model not simply to find an answer, but because we don’t know how to find the answer - yet. In which case, the learning along the way is a critical part of the process. Modelling is an expedition into the unknown, so whenever we learn more, we are winning.

The modeller’s ultimate accolade?

So how does the modelling expedition end? Sometimes, modelling is a one-off exploration with a single answer. If the work has been successful, you may hear the ultimate accolade from the client: “It all seems so obvious now. Why did we even need a model?” The modeller should have no fear of this: a happy client will always come back for more help when they find new problems to model!

Alternatively, a successful model may be used for years to test, optimise, interpret and predict as new data, scenarios and options emerge. This is the production phase of modelling, where the model must be maintained, automated, refreshed and often handed over to new users.

In the next article we will explore the new challenges that this brings and how the modeller can rise to them.

Tim Gent is an independent consultant, offering advice and support on the development, use and maintenance of all kinds of models.