It's hard to provide simple guidance, but it helps to look at a guiding principle that shaped the development of the 2050 Calculator - keep it simple.
How complex should Calculators be?
When we first wrote down a list of working titles for articles for the Calculator community, we thought an article on ‘how complex should Calculators be?’ would be great. In principle, it’s an important topic to tackle, especially as countries are now looking to refine and enhance the functionalities of their existing Calculators.
We’ll be honest – it’s not easy to provide simple guidance, in part because the question is quite challenging to answer, especially in a concise way. To answer the question, it helps to look back at one of the guiding principles that shaped the development of the 2050 Calculator – to keep it simple. Let’s remind ourselves of the advice from the original UK 2050 Calculator team:
“The abiding principle when building a Calculator is the model should be ‘as simple as possible, but no simpler’ – you have to strike a balance between making sure it is easy to understand yet still analytically robust. If you can simplify the way in which you model a sector without compromising accuracy too much, do. The overall figures, and the major driving forces in supply and demand behind them, are what you need to get right. Modelling the intricacies of each sector can end up making it more difficult for non-experts to understand what is going on. Understandably, experts may not like this approach initially, but it has been key to the Calculator’s success.”
The key takeaway message from the guidance document is that the model should be “as simple as possible, but no simpler”. The outcome of such an approach is that the Calculator should be able to provide sufficient accuracy to help guide policy formulation, with a balanced investment of time.
“As simple as possible, but no simpler” is a useful catchphrase, but what specific guidance can we provide to help implement the concept? It’s sometimes helpful to turn the argument around and think of the lessons that have been learned when the development of the Calculator has encountered problems.
What makes a Calculator development project flawed?
In the UK, Tom Counsell, one of the members of the original UK 2050 Calculator team, provided thoughts on characteristics that led to flawed Calculator development. The main barriers he identified through the course of the project were:
1. Making the first or any version of the model too complex.
Pressure from:
- More serious modelling teams who say a model isn't credible unless it includes X, Y and Z;
- Stakeholders with entrenched views who say their minds won't be changed unless the model incorporates X, Y and Z;
- Inside the team, by people who really delve into the modelling.
Outcomes:
The project will be slower. It will be harder to generate insights from it because you can't see the forest for the trees. It will be harder to use it to start a debate; too complex to be trusted; too complex for a non-expert to explore.
2. Making the range of options it explores too 'realistic'.
Pressure from:
- 'More serious' modelling teams who are used to modelling with central estimates, possibly with the occasionally high and low range
- Experts, who say there is only one possible option;
- Government, industry, etc. who say the only realistic option is their plan;
- Inside the team, from people who fear being ridiculed for proposing what some people will say are ludicrous ranges.
Outcomes:
People under-estimate uncertainty and, in doing so, do not consider important options, or consider some things to be more impossible than they are. It leads to poor debates – if people cannot put forward the option they believe in, they will not believe the model, and will therefore both criticise the model and the people with the 'realistic' views.
3. Excluding sectors and options where there is no data.
Pressure from:
- People who consider no activity data, or no good quality activity data, a good reason to exclude an area from the model.
Outcomes:
In many areas that we want to model, there’s probably no data; worse, there’s no prospect of data within the timeframe of the project (e.g., for the land use, land use change and forestry (LULUCF) sector, or the potential cost of electric trucks). The temptation will then be to exclude the area from the model. It’s always possible to estimate a range for data, even if it might be impossible to put any sort of central estimate in. By putting the range in, and then completing the model, you can see whether the lack of data matters.
4. Too much modelling.
Pressure from:
- The modelling team wanting to make the model complex and realistic, which will take longer than expected.
Outcomes:
Not enough time spent on considering the impacts of the 'ah ha' moments and starting great debates from which policies may then emerge.
Reflecting on these barriers, do you see any of these issues occurring during the development of the Calculator in your country?
What can you do to help overcome these barriers?
The diagram below summarises the barriers, with some simple suggestions on how to mitigate.
Prioritising resources – what areas of the Calculator should you concentrate on?
In general, you should aim to:
- Have as few sectors as possible so that it’s quick to use the tool;
- But cover the whole economy and all emissions (it’s best not to ignore land use or industrial process emissions);
- Include all the sectors that people expect, even if they’re small, for example, if most people think that rooftop wind turbines are an important potential energy source, then it should be included even if this is factually incorrect.
As a rule of thumb, 40 sectors are about right. We recommended starting top down, rather than working bottom up, i.e., start by thinking about the transport sector, and the fewest sub-divisions possible, rather than thinking of all the possible types of vehicles and how they might be modelled.
The different sectors in the energy economy overlap and interact with each other to produce overall supply, demand and emission figures. It’s crucial to understand this interplay and to take it into account when designing the model but not to get overwhelmed by it. Everything should be covered once, but once only.
The mutually exclusive and collectively exhaustive (MECE) principle should become a mantra for the project, so that the same energy or emissions inputs do not appear multiple times in the model, making results inaccurate. For figures that are relevant to multiple sectors, exactly where they go in the model will be a matter of personal preference. In such cases, as long as everything is properly documented, there’s no right or wrong answer.
It may be helpful to consider linked technologies together, as this will minimise interaction between sectors; for example, we recommended including one sector for heating and insulation since they interact so much. It’s also important to consider that outputs from one sector could affect other sectors in the model. Hence the sectoral interlinkage as a whole is an important factor when scoping and drawing boundaries for the sectors.
Final thoughts – more complex models are not necessarily more accurate
Some of the in-country 2050 Calculator development teams might find themselves under great pressure to make their Calculator complex. It’s not necessarily true that more complex models produce more accurate answers. This statement can be an important argument to defend against excessive model uncertainty.
The figure below summarises the trade-offs between model accuracy and complexity. There’s an ideal spot, and there’s a balance between model framework error and the number of uncertain inputs.
A more detailed argument is that there is an optimal trade that needs to be found between model errors, introduced by making model assumptions (model framework error), and model uncertainties, resulting from uncertainties in the model input required for aligning the model to country-specific circumstances. On the one hand, a model cannot be too simple as it will not capture the relevant system characteristics; while on the other hand, the quality of a complex model is difficult to assess since the high number of degrees of freedom make it possible to produce almost any desired behaviour with often plausible parameter values.
Further support
The UK-funded 2050 Calculator programme not only supports the creation of country Calculators; it also provides ongoing support to existing Calculator teams, including for model complexity.
Further support
beis2050calculator@mottmac.com
Written by John Watterson, Ricardo
Article text (excluding photos or graphics) © Imperial College London.
Photos and graphics subject to third party copyright used with permission or © Imperial College London.
