AI is likely to play a significant part in the next generation of weather forecasts. With potentially transformative change in how weather forecasts are produced possible in the coming years, how will people react to the coming changes and continue to act with confidence around the weather?
Newly-published Met Office research highlights a gap between the perceived accuracy and confidence using existing physics-based forecasts and potential AI-generated ones: a ‘confidence gap’. AI-based models are not routinely part of the Met Office forecast yet, but are being actively developed in the weather community (including at the Met Office), so the public reaction is based on limited interaction with the technology.
For physics-based models, around 80% said they would be confident making weather-related decisions, both daily and for important events, while this dropped to around 40% for potential AI forecasts. It’s this confidence gap that will have to be addressed and fully understood in order to realise the full potential of new forecasting techniques.
In addition, around 78% of those surveyed said they thought physics-based forecasts are accurate, that dropped to 45% for forecasts that could be generated with AI in the future.
"The value and impact of improvements to forecasts using AI techniques is significantly undermined if people don't trust the output to make decisions," said Professor Kirstine Dale, the Met Office’s Chief AI Officer who is leading the strategy for realising the potential benefits of AI across the forecasting process.
"The Met Office exists to help people to make better decisions to stay safe and thrive, and if a shift to forecasts which incorporates AI techniques hinders that hard-won public trust, then that’s a significant consideration when planning innovative changes."
Public confidence in forecasts and their accuracy is crucial for people to take the decisions they need to stay safe in severe weather. This summer’s extreme heat has made this particularly evident. New June daily temperature records tumbled, with 38°C reached at Lingwood, while July was also its second warmest on record according to mean temperature. It has been an impactful summer so far, with drought also an additional ongoing concern.
With severe weather, taking action on the forecast is crucial. Fortunately, the Met Office regularly run post-event surveys to assess this element of forecasting. For the Red Extreme Heat warning in June this year, 96% of people in the area were aware of it, and 97% of those took action as a result. This trust has been earned over decades.
Rational scepticism of new techniques
The underlying data on the recent research paints a more nuanced picture behind the confidence gap that was identified.
While only around 45% thought AI-generated forecasts would be accurate, around a fifth of respondents admitted they didn’t know how accurate the forecast would be from the emerging technology. In stark contrast, the ‘don’t know’ figure for physics-based accuracy was less than 1%.
Dr Edward Pope is a Science Fellow at the Met Office and lead author on the paper. He said: “AI-generated forecasts aren’t routinely available to the public, so, in the absence of direct evidence, there seems to be a rational scepticism and uncertainty about potential AI forecasting capabilities, and how they’ll perform and aid decision-making.”
“Physics-based models have the advantage of decades of forecasting the weather and the public place a certain amount of confidence in this direct experience that they have. With no direct interaction with AI-based forecasts, building confidence and trust will likely take some time, and we are planning to re-run the survey in the future to explore how perceptions might change.”
How do you build trust in new forecasting techniques?
Adapting to new and innovative science isn’t new to the Met Office. Using electronic computers for weather calculations was first explored by the Met Office in the 1950s, with the first operational numerical weather prediction forecast run by the Met Office on 2 November 1965, over 60 years ago.
Obviously, that process has been honed and improved over the years, but the basis of numerical weather prediction and generating the forecast through a series of complex calculations was a novel idea when it was first introduced.
In addition, the Met Office's parallel suite approach to forecasting improvements – which means running upgrades alongside existing methods to enable stringent testing and verification before going live – is of paramount importance for delivering more accurate forecasts. This approach underlines the commitment to robustly test new approaches, making sure that any changes to forecast production can truly make a difference before going live.
Fortunately, Dr Pope has previously written a paper looking at how users’ beliefs about forecast accuracy could be updated by experience, and how this may link to the perceived usefulness of the forecast.
He said: “What’s reassuring in this area is that we have some understanding of how prior beliefs and direct experience can feed into a person’s perceptions of forecast accuracy, which are likely to influence attitudes and behaviours to new science.
“Rational scepticism is healthy, and uncertainty is expected when new technologies emerge. The onus is on the scientific community to build early and extensive trust to give people all the information they need to make up their own minds, but this can take time."
Met Office approach to addressing the confidence gap
Agility and innovation, as well as purposeful data and intelligence, are central pillars of the Met Office’s five-year corporate strategy.
In order to deliver on ambitious aims, scientific research continues at pace on the applications of AI and how it could work with physics-based models to provide increasingly accurate, but also trusted, forecasts.
Kirstine said: “Everything we do at the Met Office is backed by peer-reviewed science, and our approach to AI for weather and climate is no different. As the UK’s national meteorological service, our role will be to ensure that improvements from AI feed into forecasts in a reliable way, underpinned by world-leading science.
“By taking this thorough and transparent approach, we can work to ensure that existing confidence in the forecast and warnings is safeguarded and builds.”
It’s likely that the next generation of forecasts will combine physics-based and AI-based approaches; maximising their combined strengths to provide fast, accurate and reliable forecasts, underpinned by scientific research and understanding.
An exciting future
The findings won't change the way forecasts are produced today, but they will help guide future research, communication and engagement as AI techniques become increasingly important within meteorology. Understanding public concerns and expectations early allows trust to be built alongside technological development.
Addressing the confidence gap may take time and further cross-disciplinary research. The Met Office is one of the leaders in this field, collaborating with leaders in data science (and other disciplines) to develop weather and climate intelligence in a robust and science-led way.
So, what’s next for AI and weather? More engagement with forecast users, peer-reviewed research, proof-of-concept studies and careful planning will guide the next steps, ensuring benefits are realised when ready. Importantly, expert meteorologists will continue to work to finetune the details of forecasts, with people at the Met Office taking part in courses to upskill in disciplines beyond conventional meteorology and into data science.
Kirstine concluded: “AI has remarkable potential in the meteorological and climate field, but trust must be maintained.”