A significant milestone has taken place as the Met Office’s operational weather and climate intelligence has transferred to a new supercomputer operated by Microsoft.
As detailed on Microsoft’s blog, this represents the first phase of the new supercomputer and is the next step on the Met Office’s technological innovation journey.
Read the blog from Microsoft in full below.
Even though the Met Office, the UK’s national weather service, had been preparing to replace its latest supercomputer for several years, it still felt like a momentous occasion.
A group of about two dozen computer operators, administrators and a couple of photographers gathered in a vast hall.
With a tap on a keyboard, the processing of over 50 billion weather observations – gathered daily for forecasting – came to an abrupt halt, accompanied by scattered applause.
Meanwhile, a new supercomputer in the Azure cloud that had been running in parallel offsite for more than a month officially took over.
With the transition smoothly completed, the Met Office is now able to take advantage of a new, cloud-based system that aims to both help people with better forecasts and help scientists advance important climate research around the world.
Met Office forecasts are used in aviation, defence, critical infrastructure, and shipping, as well as for predicting storms, floods, and gauging if Sunday will be suitable for a barbecue. The decision to entrust that critical work to an outside institution like Microsoft came only after a long process, informed by its own expertise in technology.
Founded in 1854, the Met Office has a long history of scientific trailblazing. It operated its first supercomputer 60 years ago.
“I think this is our 14th generation of supercomputer,” said Penny Endersby, the Chief Executive Officer of the Met Office, of the new supercomputer in Azure. The Met Office has almost always run its own supercomputers on site before now.
“What’s different about this one, and particularly the whole journey with Microsoft, is that we have decided this is now a job for specialists, and the core business of the Met Office is helping you make better decisions, to stay safe and to thrive through providing fantastic forecasts.”
Supercomputing for ‘science as a service’
Charles Ewen is the Met Office’s Chief Information Officer, charged with keeping the organisation’s technology at the cutting edge, and he has overseen the project to shift computing to the cloud.
“We use a technique to predict the future state of the atmosphere called numerical weather prediction,” he said. “So that takes the laws of physics that are fairly well understood and applies them at scale to observations of the current state of the atmosphere. To do that is very, very computationally expensive. It’s simulating the future state of the atmosphere. Operationally, that’s 200 to 300 terabytes of information a day.” 
Met Office Chief Information Officer Charles Ewen. Image courtesy of Chris Welsch and Microsoft.
The new supercomputer is giving the Met Office a variety of advantages, Ewen said. One of the most visible ones will be better forecasts over longer time periods.
“People ask how a bigger computer improves the weather forecast,” he said. “One big thing this new computer will allow us to do in the near future is to be able to produce 14-day forecasts with a similar kind of accuracy than we can today for seven, eight, nine days.”
Beyond that, he said the meteorologists and researchers at the Met Office are always hungry for more computing power – to run more complex weather models, to gather and analyse ever more data, he said.
He described what Microsoft is providing as “supercomputing for science as a service.”
“Essentially there’s more science than we have the compute power to deliver, so it’s not a question of how much compute do you need, it’s how much you can get.”
Charles said that one of the advantages of working in Azure is that expanding capacities for specific research projects can be done on a case-by-case basis – meaning the Met Office can take on new projects without having to build new infrastructure.
And with the revolution in artificial intelligence, the Met Office has a partner that can empower the agency to adapt its approach step-by-step.
“We don’t know yet how our CPU-based supercomputer services will be augmented with machine learning,” he said. “A lot of research is being done at the Met Office and elsewhere to find out.”
With Microsoft, he said, “We’ve got the flexibility to make those choices and make sure we’re well equipped with the right tools for the right jobs at the right times.”
Charles said that the Met Office is investing in the future of AI and machine learning with training programs and the funding of advanced degrees for some of its researchers.
“We’re already in the game, preparing our people,” he said. “We’ve put more than 100 people through a foundational machine-learning program, plus we have put around 20 people through a formal master’s program. These people are already experts in another field, like atmospheric physics.”
Giving researchers speed and capacity
Simon Vosper is the Science Director of the Met Office. In addition to its role in forecasting the weather, it is a globally leading centre for climate research; the new supercomputer in Azure gives the Met Office more power and more tools, said Simon.
“We have huge increases in capacity, which will enable us to take our science forward,” he said. “With the new Azure-based capability, we’ll be enhancing resolution, running many more models and introducing new scientific complexity.”
Met Office Director of Science Simon Vosper. Image courtesy of Chris Welsch and Microsoft.
Ségolène Berthou leads a team of researchers working on the agency’s environmental prediction capabilities.
Her specialty is the interaction between the ocean and the atmosphere and how it affects the climate and weather.
“The coupled system we’re preparing is running faster and more smoothly on the new supercomputer,” she said. “This is very good news because it means we can now be even more confident in our climate projections and have longer slices of time running this model.”
Met Office Scientific Manager Ségolène Berthou Image courtesy of Chris Welsch and Microsoft.
Ségolène said that the new capabilities mean that the Met Office can use its predictive models to forecast extreme weather events with more accuracy. One such approach is “ensembles,” which means researchers run a model several times with slightly different conditions or parameters.
“That will give us quite a large variety of forecasts because of the chaotic nature of the atmosphere, and that can help us capture the extremes with more days in advance,” she said.
Opening new horizons
Penny Endersby, the Met Office CEO, said the successful move onto the new platform for supercomputing is just the start of many new possibilities.
She explained, for example, that the Met Office’s troves of previous research and weather data have been moved into the Azure cloud - making it accessible in ways it never has been before.
Met Office Chief Executive Officer Penny Endersby. Image courtesy of Met Office.
“One of the reasons Microsoft is such an exciting partner for us is that we feel that there is much more value that can be extracted from our data by other people as well as by us,” she said.
That includes using the data for research and for training large language models.
Endersby said there isn’t much time to pause to reflect on the transition moment because there’ s always more work to do with the weather.
“A big change like this is like changing the engines when you’re flying over the Atlantic,” she said, “because you can never have a stop, which is why we did the long parallel run with the old supercomputer.”
“So, success at this phase is that nobody can tell the difference,” Penny said. “But that isn’t obviously where we want to stop because success with the program is delivering better, more valuable forecasts to the industries, governments, and citizens that we serve.”