Data assimilation is at the heart of the Met Office's weather and climate predictions
Data assimilation is a sophisticated technique that combines millions of real-world observations with the latest model forecasts to produce the most accurate possible representation of environmental systems such as the atmosphere. Data assimilation for the atmosphere is one of the crucial components of numerical weather prediction.
This process is fundamental to keeping our forecasts on track and is a key component of our Next Generation Modelling Systems (NGMS) strategic action, especially as we prepare for the capabilities of our new supercomputer.
The cyclical nature of data assimilation
Numerical weather prediction is not a one-off event but a continuous cycle. At regular intervals, every six hours for our global model and every hour for our high-resolution UK model, the process of data assimilation is repeated.
Each cycle begins with a previous forecast of the state of the atmosphere, known as the ‘background’, and incorporates millions of new observations. The goal is to correct the background to produce the best possible ‘initial conditions’ for the next forecast run.
The key ingredients in each assimilation cycle are:
- The previous model forecast (the background)
- Real-world observations from a vast array of sources
- The uncertainties associated with both the background and the observations
Understanding and managing these uncertainties is crucial, as they determine how much weight we give to each ingredient in the final "analysis", the word we use to describe the corrected state of the atmosphere.
READ MORE: Numerical Weather Prediction at 60: A journey of innovation at the Met Office
The forecast model and the background
The forecast model is a complex system of equations solved on a supercomputer, designed to predict how the atmosphere will evolve. The most recent forecast provides the background, our best estimate of the current state of the atmosphere. However, no model is perfect. There are always uncertainties, both in the way the model represents physical processes and in the initial conditions provided to the model.
Background uncertainty is particularly important in data assimilation. It reflects how much we trust our first estimate and guides how we use observations to improve it. Estimating this uncertainty is a scientific challenge, but it is essential for producing reliable forecasts.
Understanding background uncertainty
Uncertainty in the background arises from two main sources: imperfections in the model itself and inaccuracies in the initial conditions. Imagine planning a journey from point A to point B; your route will only be accurate if you start from the correct location. Just as we can't know exactly what the true state we are aiming towards is, we can't know how far from it our first guess is either. But there are reliable ways to estimate this.
Scientists use various methods to estimate background uncertainty. One approach is to analyse the uncertainties in many past forecasts, providing a long-term perspective. Another is to use an ‘ensemble’ of forecasts, each starting from slightly different conditions, to capture the range of possible outcomes for the current weather situation. Combining these approaches helps us understand and quantify the uncertainties of the day, whether it’s a calm, sunny afternoon or a turbulent thunderstorm.
The role of observations
Every global data assimilation cycle incorporates tens of millions of observations from satellites, radiosondes (weather balloons), ships, aircraft, and traditional weather stations. These diverse platforms provide comprehensive coverage of the atmosphere, measuring variables such as temperature, pressure, and wind speed.
READ MORE: What do meteorologists do at the Met Office?
However, not all observed quantities are directly comparable to those simulated by the model. For example, while a weather station might measure wind speed and direction, the model represents wind as separate eastward and northward components. To bridge this gap, we use an ‘observation operator’, a set of equations that converts model quantities into observed equivalents, ensuring that comparisons are meaningful and consistent in both space and time.
Observation uncertainty
Just as with the model and background, observations also have uncertainty. There are uncertainties associated with the instruments themselves and with how the observations are represented in the model. This ‘representation uncertainty’ arises because the model’s view of the atmosphere is necessarily simplified and “pixelated”, while the real world is continuous and complex.
Observation uncertainties are typically estimated using statistical methods. Ongoing research at the Met Office aims to improve these estimates and to make better use of observations whose uncertainties are interdependent. By refining our understanding of observation uncertainty, we can assimilate more data and further improve our forecasts.
Balancing the scales: how data assimilation works
Data assimilation can be visualised as a set of balance scales, with the background on one side and the observations on the other. The uncertainties associated with each determine how much weight they carry. The analysis, the starting point for the next forecast, is the carefully balanced combination of the two.
READ MORE: The UK land observation network: Underpinning weather and climate understanding
Sometimes, a highly valueable observation may outweigh the background entirely for a particular variable at a specific location and time. More often, the analysis represents a blend, using all available information to achieve the best possible estimate of the true state of the atmosphere.
The power of uncertainty specification
One of the strengths of the data assimilation process is its ability to infer changes in unobserved variables based on the relationships encoded in the background uncertainty. For example, if an observation indicates a drop in air pressure at a location, the data assimilation system can adjust not only the pressure but also related variables such as wind speed, based on fundamental meteorological principles.
This interconnectedness allows us to make the most of sparse observations, filling in gaps and ensuring that our forecasts are as accurate and comprehensive as possible.
Innovation and the next generation of data assimilation
The Met Office is continually working to improve the ingredients that go into data assimilation. The Next Generation Modelling Systems (NGMS) programme is at the forefront of this effort, developing improved backgrounds from our new atmospheric model, LFRic, and implementing advanced observation processing applications such as the recently successful JEDI-based Observation Processing Application.
Future developments will leverage the power of high-performance computing, especially as we continue to exploit the capabilities of our new supercomputer, and introduce more flexible systems for estimating background and observation uncertainties. These innovations will enable us to assimilate more complex data and deliver even more accurate forecasts.
Data assimilation is a complex but essential process that underpins the Met Office’s ability to provide accurate and timely weather forecasts and predictions for other parts of the Earth system, such as the ocean. By skilfully combining model forecasts and real-world observations, and by continually refining our understanding of uncertainty, we ensure that our forecasts start from the best possible position.
As we look to the future, ongoing research and technological advances will further enhance our data assimilation systems, helping us to meet the challenges of an ever-changing atmosphere and to continue delivering world-class weather prediction.
This blog post was adapted from internal communications produced by scientists working in data assimilation at the Met Office
Keep up to date with weather warnings, and you can find the latest forecast on our website, on YouTube, by following us on X and Facebook, as well as on our mobile app which is available for iPhone from the App store and for Android from the Google Play store.
