The value of verification
12 August 2013
The Met Office has been refining and improving the accuracy of its forecasts for decades. However, assessing the level of accuracy isn't straightforward so we employ a complex system called 'verification' that compares our predictions with 'the truth' that actually occurs.
There's a debate in the scientific community as to what actually constitutes 'truth' in forecasting, but taking any uncertainties into account, the Met Office uses a wide range of observations to get the most accurate picture possible. These include data from synoptic reporting stations (SYNOPS) and airports (known as METARS) on land, weather buoys on the ocean, weather balloons and aircraft. Radar data has been used for some time to verify rainfall forecasts and satellite data are also becoming increasingly important - and there are many other sources besides.
Verification systems assimilate these observations and compare them with our forecasts, producing a range of verification scores that can be applied over areas or multiple sites and over defined time periods.
Monitoring and refining this process are two teams of scientists and scientific software engineers with a shared expertise that covers meteorology, data analysis, scientific programming and database skills. And as Philip Gill, Operational Verification Team Manager points out, "Communication skills are also really important - so we can clearly impart results to our customers, our scientists, and the wider scientific community."
Verification is invaluable. From a scientific perspective, it's an objective way to monitor the performance of forecasting models - and to test whether any changes made to those models actually lead to an overall improvement. It also enables us to assess our corporate performance through the Forecast Accuracy Business Performance Measures, as well as the Key Performance Indicators agreed with customers.
We also participate in the World Meteorological Organization's Commission for Basic Systems (WMO-CBS), which offers a standard protocol for exchanging verification data with other meteorological services around the world. And of course, verification allows the Met Office to demonstrate the accuracy of its forecasting products to customers in an objective way. Many of these products are routinely verified, so customers can see how well they are performing on an ongoing basis.
Making it matter
It's not enough to simply generate verification data - it has to be relevant as well. That's why the Met Office verification teams work on breaking down statistics in ways that are most useful to each of our customers.
For example, say a customer needs to know when the temperature will drop below freezing, so they know when to grit the roads. We can generate a simple report with four possible outcomes: a hit is when the temperature is forecast to go below freezing, and it does; a miss is where it is forecast not to drop below freezing and it does; a false alarm is where it is forecast to go below freezing and it doesn't; and a correct rejection is where it is forecast not to drop below freezing and it doesn't. From counting numbers of each of these outcomes, we can provide summary figures such as percent correct and false alarm rate.
We are also able to express verification in economic terms as a Relative Economic Value score. If a customer knows what their relative costs and losses are when they make decisions based on forecasts, this score could enable them to calculate the possible cost saving a particular product could give them.
So how accurate are the Met Office's forecasts? As Philip Gill explains, "It depends on the parameter, location and time period you're looking at, but in general, we're always improving." At the moment for example, our day ahead forecast for maximum temperature for the UK is accurate within two degrees over 90% of the time, while our day ahead wind speed forecast is accurate within five knots over 90% of the time.
"Our four day forecasts today are as accurate as our one-day forecasts in 1980."
But perhaps the clearest indicator is the Met Office Core Capability Component which measures the accuracy of the mean sea level pressure forecast. This may sound less familiar, but it's actually one of the fundamental parameters used in weather forecasting. "Looking at this particular measure," says Gill, "our four-day forecast today is as accurate as our one-day forecasts were in 1980."
This overall trend towards ever-greater accuracy is down to ongoing development of the Met Office's models, the ability to assimilate more new observations, and also improvements in the Met Office's supercomputer technology.
Meeting perception with reality
It's all very well having objective data about forecast accuracy - but what is the customers' view? Here's where the Met Office Customer Attitude Survey is so important. In the survey, customers can express what they perceive the accuracy to be - and this feedback is taken very seriously. "A lot of the comments we get back are very positive," says Gill, "but if a customer's perception doesn't match up with our verification statistics, we're able to go back to them and address any concerns." As such, verification is a vital tool to support and really strengthen the relationships the Met Office has with its customers.
- Learn more about our forecast accuracy