To answer questions about how good a forecast is, the
forecast is compared to what actually happened. With weather forecasts
this is not a simple process. It is not always a case of being right
or wrong; it is useful to know how close the forecast was to the actual
weather. Therefore we use a method of verification that measures how
close the forecast is to the observed weather.
The Monthly Outlook includes forecasts of expected temperature and
rainfall categories. Five categories are used; (1) well below average,
(2) below average, (3) near average, (4) above average or (5) well above
average conditions for the time of year.
To assess the accuracy of the forecast we compare the predicted category
with the category that was actually observed to occur. We use a points-based
scoring system in which maximum points are awarded to forecasts that
are 'spot on' (i.e. the forecast category exactly matches the category
that actually occurred), fewer points are awarded for 'near misses'
(e.g. the forecast is wrong by one category), and points are subtracted
for misleading forecasts (i.e. a forecast of above normal when below
normal is observed). The score used is called the Gerrity Skill Score
(GSS), and is one of the scores recommended by the World Meteorological
Organization (WMO) for evaluation of long-range forecasts. The score
is designed so that forecasts that are always 'spot-on' would achieve
a score of 1.0, and forecasts based on simply 'forecasting' the long-term
average (category 3) would receive a score of zero. Thus a positive
score means the forecast is better than guesswork and better than assuming
future conditions will be similar to the long-term average. Although
the theoretical maximum score is 1.0, best scores achieved at the monthly
range are of order 0.6, and found in the more predictable tropical regions.
Long-term assessment
It is important to assess the performance of long-range prediction
systems over a large number of forecasts, since good (or bad) performance
over one or two forecasts may not reflect the long-term performance.
The bar chart shows Gerrity Skill Scores calculated over 115 forecasts
issued for each of the 10 UK regions between June 2002 and September
2005. The scores shown are for mean temperature (Tmean) and precipitation
for the three periods used in the Monthly Outlook: days 5-11 ahead,
days 12-18 ahead and days 19-32 ahead.
Best skill is found for the temperature forecasts and, as expected,
for the 5-11 day period. At longer ranges, scores for Tmean in 12-18
day period show best skill, and an example of a successful forecast
at this range is given below. At the 19-32 day range scores are positive
but indicate at best only marginal benefit over use of climatology.
Gerrity Skill Scores for mean temperature and
rainfall
Case study - late-winter 2004/5 cold snap
The late-winter 2004/5 cold snap over the UK in 2005 was anticipated
by the Monthly Outlook nearly two weeks in advance. The Monthly Outlook
issued on 11 February stated for the 12-18 day period (21-27 February)
that '...a sudden change to below or well-below average temperatures
is expected...'. The left-hand figure shows the predicted category for
maximum temperature for the 10 UK districts, the right-hand figure shows
the category that later occurred. The correct category was predicted
in the southern and central districts. In northern districts cold conditions
were predicted, but underestimated by one category.
 |
Fig. 1: Gerrity Skill Scores for mean temperature and rainfall |
More about the Monthly
Outlook features
