By Julian Heming
Based on an article first published December 1994 in NWP Gazette, Vol.1, No.2, pp.2-8
The Met Office routinely produces verification statistics for NWP fields from its Unified Model in both of its two forecast configurations: global and mesoscale. Forecasts of tropical cyclones (TCs) will, of course, have an impact on verification statistics for the global configuration of the model, but these will give no indication of the quality of track forecasts for individual storms and any biases in the model's handling of these features which may exist.
In 1990, work commenced to develop a semi-automatic scheme to verify forecasts of TC tracks. Since then, the scheme has evolved to produce a variety of statistics. Facilities have also been developed to process these statistics and output tables and graphical products - some of which will be described in this article.
As a first step to verifying the forecast track of a TC, an observed position of the TC centre and a definition of the forecast position in the model are required.
Fig 1. Example of tropical cyclone advisory message
Observations of TC centre locations are received in advisory messages through the Global Telecommunications System (GTS) from places such as the Joint Typhoon Warning Centre at Hawaii and the National Hurricane Centre at Miami. An example of such a message can be found in Figure 1. These are usually issued every six hours for each active TC. At the end of a TC season some centres issue "best track" observations for each storm in their area of responsibility.
Some of these observations may differ from those issued in real time because of information available after the issue of the advisory (e.g. satellite images, aircraft reconnaissance). However, in order to assess the performance of the model soon after the event. the advisory messages are used as verifying observations in this scheme.
The initial state data in advisory messages (e.g. observed position, maximum sustained wind (MSW)) are used and processed automatically in real time to produce data to initialise TCs in the model. During this process, the data are stored ready for use as input for TC track verification after the TC has ended.
The centre of a TC in the model analysis or forecast field is defined as the point with the maximum value of 850hPa relative vorticity (RV). This parameter is used as opposed to low-level wind or mean sea-level pressure (MSLP) in accordance with WMO recommendations. In most cases the centre of a TC as defined by 850hPa RV will not differ greatly from that defined by MSLP. However, in some cases, particularly when the TC has a large degree of asymmetry, the two methods will produce considerably different locations.
Fig 2. Tropical cyclone Melissa, data time: 12Z, 15 September 1994, analysis of 850 hPa Relative Vorticity (x 10-6 sec-1) and Mean Sea-level Pressure (hPa).
Fig 3. : Tropical cyclone Gordon, data time: 00Z, 12 November 1994, T+48 forecast of 850 hPa Relative Vorticity (x 10-6 sec-1) and Mean Sea-level Pressure (hPa).
Figure 2 shows the MSLP (solid lines) and 850hPa RV (dashed lines) analysis for TC Melissa. The circulation is symmetric and the points of minimum MSLP and maximum RV coincide. In Figure 3 (Hurricane Gordon) the circulation of the storm is not symmetric and the RV centre is to the north-east of the MSLP centre.
The RV field is obtained by retrieving 850hPa u- and v-components of wind from an archive of model fields and calculating the RV at each grid point. A surface fitting technique is used to accurately define the TC centre in the model should it lie between model grid points.
Before describing the verification technique, it is necessary to mention what is not verified by this scheme. Firstly, TC genesis is not verified - i.e. the forecast development of a TC initiated from an analysis where no TC was present. In this scheme a forecast is only verified if there is an observation of a TC available within six hours of the analysis time.
The scheme also does not verify TC intensity. An indication of intensity is given on advisory messages by values of MSW or central pressure. Since the core structure of a TC is beyond the resolution of current global models, NWP fields will reflect neither the true wind or central pressure values near to the centre of the TC. Hence, verification of these values is meaningless. Most TCs are represented in global models as relatively weak lows although occasionally a strong circulation will be represented, e.g. TC Melissa (Figure 2). However, even in this case the analysed central pressure was some 65hPa higher than the estimated actual value.
Since January 1998, the global model resolution has increased to near 60 km at the equator. Hence, the possibility of useful intensity forecasts has increased and will be monitored to assess their usefulness.
There are three criteria which must be met for an analysis of a TC to be verified.
If these criteria are met the observed position of the storm is interpolated to the analysis time (if necessary) and a search conducted for the grid point with a maximum value of 850hPa RV within an area of radius 5° from the observed position. Once found, the position of the TC centre in the analysis is located more accurately by a surface fitting technique as described earlier. The distance between this point and the observed position is calculated and is known as the direct positional error (DPE) of the analysis.
If an analysis is not verified by the scheme due to a failure to meet one or more of the criteria above then the subsequent forecast is also not verified. However, if an analysis is verified, subsequent forecasts are verified subject to the following conditions.
This last condition is added in order to prevent attempts to verify forecasts where the model has in fact dissipated the TC. The critical value has been tuned to 0.7x10-4s-1 for the configuration of the model in use in 2000.
The procedure to search for the forecast TC centre in the model field varies with the forecast time. At T+24 a search is made for the grid point with a maximum value of 850hPa RV within an area of radius of 7° from the observed position valid at the forecast time. For subsequent forecasts (T+48, T+72, T+96 and T+120) the search radius remains at 7°, but the search area centre is determined by linear extrapolation of the model's forecast movement of the TC over the preceding 24 hours. Once the point of maximum RV is found in the model field, the DPE value for the forecast is calculated in the same way as for the analysis.
As with any automated system, there are limitations as to how accurately TCs can be tracked and verified. One of the major problems is the use of the search radius to define an area to search for the RV maximum. Initial trials with a radius of 10° showed that for dual TC circulations, which occur fairly frequently, the automatic system may switch to tracking the wrong centre. This was particularly noticeable when the secondary TC (not being verified) had a more-intense circulation and hence a higher value of RV at its centre.
Fig 4. : Tropical cyclones Pat and Ruth. Data time 12Z, 24 September 1994, analysis of 850 hPa Relative Vorticity (x 10-6 sec-1) and Mean Sea-level Pressure (hPa).
Figure 4 shows dual TCs Pat and Ruth whose centres were closer together than 10°. Hence, verification with a radius of this value may result in the tracking of the wrong centre. On the other hand, when a very small radius was chosen, it was possible under some circumstances that storms which were poorly forecast or forecast to move rapidly or suddenly change direction could lie outside the search area and hence not be correctly verified. For forecasts a value of 7° was found to be the optimum radius in order to minimise these two problems. A value of 5° is used for analysis verification.
Despite using these optimum search radii, these problems can still occur. Hence, when verification is carried out of TCs which potentially could be incorrectly verified (e.g. dual systems), charts of model fields are examined and if necessary the verification rerun with an adjusted search radius in order to eliminate any errors.
The DPE calculated for each model analysis and forecast will give an indication of how well a TC track was forecast, but gives no information as to whether the forecast errors were resulting from a slow or fast bias in the forecast track or maybe from a tendency to steer polewards too soon. In addition, the DPE value alone may not always indicate the skill of the model in forecasting a TC, since in some tropical areas TC tracks are more predictable than in others, and individual TCs which take a 'straight-running' track should be easier to forecast than those which take more complex tracks. In order to assess some of these forecast track characteristics a number of other statistics are produced by the verification scheme.
Forecast errors in the east-west and north-south directions can be easily determined from the forecast and observed positions of a TC and are known as the DX and DY errors. However, it is often more useful to know the components of error both along the observed track of the TC and perpendicular to the track.
Fig 5. Diagrammatic explanation of forecast errors.These errors are known as the along-track (AT) and cross-track (CT) errors. AT errors give an indication of whether a forecast of TC movement is too slow or fast and CT errors can be used to determine whether the model tends to recurve a TC too soon or fail to recurve it soon enough. A diagrammatic explanation of these errors can be found in Figure 5.
AT and CT errors need to be examined together since a negative AT value could indicate a slow bias in forecasts if CT values were not large, but may be associated with a large left- or right-of-track bias if CT values were large. Since AT and CT error calculation requires knowledge of the observed position 12 hours before, they cannot be calculated for the first valid forecast of a storm.
Several meteorological centres who monitor TCs have developed models which forecast the tracks of TCs up to three days ahead using methods based on past climatology in the area and persistence. These are known as CLIPER models and are generally accepted as a benchmark against which NWP models can be assessed. The Met Office has obtained CLIPER software for all TC basins which has been incorporated into the TC verification scheme. Hence, for each NWP model analysis and forecast which is verified, the equivalent CLIPER forecast is also verified. Values of CLIPER DPE are calculated and if the NWP model values are smaller the model is said to show skill over CLIPER. Skill is defined as a percentage value from the following formula:
(CLIPER DPE - Model DPE)/CLIPER DPE x 100%
Positive skill indicates the model forecast is better than CLIPER.
Negative skill indicates the CLIPER forecast is better than the model.
Most CLIPER software requires the knowledge of the position of a TC 12 and 24 hours before the analysis time. Hence CLIPER statistics cannot be calculated for the first two forecasts of a storm's life.
The verification system also includes the facility to produce mean statistics for all the parameters above. Mean statistics can be produced for individual storms, a selection of storms, storms grouped by basin or all storms in a year.
All verification statistics for TCs from 1988 onwards have been produced and an archive established to allow storage and retrieval of these data.
Facilities have been developed to produce graphical products, some of which are based on the statistical information which is described above.
Fig 6. Tropical Cyclone KORYN, 17-28 June 1993. 12Z observed positions (circle); 12Z analysed positions (triangle) with forecast positions to T+120 (cross).
One of the most useful products is a plot of observed and forecast tracks for an individual TC. This will often be useful to illustrate a bias in the model's forecasts. For example, Figure 6 shows the observed and successive forecast tracks (24 hours apart) up to T+120 for TC Koryn in June 1993. This shows a consistent bias towards steering the storm polewards too soon and confirms what is suggested by large positive CT errors and large negative AT errors.
Fig 7. TC Vance, 20 August to 4 September 1993 - 00Z observed positions (circle); 00Z analysed positions (circle); 00Z analysed positions (triangle) with forecast positions to T+120 (cross).
Figure 7 shows an example of a good set of forecasts for TC Yancy in August/September 1993 where forecast tracks correlate closely with the observed track. The plotting package is flexible and will allow just observed tracks to be plotted or observed and forecast tracks up to a specified forecast time. In addition, all forecasts from data times 00UTC or from 12UTC may be plotted, or a selection of forecasts, or just one forecast for an individual TC.
Fig 8. Tropical cyclone tracks in the western and central Pacific Oceans, September 1994. Observed tracks with maximum sustained wind bands. 31-60 kt (circle); 61-120 kt (circle with cross).In a slight variation on the above package, the observed tracks of several TCs can be plotted on a larger chart area. This will allow the distribution of TCs over a defined period of time to be displayed. Figure 8 shows the distribution of TCs in the western and central Pacific Ocean in September 1994.
Fig 9. Tropical Cyclone Melissa, data time: 12Z, 15 September 1994, analysis of 850 hPa wind field. Streamlines indicating Convergence and Divergence.
Fig 10. Tropical cyclone Melissa, data time: 12Z, 15 September 1994, analysis of U component of wind (metres/sec). South-north cross section along 159°E longitude. Shaded area - wind out of figure.
Facilities have also been developed to produce charts of MSLP and 850hPa RV as shown in Figures 2, 3 and 4, streamlines (Figure 9) and vertical cross-sections of wind through the centre of a TC (Figure 10).
All the TC verification facilities described so far have been drawn together into a package which is readily available to users of Met Office computer systems. This allows the user to browse past statistics, run the verification system on their own experiment and produce statistical and graphical products.
The Met Office has considerably increased its efforts in recent years in the areas of TC forecasting and verification. Many facilities have been developed to enable a detailed assessment to be made of the global model's performance in the forecasting of TCs. Further development of these facilities is being undertaken.
For more details on any aspect of TC verification at the Met Office described in this article please contact Julian Heming