|Met Office Hadley Centre observations datasets|
|> Home > HadIOD >|
None at this time
At present feedback format data are ony available to Met Office users. These webpages provide basic information for those interested in this dataset. Please contact us if you are interested in using these data.
NEMO NetCDF feedback format (hereafter ‘feedback format’) is a format used for model data assimilation (see NEMO-OCE pdf). A tool to extract data from the HadIOD database into feedback format was created as part of the ERA-CLIM2 project. Feedback format data from HadIOD.220.127.116.11 and HadIOD.18.104.22.168 were made available to ERA-CLIM2 participants and as a pre-release to Met Office colleagues. The HadIOD feedback format files have been through a few minor iterations following user feedback, each having its own version number (see 'Dataset versions' below).
Data are provided in daily files, the current version is v1.2 (see below). The v1.2 files presently cover 1978-present but could be extracted to cover other periods (HadIOD.22.214.171.124 covers 1850-present). For each day, two files are provided, one containing ocean profile temperature (in situ and potential temperature) and salinity data and the other containing sea surface temperature data. Each file contains the observed data along with metadata, bias corrections, measurement uncertainties and quality control decisions for each observation. The files contain several different bias correction and uncertainty variables; it is not possible to specify values for all of these variables for every observation and unspecified values are set to missing data. At a minimum we try to provide a quality flag and some estimate of uncertainty in each measurement (in some cases providing an uncertainty may not be possible). File contents and code tables used by certain variables are described in 'File contents' below. Sample file headers can be seen here:
Observed data are provided in 2D arrays (with stations on one axis and vertical levels on the other) and their station metadata are provided in 1D arrays. For profile data the vertical level dimension is 400 (i.e. a maximum of 400 vertical levels per profile) and for surface data the vertical level dimension is set to 1 (as these sample a single vertical level).
At present there is no user guide for the feedback format files. Basic information is provided on these webpages. We also strongly recommend reading the product user guide available for the HadIOD files which is of relevance here, in particular Section 2, which gives useful general information about HadIOD data and, importantly, describes the HadIOD observation error model and how to work with the bias corrections and uncertainties provided by HadIOD. Because the names used for the bias correction and uncertainty variables in the feedback format files differ from the variable names used in the product user guide, the 'File Contents' section below includes a table showing which variable names are equivalent. The 'File Contents' section also has some further remarks comparing the two file types.
The HadIOD dataset includes supplementary files that provide additional sets of bias corrections to help users explore the bias correction uncertainty. The component of uncertainty explored by these corrections is increasingly important on longer space and time scales (e.g. if concerned with decadal trends). The use of these additional corrections is recommended and is described in the product user guide. The supplementary files could be used with the feedback format files and are linkable via the DATABASE_IDENTIFIER variable.
Please contact us if you need help.
The feedback format file versions listed below are for data extracted from HadIOD.126.96.36.199. During development some preliminary feedback format data were extracted from HadIOD.188.8.131.52 with processing similar (but not identical) to v1.0 below.
Observations from several different sources have been ingested into the HadIOD.184.108.40.206 database. The following sources are used in the feedback format files (CMEMS was included in the sst files from v1.1 onwards):
ICOADS.2.5.1 - [Woodruff et al., 2011]. The primary source of surface ocean measurements in HadIOD. These data are included from 1850-present and provide sea surface temperature measurements (not salinity) to HadIOD.
Acknowledgement: Research Data Archive/Computational and Information Systems Laboratory/National Center for Atmospheric Research/University Corporation for Atmospheric Research, et al. 1984, updated monthly. International Comprehensive Ocean-Atmosphere Data Set (ICOADS) Release 2.5, Individual Observations. Research Data Archive at the National Center for Atmospheric Research, Computational and Information Systems Laboratory. https://doi.org/10.5065/D6H70CSV. Accessed 2011 onwards.
ICOADS.3.0.2 - https://icoads.noaa.gov/. Used in HadIOD as a substitute for ICOADS.2.5.1 from April 2022 onwards (the last monthly update of ICOADS.2.5.1 was expected in March 2022 [pers. comms. ICOADS team]). ICOADS.3.0.2 (2015-present) extends the coverage of ICOADS.3.0 (1662-2014) [Freeman et al., 2017]. The ICOADS.3.0.2 near real time observations are based on blended marine observations in the TAC and BUFR formats from NOAA's National Centers for Environmental Information (NCEI) GTS collections. The ICOADS.2.5.1 near real time observations are based on GTS observations from NOAA’s National Centers for Environmental Prediction (NCEP). Users should be vigilant for any impacts caused by the transition from ICOADS.2.5.1 to ICOADS.3.0.2 e.g. we note an increase in the number of ship observations across the transition.
Acknowledgement: Research Data Archive/Computational and Information Systems Laboratory/National Center for Atmospheric Research/University Corporation for Atmospheric Research, Physical Sciences Laboratory/Earth System Research Laboratory/OAR/NOAA/U.S. Department of Commerce, Cooperative Institute for Research in Environmental Sciences/University of Colorado, National Oceanography Centre/University of Southampton, Met Office/Ministry of Defence/United Kingdom, Deutscher Wetterdienst (German Meteorological Service)/Germany, Department of Atmospheric Science/University of Washington, Center for Ocean-Atmospheric Prediction Studies/Florida State University, and National Centers for Environmental Information/NESDIS/NOAA/U.S. Department of Commerce. 2016, updated monthly. International Comprehensive Ocean-Atmosphere Data Set (ICOADS) Release 3, Individual Observations. Research Data Archive at the National Center for Atmospheric Research, Computational and Information Systems Laboratory. https://doi.org/10.5065/D6ZS2TR3. Accessed 2022 onwards.
EN.4.2.0 / EN.4.2.1 / EN.4.2.2 - [Good et al., 2013]. EN4 is the primary source of ocean profile data (and profile quality flags) in HadIOD. These data are included from 1900-present and provide temperature and salinity measurements on depth (also potential temperature) to HadIOD. From November 2017 the switch was made to using EN4.2.1 as EN4.2.0 was retired. From the perspective of HadIOD this switchover should be seamless with no discernible impact on the data. From March 2021 the switch was made to using EN.4.2.2 ahead of EN.4.2.1 being retired. This switch is also expected to be relatively seamless though users should be aware there may be a small impact on the corrections provided for XBT observations.
Acknowledgement: EN.4.2.[0,1,2] data were obtained from https://www.metoffice.gov.uk/hadobs/en4/ and are © Crown Copyright, Met Office, 2021, provided under a Non-Commercial Government Licence http://www.nationalarchives.gov.uk/doc/non-commercial-government-licence/version/2/.
EN4 acknowledges the datasets contributing to its own construction (for full details see the EN4 webpages) and which HadIOD benefits from. These are:
E.U. Copernicus Marine Environmental Monitoring Service (CMEMS) drifting buoys - product INSITU_GLO_NRT_OBSERVATIONS_013_030 (Global Ocean In Situ Near Real Time Observations), http://marine.copernicus.eu/. Daily updates of global NRT observations from the main global networks from 2010-present provided by Coriolis (France). CMEMS drifting buoy sea surface temperature data are included in the HadIOD database from February 2015 to present. In the feedback files CMEMS is used as a replacement for ICOADS drifting buoy data from June 2016 onwards following a drop in the number of drifting buoy observations in ICOADS.2.5.1 (this issue has been resolved in ICOADS.3.0.2 but we continue to use CMEMS as the source of drifing buoy observations). CMEMS drifting buoy data are also included in the feedback files from 22nd-23rd October 2014 to fill a gap where there are no ICOADS.2.5.1 data.
Acknowledgement: HadIOD is generated using E.U. Copernicus Marine Service Information.
Met Office MetDB - in November 2016, surface buoy data (drifting and moored) were absent from the ICOADS.2.5.1 Real Time updates. GTS buoy data from the Met Office Meteorological Database system were used in HadIOD as a substitute. CMEMS supersedes MetDB drifting buoys once available (sst files from v1.1 onwards).
HadIOD is subject to Crown copyright protection and is provided under a Non-Commercial Government Licence. Please do read this. Any onward licencing of the information must be for Non-Commercial purposes only.
When publishing work using the data, please quote the HadIOD version number (found in the files' 'source' attribute, see 'File contents' below), the feedback file version number (found in the files' 'file_version' attribute, see 'File contents' below), and cite the following:
Atkinson, C. P., N. A. Rayner, J. J. Kennedy and S. A. Good, 2014. An Integrated Database of Ocean Temperature and Salinity Observations, Journal of Geophysical Research: Oceans, 119, 7139-7163, doi:10.1002/2014JC010053
Users must also acknowledge the following:
If applicable, for traceability, please describe which HadIOD bias corrections were used. This information can be found in the file global attribute ‘source’. This is particularly important if any additional corrections from the supplementary files were used. See also 'how to cite the data' in the HadIOD product user guide for more information and references for bias corrections. For example:
Please contact us if you need any assistance.
File contents are summarised in tables below.
A station (located horizontally and in time) may make one or more vertical profiles, with each profile comprising one or more observed levels (depths) of some geophysical variable. Variables listed below as 'station variables' contain station metadata (e.g. time or observation type) which apply to every observed level at a station. Variables listed below as 'data variables' contain the profile observations (temperature or salinity) and observation metadata (i.e. metadata which are potentially unique to each observation e.g. depth or a bias correction). The 'station variables' are 1D arrays with dimension N_OBS and the 'data variables' are 2D arrays with dimensions N_OBS x N_LEVELS. In the SST files N_LEVELS=1 and in the profile files N_LEVELS=400. A platform (e.g. a ship, buoy or profiling float) may make multiple stations; stations made by the same platform should share the same STATION_IDENTIFIER and STATION_TYPE (though in practice may not e.g. due to ID transcription errors).
Some general remarks on HadIOD Feedback files c.f. HadIOD files:
See also the 'User Guide' section above.
Further useful information about some of the variables is given below.
'STATION_TYPE' variable values
This is the NEMO station type. For EN4 observations we use the EN4 WMO_INST_TYPE variable (see EN4 profile file format) which is a simplified subset of WMO code table 1770. For other source data we map to the categories below. These are only broad categories and some observations may not be in the correct category.
SST file QC variables
The Met Office Hadley Centre QC suite used for the ICOADS and CMEMS observations comprises basic sanity checks, position track check, climatology check and buddy check. Flags from the individual checks have been combined and mapped to the feedback format QC variables as listed below. If any of the individual checks fail, the corresponding feedback format QC flag is set to fail.
An updated version of the Met Office Hadley Centre QC suite (MDS4) has been used for the CMEMS observations. This performs a similar suite of checks to the previous suite (MDS3) used for the ICOADS observations but with improvements to the performance of checks such as the track, climatology and buddy checks. A repeated value check was also added.
The duplicate check used for ICOADS observations is created within the HadIOD suite. This flags near-surface temperature observations taken from ocean profiles that are non-preferred likely duplicates of data in the profile files. If using both the SST and profile feedback format files this flag should be applied. For CMEMS there is minimal expected duplication with the profile data and so a duplicate check is not applied.
The SST track check used for ICOADS.2.5.1 (drifting buoy observations only) is an updated version of the drifting buoy QC checks described in Atkinson et al. . SSTs from individual drifters are tracked against a satellite-based reference and persistently poor quality observations are flagged. Position reports are used to flag buoys that have run aground or been picked-up. The coverage of the check is 1986-2015.
Feedback format flags use the following codes: 1=pass, 4=fail.
Profile file QC variables
The quality controlled profile observations in HadIOD are taken from EN4. The EN4 profile QC has many individual checks (see EN4 profile file format). In the EN4 data the overall outcomes of groups of individual checks are summarised in the POSITION_QC, PROFILE_POTM_QC, PROFILE_PSAL_QC, POTM_CORRECTED_QC and PSAL_CORRECTED_QC variables. These are mapped to equivalent variables in the profile feedback format files as shown below. The OBSERVATION_QC, POTM_QC and PSAL_QC variables apply to all levels in a profile. The POTM_LEVEL_QC and PSAL_LEVEL_QC variables are for individual levels in a profile.
Feedback format flags use the following codes: 1=pass, 4=fail.
Bias correction and uncertainty variables
Bias corrections should be added to the corresponding observations, e.g.:
Further rules on applying the corrections may be given in the files' 'usage' global attribute. Do read this. For example, the 'usage' may state that "DEPTH_CORR and SST_OBS_TYPE_CORR must be applied together (and not independently)" i.e. if applying SST_OBS_TYPE_CORR to SST_OBS, then DEPTH_CORR must also be applied to DEPTH (this might be because, for example, SST corrections are made relative to some reference level).
The feedback format files use the same HadIOD observation error model as the HadIOD files. The HadIOD product user guide explains the HadIOD error model and how to work with the bias corrections and uncertainties provided by HadIOD. Reading Section 2 of the user guide is strongly recommended. Because the names used for the bias correction and uncertainty variables in the feedback format files differ from the variable names used in the product user guide, the table below lists which variable names are equivalent.
Note that the profile feedback format files do not include bias correction and uncertainty variables for quantities that are not currently specified for profile data (shown as N/A).
Commercial and media enquiries
You can access the Met Office Customer Centre, any time of the day or night by phone, fax or e-mail. Trained staff will help you find the information or products that are right for you.
© Crown Copyright