25 November 2014
Clare Hubbard, Data Policy Manager
The purpose of this document is to describe the Met Office policy on Open data in the context of the Met Office's wider approach to management of data.
This policy applies to any data used within or to create Met Office products and services in the broadest sense, including all machine readable datasets such as observational data, weather model output, post processed data, maps and written scripts (see Figure 1). The policy refers to data created as part of the Public Task and more widely.
Figure 1 Categorisation of data managed within to fulfil Met Office duties
It should be noted that Met Office management information is published on line under the Met Office Freedom of Information Publication scheme.
The provision of 'data services' and their effective re-use is becoming an increasingly important way to extend the reach of the Public Weather Service (PWS). There is also an increasing emphasis in National and International policies to release public sector information to facilitate economic growth and provide transparency.
The flow of data underpins all Met Office activities and services, with enormous volumes of data generated daily. The forecast process starts with the collection and sharing of large quantities of observational data exchanged between Met Services. Numerical Weather Prediction, on which all forecasts now depend, generates further large volumes of data. Much of the data produced is dynamic with forecasts typically updated every few hours, so that data has a very limited useful lifetime and must be disseminated rapidly to be of value. To illustrate the volume of data involved the current size of the Met Office main data archive is in excess of 5 Petabytes, costing over £2M per annum to maintain (funding shared between PWS and HCCP - Hadley Centre Climate Programme funded through DECC and DEFRA).
The Met Office has a long history of providing data services. The Met Office is one of over 190 members (National Meteorological Services) of the World Meteorological Organisation (WMO) who share data to enable each member to undertake its Public Task, this includes the exchange of Observations and agreed levels of Model data. As part of ECOMET, Met Office Public Sector Information in the form of observation bulletins and forecast models is catalogued and made available for re-use alongside the 24 other members (referred to as The Wholesale Catalogue). The Met Office Library and Archive Service curate The Nations Memory of the Weather and provide public access to the data and other articles.
The Met Office routinely transmits approximately 20 million messages a day to a wide variety of users including other National Meteorological Services, Government and Commercial Customers.
In November 2011 the Met Office responded to Governments request for opening up data by introducing DataPoint, the Met Office API. The current version of DataPoint provides access to:
In July 2014 marine observation data was the latest addition to the API.
The Met Office categorises its data into three groups:
Access to the data which has been assessed against the Open Data criteria as being Open and made available under an Open Government Licence.
Data that is made available externally but some level of management is applied for accessing the data. Licences may include Met Office Standard Terms and Conditions, Research Licences or another Licence developed specifically for that data. Data that is classified as managed may be free or charged for. Examples include:
Data generated as part of the production process, typically high volume, used in the generation of products (system to system), but not distributed externally. Internal data volumes are difficult to estimate but are in excess of 100's Terabytes/day.
Data volumes will continue to grow rapidly, typically doubling every 18 months as the resolution of weather models increases, the volume of satellite imagery grows, and as the range of weather products increases.
IT continues to develop rapidly, offering opportunities for data analysis, visualisation, storage and exchange. However this poses challenges for choosing the best technology, with the risk of chosen solutions becoming obsolescent all too rapidly.
The cost of maintaining the IT infrastructure and associated support costs to provide data are substantial. We need to ensure that robust revenue streams are identified before committing to the provision of data that others will use and commit time and effort into developing products and services.
The Met Office generates about 4% of the data that is used as an input to its operations. In some cases specific licence terms restrict the use of that data to create profitable revenue either by the Met Office or by third parties. In this case the data cannot be released as Open.
In response to the challenges described above the rationale for which Met Office data sets are provided as open data is determined by the a set of criteria which were agreed both within the Met Office and with PWS Customer Group that would be used to identify that data which would be made available under an Open Government Licence.
These criteria are shown below.
In addition to the above, data may be released as Open on request of a customer as part of the contract for which the dataset is being created. In these cases the licence terms of the data that is to be used as an input will be reviewed to ensure there are no reasons why this cannot be the case.
As new data is included within the description of the Public Task Outputs, the Met Office Data Policy Manager works with the relevant Information Asset Owners to undertake an assessment of the data against the Open Data criteria. Once the assessment is complete a recommendation is submitted to the Met Office Internal Governance Group for agreement.
Data that does not fall under the criteria set out above may still be accessible for wider usage but may be charged for. This is in line with current Government policy. The Open Data White Paper, published by the Cabinet Office in June 2012 states that the general principles for access to Public Sector Information (PSI) are that data should be provided free wherever appropriate and possible, or at a fair price where it is costly for the public sector to provide it, or where it is fairer to the UK tax payer to secure value from it. This is supported by the Re-use of Public Sector legislation which supports a Trading Funds remit 'to develop profitable commercial outlets for their services in order to offset core and central overheads, return a dividend to HM Treasury and raise money for investment.' -The re-use of Public Sector Information, The National Archives, 2005
These principles have been applied to the release of Met Office data. All of the data that meets the Open data criteria is made available free at the point of use, within the scope of the Fair Use Policy.
The Wholesale Catalogue while used to provide the mechanism for meeting our Reuse of Public Sector Information (RoPSI) duties has a global market and therefore the data supplied is charged for at market price.
The Met Office is keen to make the Open Data that it publishes as useful as possible; this led to the launch of the DataPoint API. For the majority of users there is no cost associated with the use of the DataPoint API as their usage profile is within the Fair Use Policy described. Where a user wishes to make higher volumes of requests then a contribution to the additional infrastructure is required.
This Policy will be reviewed by BIS (ShEx), Cabinet Office (Transparency Team) and the Open Data Institute (ODI) approved by the Met Office Data Policy Steering Group (DPSG). Any significant implications for Public Weather Service (PWS) will also require sign off by the PWS Customer Group (PWSCG).
The Met Office Open Data Policy will be reviewed annually; updates will be undertaken more frequently should external drivers require it.
DataPoint now has over 6200 registered users, in the region of 25 million calls for data were made during August 2014, see Figure 2. In October 2014, we saw un-precedented demand due to a single user unnecessarily requesting large amounts of data every 7 seconds.
Access to DataPoint data is made available for free under an Open Government Licence.
Some examples of the uses of DataPoint data are provided below.
Wave height and duration Marine Observation Data
Threshold based traffic light app for stargazing enthusiasts using UK site specific forecast data.
Display with a needle pointing to either a bike or London Underground symbols
Designed by Richard Pope, it applies algorithms to TFL and Met Open data to indicate which the best method to travel to work is.
The Public Weather Service reach strategy is to minimise impacts of extreme weather through warnings and provide as many members of the public both UK and overseas with accurate, consistent and local weather forecast information on all the channels where people want to access them.
Current UK government policy is strongly encouraging the release of public sector information to facilitate economic growth and demonstrate transparency. In particular the government announced a number of Open Data Measures in the autumn statement 2011. This has been followed by the publication of the Cabinet Office's Open Data White Paper and the BIS Open Data Strategy.
Domestic and EU legislation is also driving greater release of information by public sector bodies, for example the EU INSPIRE Directive, EU PSI Directive and the Government's 'Right to data' Bill. The Met Office also needs to comply with competition law by making data used for its competed services to others on a fair and level playing field.
The Met Office has a Statuary duty under the Public Records Act to make available the data and information it has created in the performance of its public duties after 30 years (moving to 20 years). Met Office is currently a Place of Physical deposit for all the paper records and we are working with the National Archive to become a place of deposit for electronic data.
The growth in web and mobile applications is placing an ever growing demand for weather data and information. However this demand is emerging and the challenge is to understand and quantify future demand, to guide the choice of technology and establish the likely costs of delivery.
An increasing number of services are based on partner arrangements or collaborations, combining the data and expertise from different organisations from the UK and internationally to supply new products, for example the joint Met Office/Environment Agency Flood Forecasting Centre and Natural Hazards Partnership (NHP). The Met Office is investigating how best to extend existing or create new ways to effectively and affordably share operational data required for these collaborations often with non-expert users, but it does not always have the copyright or intellectual property rights to do this unilaterally.
As part of its international commitments, the Met Office exchanges data with other met services. For example WMO resolution 40 defines a set of observational and model data the Met Office must provide to other National Met Services. A number of WMO initiatives are likely to guide the range and format of datasets exchanged.
Last updated: 15 April 2016