Open Data Policy
Open Data Policy
25 November 2014
Clare Hubbard, Data Policy Manager
Contents
- Purpose
- Scope
- Introduction
- Current Services
- Data Categories
- Challenges impacting the release of data
- Open Data Policy
- Process for assessing Met Office data as Open
- Data that is not 'Open'
- Services for delivering Open Data
- Commitment to Open Data
- Policy review process
- Annex A Use of Met Office Open Data
- Annex B Drivers for the release of Data
Purpose
The purpose of this document is to describe the Met Office policy on Open data in the context of the Met Office's wider approach to management of data.
Scope
This policy applies to any data used within or to create Met Office products and services in the broadest sense, including all machine readable datasets such as observational data, weather model output, post processed data, maps and written scripts (see Figure 1). The policy refers to data created as part of the Public Task and more widely.
It should be noted that Met Office management information is published on line under the Publication scheme.
Introduction
The provision of 'data services' and their effective re-use is becoming an increasingly important way to extend the reach of the Public Weather Service (PWS). There is also an increasing emphasis in National and International policies to release public sector information to facilitate economic growth and provide transparency.
The flow of data underpins all Met Office activities and services, with enormous volumes of data generated daily. The forecast process starts with the collection and sharing of large quantities of observational data exchanged between Met Services. Numerical Weather Prediction, on which all forecasts now depend, generates further large volumes of data. Much of the data produced is dynamic with forecasts typically updated every few hours, so that data has a very limited useful lifetime and must be disseminated rapidly to be of value. To illustrate the volume of data involved the current size of the Met Office main data archive is in excess of 5 Petabytes, costing over £2M per annum to maintain (funding shared between PWS and HCCP - Hadley Centre Climate Programme funded through DECC and DEFRA).
Current Services
The Met Office has a long history of providing data services. The Met Office is one of over 190 members (National Meteorological Services) of the World Meteorological Organisation (WMO) who share data to enable each member to undertake its Public Task, this includes the exchange of Observations and agreed levels of Model data. As part of ECOMET, Met Office Public Sector Information in the form of observation bulletins and forecast models is catalogued and made available for re-use alongside the 24 other members (referred to as The Wholesale Catalogue). The Met Office Library and Archive Service curate The Nations Memory of the Weather and provide public access to the data and other articles.
The Met Office routinely transmits approximately 20 million messages a day to a wide variety of users including other National Meteorological Services, Government and Commercial Customers.
In November 2011 the Met Office responded to Governments request for opening up data by introducing Met Office DataPoint, the Met Office API. The current version of DataPoint provides access to:
- Forecast and observation map layers such as radar and cloud cover for the UK
- Forecast and observed site specific data such as temperature, wind speed and direction
- Regularly updated text forecasts for mountain weather and UK regions available in xml or JSON formats.
In July 2014 marine observation data was the latest addition to the API.
The majority (where Met Office has a licence to allow re-use of that data or owns the data) of the content on the Met Office website can be re-used under an Open Government Licence in line with the website Website terms of use. Where this is published in the form of downloadable data links are provided from the DataPoint Catalogue.
Data Categories
The Met Office categorises its data into three groups:
- Open
- Managed
- Internal
Open
Access to the data which has been assessed against the Open Data criteria as being Open and made available under an Open Government Licence.
Managed
Data that is made available externally but some level of management is applied for accessing the data. Licences may include Met Office Standard Terms and Conditions, Research Licences or another Licence developed specifically for that data. Data that is classified as managed may be free or charged for. Examples include:
- UKCP09 Climate projection data which is free but has specific licence terms associated with it.
- Wholesale data, the Met Office raw model data to meet RoPSI regulations, provided under Met Office Standard Terms and Conditions, charges may apply for the information.
- Historical Observation Data that is made available under terms and conditions, with possible associated charges, based on the usage of that data; research only, private use only, other public bodies or commercial use.
Internal
Data generated as part of the production process, typically high volume, used in the generation of products (system to system), but not distributed externally. Internal data volumes are difficult to estimate but are in excess of 100's Terabytes/day.
Challenges impacting the release of data
Data Volumes
Data volumes will continue to grow rapidly, typically doubling every 18 months as the resolution of weather models increases, the volume of satellite imagery grows, and as the range of weather products increases.
Technology
IT continues to develop rapidly, offering opportunities for data analysis, visualisation, storage and exchange. However this poses challenges for choosing the best technology, with the risk of chosen solutions becoming obsolescent all too rapidly.
Funding
The cost of maintaining the IT infrastructure and associated support costs to provide data are substantial. We need to ensure that robust revenue streams are identified before committing to the provision of data that others will use and commit time and effort into developing products and services.
Licence Terms of background data
The Met Office generates about 4% of the data that is used as an input to its operations. In some cases specific licence terms restrict the use of that data to create profitable revenue either by the Met Office or by third parties. In this case the data cannot be released as Open.
Open Data Policy
In response to the challenges described above the rationale for which Met Office data sets are provided as open data is determined by the a set of criteria which were agreed both within the Met Office and with PWS Customer Group that would be used to identify that data which would be made available under an Open Government Licence.
These criteria are shown below.
- Generation of the dataset is funded by the Public Weather Service and falls within the boundaries of the agreed PWS CSA Outputs and the Met Office Public Task.
The Met Office Outputs for delivering the Public Weather Service are described in the Customer Supplier Agreement (CSA) agreed between the Met Office and the Public Weather Service Customer Group. - The data is in an agreed format that does not require specialist tools or knowledge to interpret the data.
The Met Office will consider the most appropriate formats for the data to be provided in based on an understanding of the user communities for that data. In most cases this will be based on published standards (CSV, XML, JSON) users can request other formats which will be considered where the cost of doing so is not prohibitive. - Data volumes are reasonable and can be effectively disseminated in timescales shorter than the useful life of the data, typically minutes to 3 hours (this recognises that forecast data is updated several times a day and that an old forecast has limited value)
The Met Office is keen to ensure that the data made available as Open is as useful as possible, and to minimise the risk that the data mis-leading by being time-expired. - The Met Office owns the IPR or is licensed to provide the data under an open government licence.
The Met Office uses significant amounts of 3rd party data in its production processes and to put data on its website into context, therefore it has a duty of care not to mis- represent the data by publishing it for re-use where the terms do not allow this. - The data is deemed to be operational, is well documented and described by appropriate metadata and with service management in place to monitor production processes and address operational problems as the need arises.
To ensure maximum benefit and allow the data to be used to create profitable services that generate economic growth, the data provision needs to be reliable and trusted. In order to do this a minimum standard of service is required which can only be delivered with the data being considered 'operational'. - The data is consistent with other Met Office open datasets to minimise the risk of inconsistent or conflicting forecast information, particularly where protection of life or property may be compromised.
The Met Office has been criticised in the past for inconsistent data, it can also cause confusion to the users if any differences are not understood. - The scale of public interest is not outweighed by the cost of generation and delivery.
The Met Office has limited resources for investing in the delivery of Open data and therefore it wishes to maximise the benefit of the resources used to create any new Open data by ensuring that the maximum number of users benefit from its release. Information from Analytics and consultation with the current users of Met Office Open data will be used to inform the level of interest in any particular dataset. - Before release, datasets will be assessed on the basis of long term economic sustainability, considering areas such as scaling of distribution and opportunity costs.
The Met Office has learnt from experience that there is an inherent commitment to the longevity of any dataset being made available and will therefore consider any future plans for the dataset prior to release. It will also consider the impacts on the capacity of the infrastructure required, with the possibility of limits being applied to datasets to ensure that it is economically viable to maintain the delivery of the data.
In addition to the above, data may be released as Open on request of a customer as part of the contract for which the dataset is being created. In these cases the licence terms of the data that is to be used as an input will be reviewed to ensure there are no reasons why this cannot be the case.
Process for assessing Met Office data as Open
As new data is included within the description of the Public Task Outputs, the Met Office Data Policy Manager works with the relevant Information Asset Owners to undertake an assessment of the data against the Open Data criteria. Once the assessment is complete a recommendation is submitted to the Met Office Internal Governance Group for agreement.
Data that is not 'Open'
Data that does not fall under the criteria set out above may still be accessible for wider usage but may be charged for. This is in line with current Government policy. The Open Data White Paper, published by the Cabinet Office in June 2012 states that the general principles for access to Public Sector Information (PSI) are that data should be provided free wherever appropriate and possible, or at a fair price where it is costly for the public sector to provide it, or where it is fairer to the UK tax payer to secure value from it. This is supported by the Re-use of Public Sector legislation which supports a Trading Funds remit 'to develop profitable commercial outlets for their services in order to offset core and central overheads, return a dividend to HM Treasury and raise money for investment.' -The re-use of Public Sector Information, The National Archives, 2005
These principles have been applied to the release of Met Office data. All of the data that meets the Open data criteria is made available free at the point of use, within the scope of the Terms and conditions - DataPoint.
The Wholesale Catalogue while used to provide the mechanism for meeting our Reuse of Public Sector Information (RoPSI) duties has a global market and therefore the data supplied is charged for at market price.
Services for delivering Open Data
The Met Office is keen to make the Open Data that it publishes as useful as possible; this led to the launch of the DataPoint API. For the majority of users there is no cost associated with the use of the DataPoint API as their usage profile is within the Terms and conditions - DataPoint described. Where a user wishes to make higher volumes of requests then a contribution to the additional infrastructure is required.
Commitment to Open Data
- In 2013 the DataPoint Catalogue was updated to include signposting to all the data that was available for download from the Met Office Website.
- The Met Office is working with the Open Data Institute (ODI), initial work has included ODI providing peer review of the User Experience of DataPoint, to inform improvements and more recently Met Office has joined the ODI Partner Programme. ODI will be providing advice and training in the creation of APIs and training in Open Data to the Met Office Information Asset Owners.
- In the BIS 2014 Summer Statement the Met Office, as a member of the Public Data Group, will be undertaking a review of the quality of the DataPoint service using the Open Data Certificate mechanism.
- The DataPoint User Forum (GoogleGroup) is used to gain valuable feedback on the priority of the data that should be made available and changes to the technologies and provide feedback on the data already made available.
- The Met Office is a member of the Information Fair Trading Scheme as a demonstration of its commitment to providing access to the Legal.
- Requests for additional data to be published as Open can be made through the application for public sector information or data.gov.uk.
Policy review process
This Policy will be reviewed by BIS (ShEx), Cabinet Office (Transparency Team) and the Open Data Institute (ODI) approved by the Met Office Data Policy Steering Group (DPSG). Any significant implications for Public Weather Service (PWS) will also require sign off by the PWS Customer Group (PWSCG).
The Met Office Open Data Policy will be reviewed annually; updates will be undertaken more frequently should external drivers require it.
Annex A Use of Met Office Open Data
DataPoint now has over 6200 registered users, in the region of 25 million calls for data were made during August 2014, see Figure 2. In October 2014, we saw un-precedented demand due to a single user unnecessarily requesting large amounts of data every 7 seconds.
Access to DataPoint data is made available for free under an Open Government Licence.
Some examples of the uses of DataPoint data are provided below.
Flip-dot by Uniform
Wave height and duration Marine Observation Data
Scope Nights
Threshold based traffic light app for stargazing enthusiasts using UK site specific forecast data.
Bike / Tube Barometer
Designed by Richard Pope, it applies algorithms to TFL and Met Open data to indicate which the best method to travel to work is.
Annex B Drivers for the release of Data
PWS Reach
The Public Weather Service reach strategy is to minimise impacts of extreme weather through warnings and provide as many members of the public both UK and overseas with accurate, consistent and local weather forecast information on all the channels where people want to access them.
Government Policy
Current UK government policy is strongly encouraging the release of public sector information to facilitate economic growth and demonstrate transparency. In particular the government announced a number of Open Data Measures in the autumn statement 2011. This has been followed by the publication of the Cabinet Office's Open Data White Paper and the BIS Open Data Strategy.
Legislation
Domestic and EU legislation is also driving greater release of information by public sector bodies, for example the EU INSPIRE Directive, EU PSI Directive and the Government's 'Right to data' Bill. The Met Office also needs to comply with competition law by making data used for its competed services to others on a fair and level playing field.
National Archive
The Met Office has a Statuary duty under the Public Records Act to make available the data and information it has created in the performance of its public duties after 30 years (moving to 20 years). Met Office is currently a Place of Physical deposit for all the paper records and we are working with the National Archive to become a place of deposit for electronic data.
Demand
The growth in web and mobile applications is placing an ever growing demand for weather data and information. However this demand is emerging and the challenge is to understand and quantify future demand, to guide the choice of technology and establish the likely costs of delivery.
Collaborative based services
An increasing number of services are based on partner arrangements or collaborations, combining the data and expertise from different organisations from the UK and internationally to supply new products, for example the joint Met Office/Environment Agency Flood Forecasting Centre and Natural Hazards Partnership (NHP). The Met Office is investigating how best to extend existing or create new ways to effectively and affordably share operational data required for these collaborations often with non-expert users, but it does not always have the copyright or intellectual property rights to do this unilaterally.
International
As part of its international commitments, the Met Office exchanges data with other met services. For example WMO resolution 40 defines a set of observational and model data the Met Office must provide to other National Met Services. A number of WMO initiatives are likely to guide the range and format of datasets exchanged.