a blog by Owen Boswarva

Posts tagged Ordnance Survey

17 Jul

Post: 17 July 2014

The Public Data Group is a collection of four data-rich government organisations (Companies House, Land Registry, Met Office and Ordnance Survey) that report to the UK’s Department for Business, Innovation & Skills (BIS).

The PDG organisations are trading funds, encouraged by Government to generate commercial revenue from the data assets that they control. However all four organisations make available at least some data under an open licence.

On Tuesday the Public Data Group issued a statement containing some commitments to future releases of open data. The statement is a bit short on detail, so this post is an attempt to add some context to the planned releases.


Companies House

The headline announcement in the PDG statement is the decision that Companies House will “make all of its digital data available free of charge”, from the second quarter of 2015 (April - June).

This follows previous initiatives to open up Companies House data:

The current Companies House Price List is online. My interpretation of Tuesday’s announcement is that Companies House will, at minimum, remove the £1 charge for access to individual Company Records via WebCHeck. “Electronic images” will also be free, which I think means PDF copies of the records.

The important unanswered question is whether this release will also include any new bulk downloads of data. According to the Open Definition, a dataset is only properly open data if it is available in bulk.

Bulk release is necessary for any kind of serious analysis of companies data. If the Government is serious about leveraging the free availability of Companies House data to “boost the UK economy”, then bulk release is essential.

Land Registry

According to the PDG statement Land Registry “will release their Price Paid Data for commercially owned properties for free by March 2015.”

There are no further details provided. It will be interesting to see what data is contained in that release. Land Registry currently makes available its Price Paid Data for residential transactions back to 1995, as open data. However it does not publish any statistics on commercial transactions. My past understanding was that Land Registry did not maintain a separate dataset for commercial sales (of either land or properties).

Update 9 September 2014: see this new Land Registry material on commercial and corporate ownership data. ]

There are several open questions: How complete or extensive is Land Registry’s data on prices paid for commercially owned properties? Does Land Registry intend to release data on historical as well as new transactions (bearing in mind that the initial release of residential Price Paid Data was only new transactions)? And is there likely to be resistance from commercial property owners to the open publication of sale prices?

The PDG statement also says that in 2014/15 Land Registry intends to “make the whole Index Map polygon layer covering England and Wales available at a cost recovery price.”

This is not an open data release, of course. The news may be welcomed by Land Registry licensees, but it remains to be seen how much of a saving they will realise. “Cost recovery price” should not be confused with “marginal price”. Index Map polygons are based on Ordnance Survey spatial data, so (as we have seen with the INSPIRE Index Polygons) it will be OS pricing that determines the actual cost of reuse.

Met Office

There is not much open data on the horizon from Met Office. However the PDG statement says Met Office is creating something called the “National Archive for the Nations Memory of the Weather”, and that “a selection of this will be available as Open Data.”

This could be significant or not, depending on what Met Office decides to release. Data required to maintain the nation’s “memory of the weather” could range from detailed weather observations, to historical documents, to anecdotal information about notable weather events.

Ordnance Survey - enhancements to OS OpenData

Ordnance Survey’s open data programme is more extensive than those of the other trading funds, as it was launched under the previous Government. Tuesday’s PDG statement notes several future developments.

The existing OS Street View product (raster base-mapping) will be enhanced with new features added such as car parks, major paths, major cycle routes and hill.

OS will release an “enhanced Gazetteer”. This is presumably the “Gazetteer of Great Britain" that OS demonstrated at GeoBusiness 2014 in May. I have seen some sample data for this product (via the OS Insight developer programme); it looks like a useful addition to the OS OpenData suite.

Ordnance Survey - Public Rights of Way

OS will be “working with” Defra to “provide consultancy, technology and to enhance public access (through a portal) to Rights of Way data”. I guess this is good news, though lack of a portal is not the main blockage to release of Public Rights of Way data.

I’ve written about PRoW data before. The main problem is that there has been no organised effort from central government to encourage local councils to release the data. OS is part of that problem (and therefore has to be part of the solution) because most councils use OS data to maintain their Rights of Way maps and need OS’s permission to release those maps as open data. However DCLG and the Local Government Association should be pushing this harder as well. The ideal would be publication of an open national PRoW dataset, collated from the many local sources.

We already have a pretty good portal for the 80 or so local PRoW datasets already available as open data: Barry Cornelius’s Rowmaps site.

Ordnance Survey - Derived River Network

Of the various open data commitments mentioned in the PDG statement, this is the one I am personally most excited about: Ordnance Survey plans to release a new “Derived River Network” open data product.

My understanding is that this dataset will be derived from the new Water Layer in MasterMap. (The other relevant dataset in this space, the Detailed River Network that OS developed with the Environment Agency, is being deprecated.)

The River Network is a dataset that I have been campaigning for since 2012, via the ODUG process and other channels such as the Defra Transparency Panel.

Since then more spatial data about our rivers has become available, most recently the Cycle 2 draft of the EA’s Water Framework Directive (WFD) River Waterbodies dataset (last month’s post). However I am gratified to see that there is now sufficient support for open release of a good general-purpose vector map of the river network.

The Derived River Network dataset will complement the Environment Agency’s recent open data release of live flood warning and river level feeds as well as plans to release the NaFRA national flood risk dataset.

Photo credit: Open_Data_stickers.jpg by Jonathan Gray (CC0 1.0). It’s an iconic image and I was too lazy to find something more imaginative to illustrate this post.

5 Dec

Post: 5 December 2013

Update 2 January 2014: I’ve received a response from ONS by e-mail. Please see the end of the post.

Which open data licence applies to ONS’s Postcode Directory: the Open Government Licence or the OS OpenData Licence?

Yesterday SK53, a prolific OpenStreetMap contributor, released a blog post on the subject of British postcodes. He made an argument for increasing the coverage of GB postcode information on OSM, noting that:

In 2010 postcode centroids were made available through the Ordnance Survey Open Data scheme, under the brand CodePoint Open. Subsequently it was found that the license associated with this data prevented it being used directly in OSM. More recently the Office of National Statistics have released an (identical) data set which is not encumbered by the license of CodePoint Open.

I’m not an OpenStreetMap contributor myself but I’ve been aware for some time that Ordnance Survey’s OS OpenData Licence presents difficulties for OSM. Although the OS OpenData Licence complies with the Open Definition and is in most respects the same as the Open Government Licence, it does contain some additional requirements on attribution and sub-licensing.

If the Office for National Statistics is now providing the same GB postcode data as Ordnance Survey, but under the Open Government Licence, that seems like a welcome resolution to the OSM problem.

But is that really the licensing position?


Code-Point Open vs the ONS Postcode Directory

At the moment there are two primary sources for the authoritative national geocoded postcode dataset. The first is the Code-Point Open download available from the Ordnance Survey website. The second is the series of Postcode Directory (ONSPD) downloads available from ONS’s Open Geography portal.

Most users will find the ONS portal to be the preferable source. The ONSPD downloads contain all of the Code-Point Open data, plus additional ONS fields. The ONS downloads are immediately available, whereas the Ordnance Survey supply is via a link sent to the requester’s e-mail address. The ONS downloads are provided in a choice of formats and, most importantly, ONS maintains an archive of previous releases rather than just the most recent.

If the ONS downloads are subject to the Open Government Licence, and not the OS OpenData Licence, that is another good reason to procure the data from ONS rather than from Ordnance Survey.

Postcodes and postcode coordinates are reference data, which means they are often incorporated in other data products that contain data from multiple sources. Leaving aside the particular issues for OSM, it’s always easier to build products from the most broadly re-usable data sources. The OGL is more widely used and understood and (slightly) less restrictive than the OS OpenData Licence, so it better supports the development of added-value data products.

Lack of clarity on licensing of ONS postcode datasets

The problem is that ONS is providing conflicting information about licensing for re-use of the Postcode Directory.

The Licences page on the main ONS website includes the following:

The ONS postcode products are subject to the Open Government Licence, and Ordnance Survey OpenData Licence.

That page predates the launch of the Open Geography portal. Licensing information on the Open Geography portal itself is not very well signposted; however the FAQs page includes a link back to the Licences page on the main site.

There is also licensing information included with the ONS Postcode Directory (ONSPD) downloads themselves, and that is the source of confusion. The version notes contain the following:

Copyright and reproduction

© Crown copyright 2013

You may re-use this information (not including logos) free of charge in any format or medium, under the terms of the Open Government Licence. However, the following attribution statements must be acknowledged or displayed on any product using ONS data:

‘Contains Ordnance Survey data © Crown copyright and database right 2013’

‘Contains National Statistics data © Crown copyright and database right 2013’

'Contains Royal Mail data © Royal Mail copyright and database right 2013'

Click here to view the Open Government Licence, or write to the Information Policy Team, The National Archives, Kew, London TW9 4DU

That statement confirms that the dataset includes Ordnance Survey and Royal Mail intellectual property, but the attribution requirements only go as far as those in the Open Government Licence. There is no explicit reference to the OS OpenData Licence or to the additional terms in that licence.

The same wording appears in the notes for previous versions of the ONSPD available from the Open Geography Portal.

The ONSPD downloads also include a metadata file, which does include the URL of the OS OpenData Licence:

Contains Ordnance Survey data © Crown copyright and database right [2013]

However the metadata file is in XML format, which means some re-users of the data (particularly those who download the ONSPD as a ready-to-use Access database) will likely not notice it.

What is the actual licensing position?

It’s possible that Ordnance Survey and Royal Mail have agreed that their data may be re-used under the OGL when it is released as part of the ONS Postcode Directory. However if that is the case it’s not clear why OS has not also released Code-Point Open under OGL.

I think it is more likely that the ONSPD version notes are in error, and that the copyright section should include a reference to the OS OpenData Licence.

Where does that leave re-users? The OGL and OS OpenData are both open data licences, and the differences do not really affect the scope for direct re-use of the datasets. The potential for error lies mostly with businesses who may have incorporated ONSPD data in their own products, with the risk that downstream attribution does not adequately reflect the requirements of OS and Royal Mail.

Open data licences do not normally indemnify the licensee against a breach of third party IP rights. However any licensee who has relied on the ONS version notes in error will have done so in good faith. Harm to OS and Royal Mail is likely to be trivial, and given that both organisations work closely with ONS it could be argued that they have been insufficiently vigilant in not requiring a correction to the release notes.

However for OpenStreetMap there might be practical consequences. If contributors are encouraged to add ONSPD data onto OSM, and ONS subsequently clarifies that the OS OpenData Licence will apply going forward, OSM may have to take remedial action to stop the addition of further such data.

Update (2 January 2014)

I have today received an e-mail from ONS confirming that the version notes for the ONS Postcode Directory should refer to the OS OpenData Licence terms:

I’ve recently seen your blog (dated 5 December) regarding licensing of our ONSPD and the discrepancy between the Open Government Licence and OS OpenData Licence terms. You are quite correct; thank you for drawing it to our attention. There is an inconsistency between our Licence web page and information provided with the metadata. We will correct the metadata so that it meets the full requirements of OS Opendata. The Version Notes will also contain a statement drawing customers’ attention to the inconsistency and outline the correct position.

This action has been agreed with Ordnance Survey.

30 Aug

Post: 30 August 2013

So there’s this:

Assessing the Value of Ordnance Survey OpenData to the Economy of Great Britain: Full Interim Report

I received the above report yesterday in response to a Freedom of Information request that I made to Ordnance Survey last month.

The report is an economic value study of Ordnance Survey’s OS OpenData suite of free geographic data products, prepared in November 2012 by ConsultingWhere and ACIL Tasman.

Ordnance Survey published their own synopsis of the interim report in June, following an earlier FOI request. (Although the report is “interim”, plans to deliver a final version have been shelved. More on that below.)


New and Missing Information

There are a substantial number of redactions in the released copy of the full report. You can read the FOI correspondence if you’re interested in the detail of which exemptions Ordnance Survey have applied. In a nutshell: names of individuals who are not civil servants have been blacked out (as personal information), case studies including the names of participating organisations have been blacked out (as information provided in confidence), and the internal breakdown of the amount Ordnance Survey receives under the OS OpenData contract has been blacked out (as commercially sensitive).

There are very few unredacted figures in the report that were not previously disclosed in the synopsis. The main exception seems to be the revenue estimate of £ 5.3 million per annum used as the basis of the “counterfactual” case, i.e. the income anticipated for Ordnance Survey if the OS OpenData products were fee-generating rather than free. The footnote giving the source of this estimate is, however, redacted. (See section 6.2 on page 27.)

I regret that Ordnance Survey have declined to release the breakdown of the £ 20 million per annum they receive under the “CLG contract” as compensation for losses of revenue consequent upon the decision to make the OS OpenData products free at the point of delivery. It is difficult to see how Ordnance Survey’s commercial position would be harmed by releasing the figures, and withholding them has the effect of shielding the arrangements from scrutiny. The report makes no comment on the validity of the breakdown, and indeed says rather pointedly:

we should note that the CLG contract is taken as a matter of fact. We are not in a position to comment on the level of the funding, nor the split of revenues between public and private sectors.

No Final Report

The original plan was to follow up this interim report with a final report:

For the final report, it was envisaged that the findings would be summarised to be accessible to a wider audience and the interim conclusions updated through a process of wider consultation.

However Ordnance Survey’s synopsis says:

The planned Final Report – this was intended to be a summary of the findings that would be drafted so as to be accessible to a wider audience and to enable the further updating of conclusions through wider consultation. This stage has been superseded by this Synopsis of the Interim Report.

There is a bit of a discrepancy here. Was the final report always going to be simply a summary of the findings, to “enable” wider consultation, or was the original intention to consult more widely and include updated conclusions in the final report itself? There must have been more to the plan for a final report than simply a releasable summary; else Ordnance Survey’s synopsis would have been called the final report.

The charitable explanation is that Ordnance Survey and/or BIS saw no need for a further report because the findings in the interim report were sufficient to justify OS OpenData as an ongoing economic proposition. The report’s topline conclusion is that OS OpenData will generate net growth in GDP of between £ 13.0 million and £ 28.5 million per annum by 2016 — and this estimate is “almost certainly understated” (page 1).

Why then did Ordnance Survey and BIS sit on the report for a year and a half, publishing the synopsis only after I submitted my first FOI request? Based on these findings, OS OpenData is a clear-cut success story — why is Government not holding it up as a model for future releases of economically useful public data?

We can at least speculate. Is the report too robust in its argument for open data? Perhaps the success of OS OpenData presents the “threat of a good example”? If the study’s methodology gained general acceptance it would be difficult for Government to explain why it has not pressed ahead more aggressively with release of other core reference data.

The current Government’s transparency policy has focused mainly on release of spending and performance data. Their target audience is mostly imaginary “armchair auditors” who want to bash public services improve public services and hold public authorities to account. The OS OpenData products, on the other hand, are datasets that people actually want to use. And of course (as the report makes clear but the synopsis version does not) the underpinning economic rationale for OS OpenData and UK open data policy more generally was developed by the previous Labour administration. In contrast this Government has shown little understanding of open data’s wider economic potential; its only big idea has been to open a crèche for start-ups. Most of the economic growth the report attributes to OS OpenData is from business efficiencies, not innovation or “app dev”, so the findings are not exactly on-message.

The Economics of Open Data

Despite all the redactions (including a full ten pages of case study material) the report provides a substantially more balanced perspective on the arguments for OS OpenData than does the synopsis. Most of the value is in the discussion of the study methodology and different approaches to modelling the impacts of open data. The numbers themselves I take with a grain of salt, on the “garbage in garbage out” principle; whatever the merits of the model, there’s only so much you can do with download statistics and anecdotal information from interviews. (The assumptions behind the download analysis are particularly loopy.)

The report itself is far more readable than Ordnance Survey’s synopsis, but the most stark difference is in the tone. That topline GDP estimate is buried halfway through the synopsis rather than highlighted in the executive summary. The report’s recommendations are presented verbatim in the synopsis — but there is no indication from Ordnance Survey or BIS whether they have taken any actions on the recommendations.

The report includes (as Appendix B) a copy of the peer review from economic consultants Prabhat Vaze and Patricia Seex secured to ensure the validity of the report’s approach. The synopsis, on the other hand, ignores the contents of that review as well as the report’s explanation of how the points from the review were incorporated into the methodology. It concludes instead with a two-page letter from BIS’s Chief Analyst that rubbishes the report’s findings based on inherent difficulties in information-gathering that must surely have been obvious when the research was commissioned. It is difficult to escape the impression that BIS finds the success of OS OpenData altogether rather awkward.

Does OS OpenData Have a Future?

Despite what seem to be attempts to downplay the findings, this economic value study should make it difficult for BIS to withdraw support for the OS OpenData initiative. However it does highlight a couple of dependencies of which re-users should be aware.

The first is the “CLG contract”, i.e. the £ 20 million per annum that Ordnance Survey receives from Government in compensation for loss of revenue and to fund delivery of the initiative. The terms of that contract are not public, so it is difficult to judge how reliable a basis it provides for continuation of the OS OpenData initiative.

The second dependency is the royalties payable to Royal Mail for use of data embedded in OS OpenData products. Those royalties are provided for in the CLG contract (page 29 in the report), and we may hope Royal Mail remains satisfied with the arrangement. However Royal Mail is headed for privatisation and could take a different position in future, particularly if it decides to develop data products that compete with Ordnance Survey’s.


The OS Vector MapDistrict image used in this post contains Ordnance Survey data © Crown copyright and database right 2013.

27 Mar

Post: 27 March 2013

Update Apr 23: cancellation of Pinpoint has now also been confirmed on the Royal Mail website.

According to a post on the Allies Computing website, Royal Mail have confirmed they will not proceed with their controversial Pinpoint positional data capture project. 

Andrea Martin, Managing Director of Data Services for Royal Mail, is quoted:

"Royal Mail announced in the summer of 2012 a pilot initiative in East Anglia to map the co-ordinates of home and business. The pilot explored the potential for Royal Mail to support the location-based information marketplace. Following the completion and full review of the pilot, we have decided not to progress the initiative."

Image source: Harrow Council

I’ve posted previously about the origins and progress of the Pinpoint project. If successful Pinpoint would have added geographic coordinates to Royal Mail’s existing Postcode Address File (PAF) dataset.

It’s currently unclear why Pinpoint has been cancelled, though Royal Mail’s attempt to enter the market for geocoded address data had raised eyebrows within Britain’s geographic information community.

Pinpoint would have provided competition for Ordnance Survey’s well-regarded but rather pricey AddressBase products. However the future shape of the address data market is uncertain, with Royal Mail privatisation on the cards and both Royal Mail and Ordnance Survey under pressure to unlock their address data assets as part of an “open data” National Address Dataset.

Royal Mail had reportedly allocated an investment of £10 million or more to the Pinpoint initiative.

14 Jan

Post: 14 January 2013

Data licensing can be tricky.

There are data sets that are clearly open data and data sets that are clearly not. 

Then there are those that have potential, if you look at them in the proper light …

The Indices of Multiple Deprivation (IMD) are a multi-domain measure of relative levels of deprivation within small geographic areas of the UK.

Indices for England, Scotland, Wales and Northern Ireland are produced separately by Government and have slightly difficult criteria, but it’s possible to make comparisons across the data for different countries.

IMD data sets are used widely in the public sector and for academic research. I’ve used IMD data myself as inputs into risk models for the insurance industry. The crime domain in particular is useful in predicting geographic variability of economic losses to theft, civil disorder, etc.

Last month the 2012 version of the Scottish IMD was released by the Scottish Government. Alasdair Rae has put together a useful website that visualises the SIMD 2012 data interactively on a map.

In England and Wales the small-area geography used in the IMD is the Lower Super Output Area. LSOA boundaries are unambiguously available as open data, and may be downloaded from the ONS website and elsewhere.

However in Scotland the unit of geography underlying the IMD is called a Data Zone. The Data Zone boundaries currently in use were produced in 2004 but built up from 2001 Census output areas.

There’s background information on Data Zones here and here. This is what they look like:


Pretty much all administrative geography for England, Scotland and Wales is now available as open data. So I was surprised to see the restrictive licensing terms presented on the download page for the Scottish Government’s Geography Data (which includes the Data Zone boundaries).

You can read the licence on the Scottish Neighbourhood Statistics website. (Click on Go to Data Download, then Download Geography. The SNS site doesn’t like direct links.)

The Geography Data licence starts like this:

We the Scottish Government grant you a non-exclusive non-transferable licence (without the right to sublicense) to copy and use the Data which is derived from Ordnance Survey data and as such is subject to the terms and conditions of the licence agreement between The Scottish Government and Ordnance Survey.

No transferability or sub-licensing, so right away that says it’s not an open licence. Without transferability it would be very difficult to put the Geography Data on a public website in a legally compliant manner.

The wording tells us clearly that the download includes Ordnance Survey derived data. However it’s ambiguous whether the Scottish Government is claiming any additional database rights.

The licence then says:

For any use of this material a Click-Use PSI Licence is also required.

with details of how to arrange that via the Office of Public Sector Information.

The problem is that the Click-Use Licence system was phased out from 2010, when the Open Government Licence (OGL) was introduced. The link provided on the SNS’s download page now resolves to a National Archives page about the OGL.

Click-Use Licences were typically issued for a five-year period, so in theory there could be re-users out there using the SNS Geography Data under those terms. However the OPSI is no longer issuing new Click-Use Licences.

Interpreted literally that means nobody who downloads the Geography Data subsequently can comply with the licensing terms on the download page.

It’s fairly obvious the licence on the download page is out of date. Does that mean we can substitute the terms of the Open Government Licence? That would enable us to do away with the restrictions on transferability and sub-licensing, both of which are supported by the OGL.

There is certainly a convention that the OGL simply replaces the Click-Use Licence, and we have statements to that effect from National Archives.

But I’ve never been entirely comfortable that the Government nailed that down properly, and there are substantial differences between the Click-Use Licence and the OGL. The Click-Use Licence does not specifically set out the scope of permitted re-use, and it seems to exclude mapping data. National Archives has also muddied the waters by introducing additional licences to the UK Government Licensing Framework.

Back to the Geography Data licence:

You may only use the Data for your own internal business use, that is use of Data for the internal administration and operation of your business and not for any commercial purpose, and not for financial profit or gain. Financial gain would include any profit whether direct or indirect, or benefit from the use or publication of the Data in any form.

This tells us that when the licence was written there was no intention to automatically allow full commercial re-use of the Geography Data, as would be permitted under the OGL.

So this addendum:

For any other use of the Data you will need to obtain permission from Ordnance Survey

The remainder of the licence is boilerplate to emphasise key terms from the Click-Use Licence.

Doesn’t look much like open data, does it?

This is why we shouldn’t always take data licences at face value. Licences are contracts, and subject to argument and interpretation.

In this case a literal reading of the licence would prevent anyone from actually re-using the data even for internal business purposes, because the Click-Use Licence is no longer available. It’s a reasonable assumption this is not the intention either of the Scottish Government or the Ordnance Survey. So there’s something wrong with the licence.

This is where it helps to understand the historical context, and the data specifications themselves. Reading the background information gives us some idea of which Ordnance Survey data was used to produce the Data Zones and, although it’s not conclusive, it looks as if it was simply older versions of data that is now included in Ordnance Survey’s open Boundary-Line product.

I did the sensible thing and sought an opinion from the Ordnance Survey. I had to persist a little but in due course received a detailed and helpful e-mail.

The key points from the Ordnance Survey e-mail are as follows:

Although we are unable to confirm which Ordnance Survey data has been used to create this data, we are happy to treat this data as being part of OS Boundary-Line and covered by the Open Data Licence.

As there is additional data present, we would always recommend that you seek permission of the map’s owner (in this case SNS) to ensure that there is no additional copyright enforced by them. It would appear that, as the link provided within SNS’ download licence (which refers to “Click-Use PSI Licence) is now for the Open Government Licence, the data is covered by the Open Government Licence …

On the strength of the above I would personally be confident in treating the SNS Geography Data as open data. I think Ordnance Survey permission to apply the OS OpenData Licence was the only real barrier.

It is significant that the Ordnance Survey is willing to apply the OS OpenData Licence to older versions of Boundary-Line, and presumably also to older versions of the other OS OpenData products. That makes practical sense but I don’t think it is stated explicitly on the OS website.

As regards “additional data present”, it would certainly be better to have a re-written licence on the download page itself to remove any remaining ambiguity. However based on the existing wording, and with due regard to the remarks in the OS’s e-mail, I think there is a perfectly defensible argument that the Scottish Government has left the full licensing decision to Ordnance Survey.


Update 11 July 2013:

The Scottish Government has now updated the Data Zones 2001 metadata record on to confirm that the dataset is re-usable under the OS OpenData licence.


Page 1 of 2