a blog by Owen Boswarva

Posts tagged Ordnance Survey

1 Oct

Post: 1 October 2014

Earlier this week Heather Savory, chair of the Open Data User Group (ODUG), argued in a post that Ordnance Survey’s model for licensing of geographic data is a “fundamental barrier” to maximising the beneficial use of publicly owned data in Great Britain.

Ordnance Survey does publish some very useful open data products, but its most detailed and useful geographic data is still available only on commercial terms (with special arrangements for public sector organisations).

ODUG is calling on the Government to push OS to open up most of (or all) of its data, in order to promote competition and develop the wider information market. I very much support this call.

In fairness, Ordnance Survey has done a pretty good job promoting reuse of its existing open data since the launch in 2010. A preliminary economic study prepared in 2012 estimated the OS OpenData programme would generate net GDP growth of between £13.0m and £28.5m per year by 2016. A recent project to add open data to the popular Minecraft game has had favourable publicity and, more importantly, helped introduce open geographic data to a wider, younger audience.

OS OpenData is a success story — and it is time to build on that success. But Ordnance Survey is constrained by the trading fund model, which requires it to put revenue generation ahead of the wider interests of Britain’s information economy. As in 2010, moving forward will require political support at ministerial level.

National Information Infrastructure

ODUG and the Cabinet Office are currently drawing up a list of datasets that form the National Information Infrastructure. The first draft, released in October last year, was rather uneven — but the decision to include all of Ordnance Survey’s geographic data was both sensible and obvious. OS data is not only essential to the NII — it is in many respects the foundation.

One of the key concepts behind the NII is the idea that the utility of public data is amplified by interactions between datasets. The more datasets are released as open data, the more potential there is to link those datasets in new and innovative ways.

In her post Heather Savory uses the example of the flood data Environment Agency is planning to release next year (perhaps sooner). She suggests analysis of that flood data would be more productive if we also had more detailed open data from Ordnance Survey.

At the moment OS OpenData gives us postcode centroids (in Code-Point Open) that we could use with EA flood data. But what if we also had coordinates of individual addresses, or even building outlines?

Maximising Reuse of Open Data: An Illustration

It’s always difficult to anticipate all the ways in which open data might be reused. However I would like to expand on Heather’s example, drawing on my own experience using EA flood data and OS data together in a business context. Below is a (simplified) illustration of the difference that better OS data can make to the public understanding of flood risk.

The main flood dataset Environment Agency intends to release as open data is called Risk of Flooding from Rivers and Sea (RoFRS), although it was until recently better known as the National Flood Risk Assessment (NaFRA). This dataset is an output from modelling the EA uses to prioritise investment in flood defences. RoFRS is also the source of flood risk information used mostly widely in the UK insurance industry.

RoFRS describes the likelihood of flooding at any given location, using four categories of risk (High, Medium, Low and Very Low) averaged across a grid of 50m x 50m cells. At high level, on a map, RoFRS looks like this:


Drill down to the level of an individual postcode and the data looks like this:


As you can see the data is kind of blocky at this resolution. Like any flood risk model RoFRS is only “indicative”, particularly when laid over small areas of geography. (It’s generally difficult to produce a definitive view of flood risk at address level without a site survey.) But indicative is still useful for many purposes, and better than nothing even when thinking about the risk to individual properties.

So how do we do that? The EA flood data gives us an idea of the relative likelihood of flooding for areas of land, but we need to geolocate our addresses (the flood risk “receptors”) as well. This is where the Ordnance Survey data comes in. 

At the moment, if we rely only on OS OpenData, all we have is the postcode centroids available in Code-Point Open. In the example above that centroid is on the edge of a High likelihood grid cell. We know the addresses themselves are in the vicinity of that point, but not their precise coordinates.

So, to be prudent, we might assume all of the addresses in that postcode are at a High risk of flooding. (There are techniques we can use, such as buffering, to get a more nuanced view even without address points. But that takes additional expertise.)

Here’s a better option:


If we had a polygon drawn around all of the address points in the postcode, as above, we could use GIS techniques to average the likelihood of flooding within that polygon. That would give us a better representation of the risk within the postcode, even without geolocating the individual addresses.

To do this we would need OS’s Code-Point with polygons product, which is not currently open data.

This isn’t a great approach anyway, because much of the space within the polygon is road rather than properties — and in a rural postcode properties might be few and far between. But it’s a better approach than relying only on the postcode centroid.

Or here’s an even better option:


Here we have the address points themselves. Ordnance Survey produces a number of geocoded address products, the most comprehensive of which is OS AddressBase Premium. None of those products are currently available as open data.

As you can see above, using the address points allows us to differentiate the indicative level of flood risk by individual address, rather than lumping the properties all together within the postcode. Of course we wouldn’t be confident of these results if buying a house, but for mechanical analysis of a large portfolio of properties this is a respectable approach.

But … there’s an even better option:


Here we have building outlines, in addition to address points, for the individual properties. As you can see, in some instances the address point is outside the flood cell but part of the property is within the flood cell. That gives us a more detailed understand of the likelihood of flooding at each address. (Though bear in mind we are really pushing the resolution of the flood model here.)

We don’t have the building outlines as open data. They are included within the Topography Layer of OS MasterMap, Ordnance Survey most detailed and comprehensive mapping product.

Beyond Addresses - Other Contexts

In the above I’ve mentioned various Ordnance Survey data products that might be useful for analysing flood risk in a residential postcode. However OS publishes many other datasets, such as the OS MasterMap ITN Layer and the OS MasterMap Sites Layer, which would be similarly useful if analysing flood risk to transport infrastructure, commercial properties, etc. And risk to property is only one context in which OS data can amplify the utility of flood data; there are applications in reducing risk to life and promoting community resilience as well.

Disclaimer: The above is an illustration only. Proper analysis of flood risk, even in the jejune environment of the insurance industry, is a bit more complicated. If you’re a hydrologist please don’t make fun of me.

Image credits: The images in this post may contain Ordnance Survey data and/or Environment Agency data or derivations thereof. Or maybe I just drew all the lines with a ruler. Who knows? Derived data is confusing.

17 Jul

Post: 17 July 2014

The Public Data Group is a collection of four data-rich government organisations (Companies House, Land Registry, Met Office and Ordnance Survey) that report to the UK’s Department for Business, Innovation & Skills (BIS).

The PDG organisations are trading funds, encouraged by Government to generate commercial revenue from the data assets that they control. However all four organisations make available at least some data under an open licence.

On Tuesday the Public Data Group issued a statement containing some commitments to future releases of open data. The statement is a bit short on detail, so this post is an attempt to add some context to the planned releases.


Companies House

The headline announcement in the PDG statement is the decision that Companies House will “make all of its digital data available free of charge”, from the second quarter of 2015 (April - June).

This follows previous initiatives to open up Companies House data:

The current Companies House Price List is online. My interpretation of Tuesday’s announcement is that Companies House will, at minimum, remove the £1 charge for access to individual Company Records via WebCHeck. “Electronic images” will also be free, which I think means PDF copies of the records.

The important unanswered question is whether this release will also include any new bulk downloads of data. According to the Open Definition, a dataset is only properly open data if it is available in bulk.

Bulk release is necessary for any kind of serious analysis of companies data. If the Government is serious about leveraging the free availability of Companies House data to “boost the UK economy”, then bulk release is essential.

Land Registry

According to the PDG statement Land Registry “will release their Price Paid Data for commercially owned properties for free by March 2015.”

There are no further details provided. It will be interesting to see what data is contained in that release. Land Registry currently makes available its Price Paid Data for residential transactions back to 1995, as open data. However it does not publish any statistics on commercial transactions. My past understanding was that Land Registry did not maintain a separate dataset for commercial sales (of either land or properties).

Update 9 September 2014: see this new Land Registry material on commercial and corporate ownership data. ]

There are several open questions: How complete or extensive is Land Registry’s data on prices paid for commercially owned properties? Does Land Registry intend to release data on historical as well as new transactions (bearing in mind that the initial release of residential Price Paid Data was only new transactions)? And is there likely to be resistance from commercial property owners to the open publication of sale prices?

The PDG statement also says that in 2014/15 Land Registry intends to “make the whole Index Map polygon layer covering England and Wales available at a cost recovery price.”

This is not an open data release, of course. The news may be welcomed by Land Registry licensees, but it remains to be seen how much of a saving they will realise. “Cost recovery price” should not be confused with “marginal price”. Index Map polygons are based on Ordnance Survey spatial data, so (as we have seen with the INSPIRE Index Polygons) it will be OS pricing that determines the actual cost of reuse.

Met Office

There is not much open data on the horizon from Met Office. However the PDG statement says Met Office is creating something called the “National Archive for the Nations Memory of the Weather”, and that “a selection of this will be available as Open Data.”

This could be significant or not, depending on what Met Office decides to release. Data required to maintain the nation’s “memory of the weather” could range from detailed weather observations, to historical documents, to anecdotal information about notable weather events.

Ordnance Survey - enhancements to OS OpenData

Ordnance Survey’s open data programme is more extensive than those of the other trading funds, as it was launched under the previous Government. Tuesday’s PDG statement notes several future developments.

The existing OS Street View product (raster base-mapping) will be enhanced with new features added such as car parks, major paths, major cycle routes and hill.

OS will release an “enhanced Gazetteer”. This is presumably the “Gazetteer of Great Britain" that OS demonstrated at GeoBusiness 2014 in May. I have seen some sample data for this product (via the OS Insight developer programme); it looks like a useful addition to the OS OpenData suite.

Ordnance Survey - Public Rights of Way

OS will be “working with” Defra to “provide consultancy, technology and to enhance public access (through a portal) to Rights of Way data”. I guess this is good news, though lack of a portal is not the main blockage to release of Public Rights of Way data.

I’ve written about PRoW data before. The main problem is that there has been no organised effort from central government to encourage local councils to release the data. OS is part of that problem (and therefore has to be part of the solution) because most councils use OS data to maintain their Rights of Way maps and need OS’s permission to release those maps as open data. However DCLG and the Local Government Association should be pushing this harder as well. The ideal would be publication of an open national PRoW dataset, collated from the many local sources.

We already have a pretty good portal for the 80 or so local PRoW datasets already available as open data: Barry Cornelius’s Rowmaps site.

Ordnance Survey - Derived River Network

Of the various open data commitments mentioned in the PDG statement, this is the one I am personally most excited about: Ordnance Survey plans to release a new “Derived River Network” open data product.

My understanding is that this dataset will be derived from the new Water Layer in MasterMap. (The other relevant dataset in this space, the Detailed River Network that OS developed with the Environment Agency, is being deprecated.)

The River Network is a dataset that I have been campaigning for since 2012, via the ODUG process and other channels such as the Defra Transparency Panel.

Since then more spatial data about our rivers has become available, most recently the Cycle 2 draft of the EA’s Water Framework Directive (WFD) River Waterbodies dataset (last month’s post). However I am gratified to see that there is now sufficient support for open release of a good general-purpose vector map of the river network.

The Derived River Network dataset will complement the Environment Agency’s recent open data release of live flood warning and river level feeds as well as plans to release the NaFRA national flood risk dataset.

Photo credit: Open_Data_stickers.jpg by Jonathan Gray (CC0 1.0). It’s an iconic image and I was too lazy to find something more imaginative to illustrate this post.

5 Dec

Post: 5 December 2013

Update 2 January 2014: I’ve received a response from ONS by e-mail. Please see the end of the post.

Which open data licence applies to ONS’s Postcode Directory: the Open Government Licence or the OS OpenData Licence?

Yesterday SK53, a prolific OpenStreetMap contributor, released a blog post on the subject of British postcodes. He made an argument for increasing the coverage of GB postcode information on OSM, noting that:

In 2010 postcode centroids were made available through the Ordnance Survey Open Data scheme, under the brand CodePoint Open. Subsequently it was found that the license associated with this data prevented it being used directly in OSM. More recently the Office of National Statistics have released an (identical) data set which is not encumbered by the license of CodePoint Open.

I’m not an OpenStreetMap contributor myself but I’ve been aware for some time that Ordnance Survey’s OS OpenData Licence presents difficulties for OSM. Although the OS OpenData Licence complies with the Open Definition and is in most respects the same as the Open Government Licence, it does contain some additional requirements on attribution and sub-licensing.

If the Office for National Statistics is now providing the same GB postcode data as Ordnance Survey, but under the Open Government Licence, that seems like a welcome resolution to the OSM problem.

But is that really the licensing position?


Code-Point Open vs the ONS Postcode Directory

At the moment there are two primary sources for the authoritative national geocoded postcode dataset. The first is the Code-Point Open download available from the Ordnance Survey website. The second is the series of Postcode Directory (ONSPD) downloads available from ONS’s Open Geography portal.

Most users will find the ONS portal to be the preferable source. The ONSPD downloads contain all of the Code-Point Open data, plus additional ONS fields. The ONS downloads are immediately available, whereas the Ordnance Survey supply is via a link sent to the requester’s e-mail address. The ONS downloads are provided in a choice of formats and, most importantly, ONS maintains an archive of previous releases rather than just the most recent.

If the ONS downloads are subject to the Open Government Licence, and not the OS OpenData Licence, that is another good reason to procure the data from ONS rather than from Ordnance Survey.

Postcodes and postcode coordinates are reference data, which means they are often incorporated in other data products that contain data from multiple sources. Leaving aside the particular issues for OSM, it’s always easier to build products from the most broadly re-usable data sources. The OGL is more widely used and understood and (slightly) less restrictive than the OS OpenData Licence, so it better supports the development of added-value data products.

Lack of clarity on licensing of ONS postcode datasets

The problem is that ONS is providing conflicting information about licensing for re-use of the Postcode Directory.

The Licences page on the main ONS website includes the following:

The ONS postcode products are subject to the Open Government Licence, and Ordnance Survey OpenData Licence.

That page predates the launch of the Open Geography portal. Licensing information on the Open Geography portal itself is not very well signposted; however the FAQs page includes a link back to the Licences page on the main site.

There is also licensing information included with the ONS Postcode Directory (ONSPD) downloads themselves, and that is the source of confusion. The version notes contain the following:

Copyright and reproduction

© Crown copyright 2013

You may re-use this information (not including logos) free of charge in any format or medium, under the terms of the Open Government Licence. However, the following attribution statements must be acknowledged or displayed on any product using ONS data:

‘Contains Ordnance Survey data © Crown copyright and database right 2013’

‘Contains National Statistics data © Crown copyright and database right 2013’

'Contains Royal Mail data © Royal Mail copyright and database right 2013'

Click here to view the Open Government Licence, or write to the Information Policy Team, The National Archives, Kew, London TW9 4DU

That statement confirms that the dataset includes Ordnance Survey and Royal Mail intellectual property, but the attribution requirements only go as far as those in the Open Government Licence. There is no explicit reference to the OS OpenData Licence or to the additional terms in that licence.

The same wording appears in the notes for previous versions of the ONSPD available from the Open Geography Portal.

The ONSPD downloads also include a metadata file, which does include the URL of the OS OpenData Licence:

Contains Ordnance Survey data © Crown copyright and database right [2013]

However the metadata file is in XML format, which means some re-users of the data (particularly those who download the ONSPD as a ready-to-use Access database) will likely not notice it.

What is the actual licensing position?

It’s possible that Ordnance Survey and Royal Mail have agreed that their data may be re-used under the OGL when it is released as part of the ONS Postcode Directory. However if that is the case it’s not clear why OS has not also released Code-Point Open under OGL.

I think it is more likely that the ONSPD version notes are in error, and that the copyright section should include a reference to the OS OpenData Licence.

Where does that leave re-users? The OGL and OS OpenData are both open data licences, and the differences do not really affect the scope for direct re-use of the datasets. The potential for error lies mostly with businesses who may have incorporated ONSPD data in their own products, with the risk that downstream attribution does not adequately reflect the requirements of OS and Royal Mail.

Open data licences do not normally indemnify the licensee against a breach of third party IP rights. However any licensee who has relied on the ONS version notes in error will have done so in good faith. Harm to OS and Royal Mail is likely to be trivial, and given that both organisations work closely with ONS it could be argued that they have been insufficiently vigilant in not requiring a correction to the release notes.

However for OpenStreetMap there might be practical consequences. If contributors are encouraged to add ONSPD data onto OSM, and ONS subsequently clarifies that the OS OpenData Licence will apply going forward, OSM may have to take remedial action to stop the addition of further such data.

Update (2 January 2014)

I have today received an e-mail from ONS confirming that the version notes for the ONS Postcode Directory should refer to the OS OpenData Licence terms:

I’ve recently seen your blog (dated 5 December) regarding licensing of our ONSPD and the discrepancy between the Open Government Licence and OS OpenData Licence terms. You are quite correct; thank you for drawing it to our attention. There is an inconsistency between our Licence web page and information provided with the metadata. We will correct the metadata so that it meets the full requirements of OS Opendata. The Version Notes will also contain a statement drawing customers’ attention to the inconsistency and outline the correct position.

This action has been agreed with Ordnance Survey.

30 Aug

Post: 30 August 2013

So there’s this:

Assessing the Value of Ordnance Survey OpenData to the Economy of Great Britain: Full Interim Report

I received the above report yesterday in response to a Freedom of Information request that I made to Ordnance Survey last month.

The report is an economic value study of Ordnance Survey’s OS OpenData suite of free geographic data products, prepared in November 2012 by ConsultingWhere and ACIL Tasman.

Ordnance Survey published their own synopsis of the interim report in June, following an earlier FOI request. (Although the report is “interim”, plans to deliver a final version have been shelved. More on that below.)


New and Missing Information

There are a substantial number of redactions in the released copy of the full report. You can read the FOI correspondence if you’re interested in the detail of which exemptions Ordnance Survey have applied. In a nutshell: names of individuals who are not civil servants have been blacked out (as personal information), case studies including the names of participating organisations have been blacked out (as information provided in confidence), and the internal breakdown of the amount Ordnance Survey receives under the OS OpenData contract has been blacked out (as commercially sensitive).

There are very few unredacted figures in the report that were not previously disclosed in the synopsis. The main exception seems to be the revenue estimate of £ 5.3 million per annum used as the basis of the “counterfactual” case, i.e. the income anticipated for Ordnance Survey if the OS OpenData products were fee-generating rather than free. The footnote giving the source of this estimate is, however, redacted. (See section 6.2 on page 27.)

I regret that Ordnance Survey have declined to release the breakdown of the £ 20 million per annum they receive under the “CLG contract” as compensation for losses of revenue consequent upon the decision to make the OS OpenData products free at the point of delivery. It is difficult to see how Ordnance Survey’s commercial position would be harmed by releasing the figures, and withholding them has the effect of shielding the arrangements from scrutiny. The report makes no comment on the validity of the breakdown, and indeed says rather pointedly:

we should note that the CLG contract is taken as a matter of fact. We are not in a position to comment on the level of the funding, nor the split of revenues between public and private sectors.

No Final Report

The original plan was to follow up this interim report with a final report:

For the final report, it was envisaged that the findings would be summarised to be accessible to a wider audience and the interim conclusions updated through a process of wider consultation.

However Ordnance Survey’s synopsis says:

The planned Final Report – this was intended to be a summary of the findings that would be drafted so as to be accessible to a wider audience and to enable the further updating of conclusions through wider consultation. This stage has been superseded by this Synopsis of the Interim Report.

There is a bit of a discrepancy here. Was the final report always going to be simply a summary of the findings, to “enable” wider consultation, or was the original intention to consult more widely and include updated conclusions in the final report itself? There must have been more to the plan for a final report than simply a releasable summary; else Ordnance Survey’s synopsis would have been called the final report.

The charitable explanation is that Ordnance Survey and/or BIS saw no need for a further report because the findings in the interim report were sufficient to justify OS OpenData as an ongoing economic proposition. The report’s topline conclusion is that OS OpenData will generate net growth in GDP of between £ 13.0 million and £ 28.5 million per annum by 2016 — and this estimate is “almost certainly understated” (page 1).

Why then did Ordnance Survey and BIS sit on the report for a year and a half, publishing the synopsis only after I submitted my first FOI request? Based on these findings, OS OpenData is a clear-cut success story — why is Government not holding it up as a model for future releases of economically useful public data?

We can at least speculate. Is the report too robust in its argument for open data? Perhaps the success of OS OpenData presents the “threat of a good example”? If the study’s methodology gained general acceptance it would be difficult for Government to explain why it has not pressed ahead more aggressively with release of other core reference data.

The current Government’s transparency policy has focused mainly on release of spending and performance data. Their target audience is mostly imaginary “armchair auditors” who want to bash public services improve public services and hold public authorities to account. The OS OpenData products, on the other hand, are datasets that people actually want to use. And of course (as the report makes clear but the synopsis version does not) the underpinning economic rationale for OS OpenData and UK open data policy more generally was developed by the previous Labour administration. In contrast this Government has shown little understanding of open data’s wider economic potential; its only big idea has been to open a crèche for start-ups. Most of the economic growth the report attributes to OS OpenData is from business efficiencies, not innovation or “app dev”, so the findings are not exactly on-message.

The Economics of Open Data

Despite all the redactions (including a full ten pages of case study material) the report provides a substantially more balanced perspective on the arguments for OS OpenData than does the synopsis. Most of the value is in the discussion of the study methodology and different approaches to modelling the impacts of open data. The numbers themselves I take with a grain of salt, on the “garbage in garbage out” principle; whatever the merits of the model, there’s only so much you can do with download statistics and anecdotal information from interviews. (The assumptions behind the download analysis are particularly loopy.)

The report itself is far more readable than Ordnance Survey’s synopsis, but the most stark difference is in the tone. That topline GDP estimate is buried halfway through the synopsis rather than highlighted in the executive summary. The report’s recommendations are presented verbatim in the synopsis — but there is no indication from Ordnance Survey or BIS whether they have taken any actions on the recommendations.

The report includes (as Appendix B) a copy of the peer review from economic consultants Prabhat Vaze and Patricia Seex secured to ensure the validity of the report’s approach. The synopsis, on the other hand, ignores the contents of that review as well as the report’s explanation of how the points from the review were incorporated into the methodology. It concludes instead with a two-page letter from BIS’s Chief Analyst that rubbishes the report’s findings based on inherent difficulties in information-gathering that must surely have been obvious when the research was commissioned. It is difficult to escape the impression that BIS finds the success of OS OpenData altogether rather awkward.

Does OS OpenData Have a Future?

Despite what seem to be attempts to downplay the findings, this economic value study should make it difficult for BIS to withdraw support for the OS OpenData initiative. However it does highlight a couple of dependencies of which re-users should be aware.

The first is the “CLG contract”, i.e. the £ 20 million per annum that Ordnance Survey receives from Government in compensation for loss of revenue and to fund delivery of the initiative. The terms of that contract are not public, so it is difficult to judge how reliable a basis it provides for continuation of the OS OpenData initiative.

The second dependency is the royalties payable to Royal Mail for use of data embedded in OS OpenData products. Those royalties are provided for in the CLG contract (page 29 in the report), and we may hope Royal Mail remains satisfied with the arrangement. However Royal Mail is headed for privatisation and could take a different position in future, particularly if it decides to develop data products that compete with Ordnance Survey’s.


The OS Vector MapDistrict image used in this post contains Ordnance Survey data © Crown copyright and database right 2013.

27 Mar

Post: 27 March 2013

Update Apr 23: cancellation of Pinpoint has now also been confirmed on the Royal Mail website.

According to a post on the Allies Computing website, Royal Mail have confirmed they will not proceed with their controversial Pinpoint positional data capture project. 

Andrea Martin, Managing Director of Data Services for Royal Mail, is quoted:

"Royal Mail announced in the summer of 2012 a pilot initiative in East Anglia to map the co-ordinates of home and business. The pilot explored the potential for Royal Mail to support the location-based information marketplace. Following the completion and full review of the pilot, we have decided not to progress the initiative."

Image source: Harrow Council

I’ve posted previously about the origins and progress of the Pinpoint project. If successful Pinpoint would have added geographic coordinates to Royal Mail’s existing Postcode Address File (PAF) dataset.

It’s currently unclear why Pinpoint has been cancelled, though Royal Mail’s attempt to enter the market for geocoded address data had raised eyebrows within Britain’s geographic information community.

Pinpoint would have provided competition for Ordnance Survey’s well-regarded but rather pricey AddressBase products. However the future shape of the address data market is uncertain, with Royal Mail privatisation on the cards and both Royal Mail and Ordnance Survey under pressure to unlock their address data assets as part of an “open data” National Address Dataset.

Royal Mail had reportedly allocated an investment of £10 million or more to the Pinpoint initiative.

Page 1 of 2