mapgubbins

Update Apr 23: cancellation of Pinpoint has now also been confirmed on the Royal Mail website.

According to a post on the Allies Computing website, Royal Mail have confirmed they will not proceed with their controversial Pinpoint positional data capture project. 

Andrea Martin, Managing Director of Data Services for Royal Mail, is quoted:

“Royal Mail announced in the summer of 2012 a pilot initiative in East Anglia to map the co-ordinates of home and business. The pilot explored the potential for Royal Mail to support the location-based information marketplace. Following the completion and full review of the pilot, we have decided not to progress the initiative.”

I’ve posted previously about the origins and progress of the Pinpoint project. If successful Pinpoint would have added geographic coordinates to Royal Mail’s existing Postcode Address File (PAF) dataset.

It’s currently unclear why Pinpoint has been cancelled, though Royal Mail’s attempt to enter the market for geocoded address data had raised eyebrows within Britain’s geographic information community.

Pinpoint would have provided competition for Ordnance Survey’s well-regarded but rather pricey AddressBase products. However the future shape of the address data market is uncertain, with Royal Mail privatisation on the cards and both Royal Mail and Ordnance Survey under pressure to unlock their address data assets as part of an “open data” National Address Dataset.

Royal Mail had reportedly allocated an investment of £10 million or more to the Pinpoint initiative.

Data licensing can be tricky.

There are data sets that are clearly open data and data sets that are clearly not. 

Then there are those that have potential, if you look at them in the proper light …

The Indices of Multiple Deprivation (IMD) are a multi-domain measure of relative levels of deprivation within small geographic areas of the UK.

Indices for England, Scotland, Wales and Northern Ireland are produced separately by Government and have slightly difficult criteria, but it’s possible to make comparisons across the data for different countries.

IMD data sets are used widely in the public sector and for academic research. I’ve used IMD data myself as inputs into risk models for the insurance industry. The crime domain in particular is useful in predicting geographic variability of economic losses to theft, civil disorder, etc.

Last month the 2012 version of the Scottish IMD was released by the Scottish Government. Alasdair Rae has put together a useful website that visualises the SIMD 2012 data interactively on a map.

In England and Wales the small-area geography used in the IMD is the Lower Super Output Area. LSOA boundaries are unambiguously available as open data, and may be downloaded from the ONS website and elsewhere.

However in Scotland the unit of geography underlying the IMD is called a Data Zone. The Data Zone boundaries currently in use were produced in 2004 but built up from 2001 Census output areas.

There’s background information on Data Zones here and here. This is what they look like:

Pretty much all administrative geography for England, Scotland and Wales is now available as open data. So I was surprised to see the restrictive licensing terms presented on the download page for the Scottish Government’s Geography Data (which includes the Data Zone boundaries).

You can read the licence on the Scottish Neighbourhood Statistics website. (Click on Go to Data Download, then Download Geography. The SNS site doesn’t like direct links.)

The Geography Data licence starts like this:

We the Scottish Government grant you a non-exclusive non-transferable licence (without the right to sublicense) to copy and use the Data which is derived from Ordnance Survey data and as such is subject to the terms and conditions of the licence agreement between The Scottish Government and Ordnance Survey.

No transferability or sub-licensing, so right away that says it’s not an open licence. Without transferability it would be very difficult to put the Geography Data on a public website in a legally compliant manner.

The wording tells us clearly that the download includes Ordnance Survey derived data. However it’s ambiguous whether the Scottish Government is claiming any additional database rights.

The licence then says:

For any use of this material a Click-Use PSI Licence is also required.

with details of how to arrange that via the Office of Public Sector Information.

The problem is that the Click-Use Licence system was phased out from 2010, when the Open Government Licence (OGL) was introduced. The link provided on the SNS’s download page now resolves to a National Archives page about the OGL.

Click-Use Licences were typically issued for a five-year period, so in theory there could be re-users out there using the SNS Geography Data under those terms. However the OPSI is no longer issuing new Click-Use Licences.

Interpreted literally that means nobody who downloads the Geography Data subsequently can comply with the licensing terms on the download page.

It’s fairly obvious the licence on the download page is out of date. Does that mean we can substitute the terms of the Open Government Licence? That would enable us to do away with the restrictions on transferability and sub-licensing, both of which are supported by the OGL.

There is certainly a convention that the OGL simply replaces the Click-Use Licence, and we have statements to that effect from National Archives.

But I’ve never been entirely comfortable that the Government nailed that down properly, and there are substantial differences between the Click-Use Licence and the OGL. The Click-Use Licence does not specifically set out the scope of permitted re-use, and it seems to exclude mapping data. National Archives has also muddied the waters by introducing additional licences to the UK Government Licensing Framework.

Back to the Geography Data licence:

You may only use the Data for your own internal business use, that is use of Data for the internal administration and operation of your business and not for any commercial purpose, and not for financial profit or gain. Financial gain would include any profit whether direct or indirect, or benefit from the use or publication of the Data in any form.

This tells us that when the licence was written there was no intention to automatically allow full commercial re-use of the Geography Data, as would be permitted under the OGL.

So this addendum:

For any other use of the Data you will need to obtain permission from Ordnance Survey

The remainder of the licence is boilerplate to emphasise key terms from the Click-Use Licence.

Doesn’t look much like open data, does it?

This is why we shouldn’t always take data licences at face value. Licences are contracts, and subject to argument and interpretation.

In this case a literal reading of the licence would prevent anyone from actually re-using the data even for internal business purposes, because the Click-Use Licence is no longer available. It’s a reasonable assumption this is not the intention either of the Scottish Government or the Ordnance Survey. So there’s something wrong with the licence.

This is where it helps to understand the historical context, and the data specifications themselves. Reading the background information gives us some idea of which Ordnance Survey data was used to produce the Data Zones and, although it’s not conclusive, it looks as if it was simply older versions of data that is now included in Ordnance Survey’s open Boundary-Line product.

I did the sensible thing and sought an opinion from the Ordnance Survey. I had to persist a little but in due course received a detailed and helpful e-mail.

The key points from the Ordnance Survey e-mail are as follows:

Although we are unable to confirm which Ordnance Survey data has been used to create this data, we are happy to treat this data as being part of OS Boundary-Line and covered by the Open Data Licence.

As there is additional data present, we would always recommend that you seek permission of the map’s owner (in this case SNS) to ensure that there is no additional copyright enforced by them. It would appear that, as the link provided within SNS’ download licence (which refers to “Click-Use PSI Licence) is now for the Open Government Licence, the data is covered by the Open Government Licence …

On the strength of the above I would personally be confident in treating the SNS Geography Data as open data. I think Ordnance Survey permission to apply the OS OpenData Licence was the only real barrier.

It is significant that the Ordnance Survey is willing to apply the OS OpenData Licence to older versions of Boundary-Line, and presumably also to older versions of the other OS OpenData products. That makes practical sense but I don’t think it is stated explicitly on the OS website.

As regards “additional data present”, it would certainly be better to have a re-written licence on the download page itself to remove any remaining ambiguity. However based on the existing wording, and with due regard to the remarks in the OS’s e-mail, I think there is a perfectly defensible argument that the Scottish Government has left the full licensing decision to Ordnance Survey.

Why is there no open national map of Public Rights of Way in England and Wales? Or to put the question another way, why isn’t the information that local councils maintain on Public Rights of Way freely available for re-use?

This post is specifically about open data release of vector mapping data for Public Rights of Way (PRoW) maintained by local surveying authorities. The statutory basis and policy background to PRoW information is quite complicated so I’m only going to sketch it in. However there are some links at the end of the post if you want to read into the subject further.

Surveying authorities

In England and Wales public rights of way are routes on which the public has a legally protected right to pass, on foot and sometimes by other modes of travel as well.

Surveying authorities (usually county councils or unitary authorities) are required by law to maintain a Definitive Map of public rights of way in their local areas. Each right of way also has a written description called a Definitive Statement.

Definitive Maps have always been available to the public for inspection at local offices of surveying authorities. However these days most authorities also put their PRoW information on the web. Some deliver this information via interactive mapping applications, others simply as raster maps or in PDF documents.

Whether on paper or online, surveying authorities usually present their Definitive Map information on Ordnance Survey background mapping.

Keeping Definitive Maps up to data is an area of considerable activity for surveying authorities. There is an ongoing effort to capture all unrecorded footpaths and bridleways created before 1949, because the Countryside and Rights of Way Act 2000 mandates that these cannot be recorded after 2025. DeFRA has recently consulted on new proposals to modernise the recording of rights of way.

Ordnance Survey use of PRoW data

Ordnance Survey began to add rights of way information to its maps back in 1960. Rights of way are now consistently shown on Ordnance Survey 1:25,000 (Explorer) maps and 1:50,000 (Landranger) maps, and in OS digital raster mapping products of equivalent or larger scale.

Although surveying authorities provide their PRoW information to Ordnance Survey for use on its maps, the Definitive Map and Definitive Statements held locally remain the authoritative record for statutory purposes.

When the previous Government prevailed upon Ordnance Survey to release its OS OpenData suite of products in April 2010, one of the main grumbles raised by digital mapping enthusiasts was the failure to include any Public Rights of Way data.

There was an excellent discussion of this issue on the Ordnance Survey’s blog at the time, in which Ordnance Survey staff did a very good job explaining the related problems.

Intellectual property rights and the derived data issue

In a nutshell there were two barriers. First, the main IP rights for Definitive Map data are held by the local surveying authorities, because they originate it and have statutory responsibility to maintain it. Second, most (and quite possibly all) surveying authorities prepare their Definitive Map by drawing over features on Ordnance Survey maps. This makes some of the PRoW information “derived data” that relies on sub-licensing of Ordnance Survey’s IP rights. 

The upshot was that back in 2010 neither Ordnance Survey nor the surveying authorities were in a position to release PRoW mapping data for open re-use. However there was at least an awareness of public interest in open re-use of the data.

The 2011 Autumn Statement

Last year the Government’s Autumn Statement included a number of open data measures promoted by the Cabinet Office. One of these was a note that “Ordnance Survey has committed to amend its derived data restrictions on Local Authorities’ Public Rights of Way data, enabling this to be released more easily as Open Data.”

This implicitly put the onus on local authorities to release the data, once Ordnance Survey had cleared the way. This was one of two policy approaches the Cabinet Office could have taken. The other would have been to secure a collective commitment from local authorities to allow Ordnance Survey to release its own processed version of their data.

Ordnance Survey itself hasn’t made any public statement on how it has implemented the Autumn Statement commitment. However I gather from online discussions that it has been telling any local council inhibited by the derived data issue that they are now free to release their Definitive Map data for re-use under the OS OpenData Licence.

The OS OpenData Licence is the same licence that applies to Ordnance Survey’s own OpenData products. It basically incorporates the Open Government Licence (OGL) with additional attribution requirements to acknowledge Ordnance Survey as source of some of the data.

Recent open data releases of PRoW vector data

That’s basically where we are today. In principle the derived data issue is no longer a barrier. Any surveying authority that holds its PRoW Definitive Map as digitised vector data can simply provide that data as a download on its website.

The problem is that very few have done so. Honourable exceptions are Hampshire County Council and Devon County Council, both of which provide their data as downloads in KML and ESRI Shapefile formats. Hampshire specifies the OS OpenData Licence; clarification from Devon on licensing is pending but it’s reasonable to assume they will take the same line. Additionally Worcestershire County Council’s vector data is re-usable via WMS, also under the OS OpenData Licence.

UK volunteers who work on OpenStreetMap are keen to add PRoW vector map to their grand project, so long as the licensing can be properly cleared and documented. The OpenStreetMap Wiki includes a table of surveying authorities with notes on their view services and on availability for re-use of the Definitive Maps and Definitive Statements. (I am grateful to Rob Nickerson for bringing this to my attention.) 

However OSM has exacting standards, and represents only one potential type of re-use for the PRoW vector data. Clearly an approach based on engaging with surveying authorities individually is going to be slow going.

Next steps - where’s the follow-through?

If we want a comprehensive release of Public Rights of Way open data within the life of the current Government it’s going to require a push from the centre.

As I’ve remarked before, Francis Maude et al in the Cabinet Office seem to be better at making policy announcements on open data than they are on the mechanics of delivery. However I would hope PRoW data is an issue that would fit well on the agenda of the new Open Data User Group. 

As PRoW data is not separately productised by Ordnance Survey, and surveying authorities don’t seem to be deriving significant revenue from it, there should be no vested interests to undermine the business case for wider open data release.

Ideally I would like to see Ordnance Survey release the vector data it holds for Public Rights of Way either as a new product in its OS OpenData suite or as an enhancement to OS VectorMap District. This version of the data might not be “authoritative” to the standard of surveying authorities; but then the point of open data release is to maximise re-use of a data set for additional purposes rather than use for its original purpose.

The real practical question is whether Ordnance Survey actually holds PRoW data with full coverage in vector format. Back in April 2010 there were reportedly still some gaps.

My hope is that Ordnance Survey already maintains a discrete PRoW data set for its own production purposes. There is always a limit to how much additional preparatory work we can reasonably expect a public body to do in support of an open data release.

At the same time PRoW data is an issue that the Local Public Data Panel could perhaps take up. The LPDP could encourage more local authorities to release the data themselves, perhaps with best practice guidance. It could also organise local consents for Ordnance Survey to release the national data set, with suitable attribution of surveying authority copyrights.

_____________________________________________________

Update (Aug 29):  PSMA Exemption Process

Steven Campbell, Chair of the PSMA User Group, has pointed out that the Public Sector Mapping Agreement includes a formal exemption process, under which PSMA members can ask the Ordnance Survey for permission to release derived data under either OS OpenData or Free to Use terms. 

There’s a detailed guide to PSMA arrangements on the Ordnance Survey website. Most if not all surveying authorities should be PSMA members.

It seems to me that the Autumn Statement commitment is effectively equivalent to an informal but blanket application of the exemption process to Public Rights of Way data. If we could persuade Ordnance Survey to confirm that to PSMA members, ideally with a statement online that everybody could see, it would go some considerable way to clarifying the licensing position.

_____________________________________________________

Want to know more about Public Rights of Way?

Natural England publishes a guide to Definitive Maps and changes to public rights of way, which is free to download.

There’s a membership organisation for professionals involved in the management of public rights of way and access, IPROW. Their Good Practice Guide is full of information.

The PRoW information on the previous Government’s version of the DeFRA website lacks the more recent changes in policy and legislation but remains useful as background.

This Ordnance Survey Blog post from last year is also good: Everything you need to know about Rights of Way.

Subsequent to my post in March, a few additional details have emerged on Royal Mail’s Positional Data Capture project. The objective of this project is to collect location information for all Royal Mail delivery address points in the UK. 

Last week a Communication Workers Union (CWU) representative posted a Deployment Guidelines document on the RoyalMailChat online message board. (Free registration required to view the full message.)

Following is an edited extract of key points:

Branches may be aware of the previous trial regarding the Positional Data Capture (PDC) initiative which took place in London, Cardiff and Lampeter last year. PDC is part of the new products and services strategy and will seek to establish Royal Mail as a major player in the positional data market; currently worth £100m a year and growing. This information is commercially attractive to Royal Mail who intends to sell the data captured to a host of organisations and companies who have shown an interest in obtaining this information.

Royal Mail, as the Universal service Provider for the UK, has a significant advantage over any other delivery or logistics company in that it has access to 28.2 million address points and has data systems such as A plus which capture every post coded delivery point in the UK. This information will be used to overlay positional data information, such as the address point and the delivery access point using satellite technology GPRS.

National Objectives 

• To capture all address points and access points circa 28.2m
• To achieve data accuracy of c95% accurate to within <5m
• To achieve 60% positive experience from the volunteers involved in PDC covering training and use of the technology.
• To provide a commercially viable product and enhance Royal Mail’s portfolio of products and services

Locations

All Royal Mail Delivery Office locations will be covered.

Duration

The deployment will be divided into 2 phases, timeline highlighted below. 

Anglia = 02/07/12 – 24/09/12

Thames Valley, South East, South West, London, Wales, Midlands, North West England, North East England, Scotland and Northern Ireland = 03/09/12 – 30/06/13

The programme outlined above is still provisional and is subject to change.

In addition to the above information, Nigel Pindar (Solutions Architect on the PDC project from May 2011 to March 2012) has posted the following note to his LinkedIn profile. (I’ve added the vendor links.)

Positional Data Capture (PDC) is a new Royal Mail product-offering, providing hitherto unachievable levels of GPS data accuracy for the UK’s 28.8 million addresses. It uses GPRS or WiFi to upload data from mobile devices to a PHP/Java, Windows, Apache & MySQL based Web Portal hosted on a PaaS infrastructure provided by BT.

This is a £10m project which will enable RMG to move into a completely new market, providing GPS data to a variety of new customers.

I authored both the Logical Design and Physical Design documentation, successfully taking them through RMG’s gated project management process and achieving sign-off on them.

I am RMG’s solutions/technical lead liaising with vendors; Trimble (the application vendor), BT, CSC, AFD and Accenture; ensuring that the solution components they’re supplying will integrate successfully with each other.

I’m assuming the above description applies only to the technology and vendors involved in the project to capture the data, and not to any proposed customer solution. It remains unclear what services or products Royal Mail plans to offer to the location intelligence market once it has successfully captured and cleansed the data.

It will be interesting to see whether Royal Mail decides to focus simply on production of address data sets for use in third-party software, i.e. in direct competition with the National Address Gazetteer (NAG) and Ordnance Survey’s AddressBase products, or whether it also intends to offer the data as part of its own software/service proposition.

I’m very skeptical that this project will produce a national geo-referenced address data set with higher positional accuracy than the existing NAG and AddressBase products. Any large data set produced from a mass labour effort is likely to be of variable quality. However it’s certainly plausible that the Royal Mail data will be competitive in much of the market for this type of information.

A particular selling point for Royal Mail will be the inclusion of address location data for Northern Ireland, since the existing available products only cover Great Britain. On the other hand Royal Mail is only collecting location data for postal delivery points, and missing out the 1.5 million or so “objects without postal addresses” included in the Plus and Premium versions of AddressBase. My current best guess is that Royal Mail will present its positional data as an added-value alternative for existing PAF customers and as a lower-price alternative for commercial licensees of the basic version of AddressBase.

Is Royal Mail developing its own geo-referenced national address data set, to compete with National Address Gazetteer products such as Ordnance Survey’s AddressBase?

Following is an item from a recent e-bulletin for members of the Communications Workers Union (CWU):

Positional Data Capture Update

Royal Mail want to develop a new product called Positional Data Capture (PDC). This involves assigning an exact location to every address point in the country. Such information is likely to be extremely useful to mapping companies such as Google and for sat navs as it would be far more accurate than what is currently available. It is also likely to be worth a fair bit of money.

Royal Mail is the only company able to gather this information and, just as important, update it, as they are the only ones with access to every address point. The plan is to ask for between one and five OPGs [postal staff] from every DO [delivery office] to come off their duty for around three weeks and go round and capture the data using a piece of specially developed kit. Full training will be given to those who volunteer.

We are still waiting to agree the final details of how this will be done but the aim is to start the project in the Anglia Region at the beginning of July. Further information will be provided as it becomes available.

Last year Metaskil, a software services company based in Reading, revealed that it had provided project management resource for the initial phase of the project. However so far Royal Mail itself has made no public announcements.

The CWU item reads a little oddly as it seems to have been written by somebody who is unfamiliar with the existing National Address Gazetteer (NAG). Although the geo-referencing of postal delivery points in NAG isn’t perfect, it’s unlikely that a mass data capture exercise by postal staff will produce a higher quality product.

However an exercise of this type would give Royal Mail ownership of a standalone address product that it could market to customers in competition with the entry-level version of AddressBase.

Is there a market for another address data product?

The National Address Gazetteer itself is relatively new. When plans for NAG were announced in late 2010 it was with the express intention of creating “one definitive source of accurate spatial address data”. The idea was to move away from the duplication and confusion of the previous system under which local authorities and Ordnance Survey collected information separately.

When the Office of Fair Trading gave the go-ahead to NAG it recognised that the venture would create a monopoly in the market. However the OFT cited broad support from both public and private sector customers for the creation of a single database.

Local authorities, emergency services and other public bodies have comprehensive and affordable access to AddressBase under the Public Sector Mapping Agreement, so a product with fewer features from Royal Mail is unlikely to make much headway in that market.

However Ordnance Survey is vulnerable to competition in private sector markets, particularly among firms that require geo-referenced address data with national coverage. The eye-watering cost of corporate licences for AddressBase (starting at £129,950 + VAT per year) have so far been a barrier to take-up in the private sector.

In January a group of large UK companies wrote to Cabinet Office minister Francis Maude to protest the AddressBase pricing as “prohibitive for almost all potential customers”.

Royal Mail could pick up a lot of business, if it prices affordably and can demonstrate a reasonable level of data quality.

Outstanding questions

Will Royal Mail’s Positional Data Capture project produce a viable product? There are a number of unresolved questions that only Royal Mail can answer.

Royal Mail would be competing against itself to some extent. Royal Mail receives licensing revenue from AddressBase products, because they include data from Royal Mail’s Postcode Address File (PAF).

As long as the Shareholder Executive retains its influence over Royal Mail we can assume that PAF data will remain available for use in NAG and AddressBase. However the statutory requirement for Royal Mail to remain in public ownership was repealed last year. It’s possible that in future a privatised Royal Mail could improve the competitive position of its own data products by withdrawing PAF from NAG or at least negotiate higher licensing fees.

This project also suggests Royal Mail is fairly confident that Government has no intention of releasing NAG or AddressBase as ‘open data’ in the foreseeable future. The labour costs of capturing the coordinates of every postal delivery point must represent a substantial investment. I doubt Royal Mail would make that investment without some assurance from Government that it intends to maintain a commercial market for the data.

It’s also possible Royal Mail plans to go beyond simple geo-referencing of delivery points. The clever thing to do would be to use the same exercise to capture additional property attributes, such as building type and number of floors, for which there are currently no reliable sources of national data.