mapgubbins

Yesterday Land Registry announced that additional Price Paid Data will be released later this year under the Open Government Licence. Price Paid Data is a dataset containing records of the sale price for every residential property sold at full market value in England and Wales from 1995 onwards.

Price Paid Data for registrations between 1 January 2009 and 31 January 2012 will be released on 28 June 2013, and the remaining data back to 1 January 1995 will be released by November 2013.

Monthly updates from 1 February 2012 onwards are already available as open data. However yesterday’s announcement means that from November the full dataset, some 17 million records, will be available for open re-use free of charge.

I’m not going to clap too hard about this announcement, in part because of the amount of time and effort it has taken to prise the full dataset out of Land Registry’s hands, and in part on principle; it is after all publicly funded data that Land Registry maintain under statutory authority (so it belongs to us dammit).

However Land Registry’s announcement is fundamentally good news. I’ve been critical of Land Registry’s intransigence on open data, and still have my doubts (why the delay until November?), but they are certainly making progress. In addition to yesterday’s announcement, the dataset inventory that Land Registry published last week is an exemplar of good practice that I would be happy to see all public authorities embrace.

Have a biscuit, Land Registry.

Open data release of Price Paid Data was originally announced in the 2011 Autumn Statement, then walked back. I’ve written about this in previous posts. Land Registry did begin to release monthly updates to the dataset as open data. However they managed to protect their commercial interests by treating most of the dataset as a separate “historical” product, along with some waffling about evaluating the potential impact of a full open data release.

In October of last year I nominated Price Paid Data as a candidate for open data release via the new Open Data User Group data request process. The ODUG picked this up with additional material and input from others, and submitted a benefit case to the Data Strategy Board.

It’s unclear at the moment whether Land Registry will receive any compensation from the DSB’s “buy-back” fund for releasing Price Paid Data.* However I am in no doubt that the ODUG’s benefit case was crucial to keeping the spotlight on this dataset and influencing Land Registry’s decision to release it as open data.

The full Price Paid Data release is significant for several reasons:

• Price Paid Data is a reference dataset, which is to say it’s likely to be most useful when combined with data from other sources. Reference data is what we mean when we talk about information as infrastructure. It has value not just in itself but because it underpins wider analysis and enhances the utility of other datasets. Ministers and senior civil servants like to talk about how many thousands of public datasets have been listed on Data.gov.uk, but frankly much of that data is chaff; reference data is the wheat. It’s data that matters.

• Land Registry are moving Price Paid Data from a commercial licensing model to an open licensing model. This is relatively unusual. Under the current Government, open data policy has focused mainly on releasing categories of data that have no established revenue streams attached to them. Consequently the economic promises made for open data have been slow to materialise in the UK. If government departments really want to drive economic benefits from re-use of public sector information, they will eventually have to grasp the nettle and wind down their commercial licensing operations. Charging for public data sucks the dynamism out of information markets, by creating barriers to entry for SMEs and restricting the flexibility with which data can be used — particularly in apps and on the web.

• Price Paid Data is the first open data release identifiably linked to the DSB/ODUG process for unlocking datasets at the request of users. ODUG members have been very active in raising the profile of arguments for open data, but progress has been rather slow on raw release. Now that the first dataset has emerged from the request pipeline, the ODUG can point to this as a signal achievement.

Right. Now let’s talk about unlocking Land Registry’s cadastral data …

* Update: I’ve been told open data release of Price Paid Data has been agreed without use of ODUG/DSB funds.

I’ve made a basic tool to display and browse the Land Registry’s Price Paid information for February 2012 on a map, and put it on my website here:

http://www.owenboswarva.com/ppi/

As mentioned in previous posts, Land Registry maintains a dataset of more than 17 million residential property transfers in England and Wales going back to 1995. From last week Land Registry is releasing monthly updates to that dataset as ‘open data’ under the Open Government Licence.

The first open data available is a csv file containing 56,043 rows of data recorded in the month of February 2012. 93% of the rows of data are new transaction records, i.e. additions to the main Price Paid dataset. The remainder are either changes to existing records or records that Land Registry considers should be deleted from the main dataset.

However only 37% of the additional rows of data apply to property transactions that were effective in February 2012. The other 63% of rows covers property transactions from previous months — mainly January 2012 (44%) and December 2011 (13%).

This rather underlines the difficulty of treating the monthly updates as a discrete dataset. Due to the time lag in reporting of transactions it will be some months before we have a reasonably complete set of transaction data for February 2012.

The map image above shows the geographic distribution of the 19,210 transaction records so far available for February 2012. I have geo-referenced these records to postcode level, relying mainly on the Ordnance Survey’s Code-Point Open dataset. The tool enables users to search transaction records from a map interface and look at the details of individual transactions.

The Land Registry has today published its latest Price Paid information on residential property sales in England and Wales. For the first time this data has been released under the Open Government Licence, as ‘open data’. This means it can be re-used by the public and third-party developers both for personal and commercial purposes.

Unfortunately this open data release contains only a month’s worth of additions and amendments to the Price Paid information dataset. (The release contains data for February 2012, rather than January 2012 as previously expected.) The main dataset, which contains more than 17 million records of house transfers going back to 1995, remains closed and available only on commercial terms.

Details of charging policy removed

The licence cost for the whole dataset is more than £50,000, which makes it affordable only for big businesses. The Price Paid information is gathered by Land Registry under statutory authority as part of its public task.

Until today the licensing costs for chargeable Price Paid information were published on the Land Registry’s website at the link above as follows:

Subscription – We license the data for an annual subscription plus a charge for each address bought.

- Annual subscription is £2,200 plus VAT

- £0.008775 plus VAT per address update supplied, for each product that incorporates the data (with a minimum payment of £100).

Ad hoc purchases – For customers who want to make ad hoc purchases we charge for the data you want to reuse on the basis of £0.0117 per address you take.

This morning those details were removed from the website. It’s possible that means the charging policy is under review. More likely Land Registry knows its charging policy is unreasonable and doesn’t want too much scrutiny. So much for transparency.

Descoping

When the open data release of Price Paid information was originally announced in the Autumn Statement, it initially appeared that the Cabinet Office had decided to release the whole dataset.

This seemed to make a lot of sense. The Land Registry’s Price Paid information is arguably one of the nation’s core reference datasets. Open data release of the Price Paid dataset would produce measurable economic benefits by reducing the input costs for existing licensees in the property and conveyancing sectors and, more importantly, by removing a barrier to entry for innovative small and medium sized businesses that cannot currently afford to access the data in bulk.

By comparison, open data release of only the newest Price Paid information is unlikely to generate significant economic benefits. The main effect will be to make the addresses of new homeowners readily available to direct marketers.

Earlier this month I wrote a longer blog post on this subject.

When the Cabinet Office announced back in November, as one of the Open Data measures in the Autumn Statement, that

from 1 March 2012 Land Registry will make available “Price Paid information” showing all residential property sales in England and Wales at address level

what do you suppose it meant by that?

It seemed simple enough. By law all residential property sales are lodged with Land Registry. Land Registry maintains a Price Paid information dataset with the full address, date of transfer, property type, whether the property is new and whether it is freehold or leasehold. 

The full dataset contains records of more than 15 million house transfers, going back to 1 January 1995.

At the moment licensing costs for the full Price Paid dataset effectively put it beyond the means of non-profit organisations, SMEs, app developers and members of the public.

Based on Land Registry’s charging policy you need pretty deep pockets to license the full dataset or even the most recent few years of data. The licensing fee for Price Paid data from 2005 to present, for example, is more than £50,000 plus an annual subscription to maintain re-use rights. The full dataset is mainly used only by large firms ancillary to the property and conveyancing sectors.

However Price Paid information is potentially very useful for any number of other applications, both commercial and non-commercial.

When the Government said in the Autumn Statement that it would “make available for free a range of core reference datasets … to support the development of high-value data businesses” the Price Paid information seemed like an excellent choice.

Which brings us to March 2012

Based on the Autumn Statement announcement Land Registry was expected to release the Price Paid information yesterday, as open data under the Open Government Licence.

That deadline has slipped, and the latest word on the open data grapevine is that the Price Paid information for residential property sales in January 2012 will now be released at the end of this month.

You will then be able to download that data from the Land Registry website, to use as you like.

But that’s all: one month of data. The following month the data for February 2012 will be available, and so forth.

Monthly updates will be open data — but the bulk of the Price Paid dataset will remain closed, still available only on commercial terms to Land Registry’s big business customers.

If you’re a researcher who wants to use the historic data for socio-economic analysis, or a small developer with a cool idea for a new app, you’re out of luck. Come back in a couple of years, when the monthly updates have built up into a sufficiently large body of open data to be useful.

What’s happened here is called “descoping”

Descoping is the practice of abandoning or weakening the objectives of a project, usually to save money. In a normal business context, when that project is work on behalf of a client, the client can minimise the potential for descoping by compelling performance to an agreed specification.

When it comes to open data releases, descoping is usually a sign that the public body involved isn’t really committed to opening up their data.

My guess is the chain of events went something like this. The Cabinet Office was keen to launch the Public Data Group with a package of new open data. The Price Paid information fit the criteria because it’s a reference dataset and likely to be popular with open data developers and SMEs once the cost barrier is removed.

Land Registry derives most of its income from registrations and copies of land records, so could afford to offer the Price Paid information as open data — if only to show willingness. (Many of the other datasets held by Land Registry contain personal data and are therefore not eligible for open data release.)

But the Cabinet Office only cares about the big picture. Once the Autumn Statement announcement was agreed it left Land Registry to handle the practicalities of delivering the data. Land Registry decided that by taking a narrow interpretation of the announcement it could get away with releasing updates only, and safeguard some of its revenue.

Or could this just be a misunderstanding?

Perhaps the Cabinet Office only ever expected Land Registry to release the monthly updates, and the Autumn Statement was just short on detail.

We can test that by looking at the Public Data Group business case prepared by the Cabinet Office to support the open data package announced in the Autumn Statement. The business case was released recently in response to a Freedom of Information request from Harry Metcalfe via the WhatDoTheyKnow? website.

In the business case Land Registry figures estimate an annual loss of revenue of £600,000 from open data release of the Price Paid information. We are told that Land Registry has 37 existing subscribers for this data, and additionally receives about ten ad hoc requests per month for specific subsets of the data.

Combine that with the figures in Land Registry’s charging policy for the Price Paid information (annual subscription £2,200 + VAT and £0.008775 + VAT per address update supplied), along with Land Registry’s statistics on the number of houses sold each year in England and Wales.

It’s clear that, even as a ballpark estimate, a £600,000 annual loss of revenue is plausible only if it includes Land Registry’s income from licensing of the historic “archived” data.

There go the economic benefits

The PDG business case as a whole has a major weakness: its benefit analysis is based mainly on a generic application of the economic arguments for open data.

If the Cabinet Office had made some attempt to attach benefits to each of the specific datasets in the Autumn Statement package, with reference to their actual characteristics and potential uses, that would have discouraged any descoping of the release.

In respect of the Price Paid information, the business case counts the benefit of savings to existing licensees of the data (based on the full loss of revenue estimated by Land Registry). The business case also notes that “this move will enable more SMEs to establish added value services and products based on the core reference data to some of the key public information organisations in the UK”.

Now that the release has been descoped, that analysis no longer holds. If Land Registry has successfully protected most of its licensing revenue, that will correspondingly reduce the input cost savings to existing licensees. And if most of the dataset remains closed, there is much less scope for innovation by SMEs. Existing big business licensees will retain their market advantages.

So are the monthly updates good for anything?

Open data release of monthly updates to the Price Paid information is better than nothing, of course. As suggested above this data will become more useful as the months go by and it builds up into a larger body of open data.

Developers may find innovative uses that only require data for new residential property transactions. I can’t think of that many myself, but innovations are by definition non-obvious.

The most immediate application for this data is direct marketing. Prior to this release, the terms under which the Price Paid dataset was available from Land Registry precluded use of the data for direct marketing purposes.

As the monthly data is available under the Open Government Licence, that direct marketing restriction cannot apply.

As a near-term effect of this open data release, new homeowners can look forward to targetted advertising from small and medium sized businesses that could not previously afford the Price Paid data.

In fact, I understand that (in the interests of fairness no doubt) Land Registry plans to remove the direct marketing restriction from licensing terms for its commercial customers. They will now also be able to use the closed historical Price Paid data for direct marketing purposes.

Result!

Seriously though.

Land Registry’s Price Paid information is a core national reference dataset that documents residential property transactions in England and Wales at address level over a period of more than 15 years.

The monthly updates are only an incremental addition to that body of data and, by themselves, have nowhere near the same potential for economic use as open data.

This release is a damp squib for the Cabinet Office’s open data agenda, and just one more disappointing example of the gap between rhetoric and follow-through in the Government’s approach to innovation and support for SMEs.

If the Cabinet Office is serious about unlocking public sector data (and the jury is still out on that) it needs to do a lot better.