mapgubbins

Green Belt Boundaries for England - Still Not Open Data

Thu, 16 May 2013 06:35:00 +0100

Some of us with an interest in open data were quite excited when, in late November of last year, the Telegraph not only published the first detailed web-based map of the Green Belt in England, but also made the full underlying dataset (a shapefile obtained from the Department for Communities and Local Government) available for download.

According to the Telegraph article:

This map is the first time it has been possible for members of the public to easily see which areas are green belt land, and which are not.

The Department for Communities and Local Government released the data for the 2011 green belt to the Telegraph, and it is being made available here to view, explore, share and download.

Previously the data has only been available at a cost of tens of thousands of pounds from a third party, despite the location of green belt land being identified by councils using taxpayer money.

Expert users may also download a copy of the green belt map (29MB ZIP file) for use in geographic information systems (GIS).

Green Belt boundaries are a classic example of national core reference data that should be available for public re-use under an open data licence. I first wrote about this dataset back in January 2012, and Alasdair Rae made a strong case for open data release in an article on the Guardian website in August 2012.

The apparent release of the Green Belt dataset by DCLG therefore looked like progress (even if using the Telegraph to distribute it seemed a bit unusual). No less than Ed Parsons, Google’s chief geospatial technologist, applauded the data release in a blog post. Was DCLG “the new opengov data poster child”?

Alas, no. It was all a cock-up …

I have now obtained, under the Freedom of Information Act, the relevant e-mail correspondence between DCLG, the Telegraph and Ordnance Survey. The Green Belt shapefile has not been released as open data, or even to share, by DCLG. Indeed DCLG claim never to have given the Telegraph permission to make the data available for distribution at all.

You can download my FOI request and DCLG’s full response, including the e-mails. However following are the key points:

DCLG provided the Green Belt shapefile to the Telegraph in January 2012 in response to a request. The Telegraph indicated they wanted to convert the data “for display on interactive visualisations and/or as a research base for stories about the Green Belt”.

DCLG confirmed that the Telegraph had an Ordnance Survey media licence. This is relevant because the shapefile contains Ordnance Survey derived data. However according to DCLG the Telegraph contact “did not say that he wanted to make files available for download”.

Distribution of the data to third parties is well beyond the scope of the OS media licence (and in any case DCLG believes that licence expired in July 2012).

In December 2012, shortly after the Telegraph article was published and the shapefile was made available for download from the Telegraph website, Ordnance Survey told DCLG: “my initial thoughts are that the download is without any licence at all and published on a Google API which if derived from OS data would appear to be beyond the terms of the PSMA”. (PSMA is the agreement that allows local authorities to use Ordnance Survey data.)

DCLG and Ordnance Survey discussed how to approach the Telegraph over this licensing issue, but apparently did not do so.

DCLG and Ordnance Survey picked up the correspondence again in February as a result of my query. Ordnance Survey said they were taking a “lenient approach” and would be “happy with the data being made available as equivalent to open data” under the PSMA exemption process.

However DCLG have not submitted any exemption request to Ordnance Survey. There is some suggestion that this is because the shapefile contains local authority IP. However that may just be an excuse. As I’ve pointed out to DCLG, local authorities usually cite Ordnance Survey as the main barrier to open re-use of their boundary data.

DCLG’s conclusion is that “the shapefile published by the Telegraph is in effect unlicensed as it was published without our permission. Notwithstanding that, any Green Belt boundaries provided by local authorities (as in the published shapefile) may only be used for non-commercial uses.”

So what does all this mean?

The first and most obvious point is that DCLG have confirmed the Green Belt shapefile is not currently open data. That’s unfortunate, but Ordnance Survey’s position is encouraging and we can continue to press DCLG to resolve any remaining issues with local authorities.

The complication is that the Green Belt data is now already in circulation, unaccompanied by any clear licensing information. The shapefile remains available for download on the Telegraph website at the time of writing, and is also now available from the London Datastore and ShareGeo Open repositories. It has also been added as a layer to RPTI’s Map for England.

Without any licensing information (other than some obscure metadata), and given that DCLG did not authorise distribution of the shapefile, there is an argument that technically nobody downloading the data has any proper basis on which to re-use it — even for non-commercial purposes. DCLG may have told me that non-commercial use is okay, but they have no mechanism to communicate that more widely to downloaders.

On the other hand, downloaders may already be re-using the Green Belt shapefile in good faith, including for commercial purposes. Since DCLG have made little or no attempt to discourage distribution of the shapefile, they may find it difficult to challenge that re-use.

_____________________________________

Update (May 20):

DCLG have written to clarify that they are “looking to publish Green Belt boundary information ourselves in future and so will consult with local authorities and Ordnance Survey to agree the relevant permissions”.

UK Open Data: An Institutional Timeline

Tue, 07 May 2013 06:58:45 +0100

I’ve started a spreadsheet in Google Docs as an attempt to track on an ongoing basis all of the officially recognised and active boards, panels and advisory groups that meet to discuss and implement UK transparency and open data policy.

The spreadsheet includes a timeline from the 2010 General Election to present, with links to any published minutes.

A few of the groups (e.g. the Local Public Data Panel and the Location Council) were set up under the previous Government. However the majority follow from initiatives announced in PM Cameron’s letters of May 2010 and July 2011 and in last year’s Open Data White Paper.

In principle, UK policy on transparency and open data is led by the Cabinet Office and propagated through government via Sector Boards. Sector Boards take as a reference point the Public Data Principles drawn up by the Public Sector Transparency Board.

Data.gov.uk has a guide that explains the relationships between most of the various groups.

According to the latest tally (in a self-assessment report released by the Cabinet Office last month) the UK so far has “11 active Sector Boards in Transport, Social Mobility, Health and Social Care, Tax, Welfare, Research, Local Public Data Panel, Location Council and Criminal Justice.”

Additionally FCO, BIS and DfE have “established internal panels that replicate the Sector Board terms of reference”. Defra has also recently established an internal Sector Board and appointed an external data user.

It’s a bit difficult to identify all of those 11 “active Sector Boards” but I think this is most of the list:

Transport Sector Transparency Board (DfT)
Social Mobility Transparency Board (DPM)
Health and Social Care Transparency Panel (Department of Health)
Tax Sector Transparency Board (HMRC)
Welfare Sector Transparency Board (DWP)
Research Sector Transparency Board (cross-cutting; Higher Education)
Local Public Data Panel (DCLG, sort of)
Location Council (cross-cutting; Defra, BIS)
Crime and Justice Transparency Sector Panel (MOJ, Home Office)

The other two may (or may not) be the International Development Sector Transparency Board announced by DfID in December and DCLG’s Public Data and Transparency Programme Board.

Many of the Sector Boards have not released any minutes. The situation at DCLG is particularly murky. DCLG’s website suggests the Public Data and Transparency Programme Board might have been disbanded at the end of 2012. The Local Public Data Panel has not published any minutes since May 2012 and its terms of reference indicate it was to be wound down; however based on subsequent passing references I suspect it has only adopted a lower profile.

The lack of output from the Department for Education is also noticeable, given that it is one of the “key delivery departments” flagged for special attention in the PM’s letter of July 2011.

In general the availability of minutes from Sector Boards seems to mirror actual progress in releasing open data, with Transport and Health running ahead of other departments.

Full responses to the draft Code of Practice (Datasets) consultation

Fri, 03 May 2013 03:04:00 +0100

Several months ago the Cabinet Office ran a public consultation on a draft Code of Practice (Datasets) to help public authorities implement some upcoming changes to the Freedom of Information Act.

Those changes will require public authorities to provide datasets requested under FOI in a re-usable format and with a licence for re-use, where reasonably practicable. The draft Code itself and the explanatory material are on the Data.gov.uk website.

In early March the Cabinet Office published a summary of responses to the consultation, but not the responses themselves. I submitted a Freedom of Information request to the Cabinet Office for copies of all responses to the consultation, and received them yesterday.

This zip file contains the contents of e-mail responses from the following organisations:

Department for Communities and Local Government (DCLG)
Department for Environment, Food and Rural Affairs (Defra)
East Cambridgeshire District Council
Essex County Council
Food and Environment Research Agency (Fera)
Ipsos Mori
Land Registry
London Borough of Bexley
Merton Council
Ministry of Defence (MOD)
Archives and Records Association
Chartered Institute of Library & Information Professionals (CILIP)
Health and Social Care Information Centre (HSCIC)
Transport for London (TfL)
University of Oxford
Universities UK (UUK)
Wakefield Council
West Yorkshire Police

Also enclosed is a copy of 50 comments submitted by 17 users via Data.gov.uk. (User names are omitted.)

The following four responses were already on the web:

I have previously blogged my own comments on the Code.

The Cabinet Office’s summary has picked out the main themes, but the full responses provide insight into the concerns of individual organisations. The local authority responses in particular cast doubt on whether FOI practioners will be able to confidently handle the additional issues around formatting and re-use of datasets.

The FOI dataset provisions were originally expected to take effect in April, but have been delayed. We have not yet seen publication of either a revised Code of Practice or the Fee Regulations that will accompany the new FOI provisions.

The Fee Regulations will be crucial in determining whether these provisions are used to promote an open data or a chargeable approach to re-use of datasets released by public authorities under FOI.

Land Registry to release Price Paid Data under Open Government Licence

Wed, 01 May 2013 05:48:00 +0100

Yesterday Land Registry announced that additional Price Paid Data will be released later this year under the Open Government Licence. Price Paid Data is a dataset containing records of the sale price for every residential property sold at full market value in England and Wales from 1995 onwards.

Price Paid Data for registrations between 1 January 2009 and 31 January 2012 will be released on 28 June 2013, and the remaining data back to 1 January 1995 will be released by November 2013.

Monthly updates from 1 February 2012 onwards are already available as open data. However yesterday’s announcement means that from November the full dataset, some 17 million records, will be available for open re-use free of charge.

I’m not going to clap too hard about this announcement, in part because of the amount of time and effort it has taken to prise the full dataset out of Land Registry’s hands, and in part on principle; it is after all publicly funded data that Land Registry maintain under statutory authority (so it belongs to us dammit).

However Land Registry’s announcement is fundamentally good news. I’ve been critical of Land Registry’s intransigence on open data, and still have my doubts (why the delay until November?), but they are certainly making progress. In addition to yesterday’s announcement, the dataset inventory that Land Registry published last week is an exemplar of good practice that I would be happy to see all public authorities embrace.

Have a biscuit, Land Registry.

Open data release of Price Paid Data was originally announced in the 2011 Autumn Statement, then walked back. I’ve written about this in previous posts. Land Registry did begin to release monthly updates to the dataset as open data. However they managed to protect their commercial interests by treating most of the dataset as a separate “historical” product, along with some waffling about evaluating the potential impact of a full open data release.

In October of last year I nominated Price Paid Data as a candidate for open data release via the new Open Data User Group data request process. The ODUG picked this up with additional material and input from others, and submitted a benefit case to the Data Strategy Board.

It’s unclear at the moment whether Land Registry will receive any compensation from the DSB’s “buy-back” fund for releasing Price Paid Data.* However I am in no doubt that the ODUG’s benefit case was crucial to keeping the spotlight on this dataset and influencing Land Registry’s decision to release it as open data.

The full Price Paid Data release is significant for several reasons:

Price Paid Data is a reference dataset, which is to say it’s likely to be most useful when combined with data from other sources. Reference data is what we mean when we talk about information as infrastructure. It has value not just in itself but because it underpins wider analysis and enhances the utility of other datasets. Ministers and senior civil servants like to talk about how many thousands of public datasets have been listed on Data.gov.uk, but frankly much of that data is chaff; reference data is the wheat. It’s data that matters.

Land Registry are moving Price Paid Data from a commercial licensing model to an open licensing model. This is relatively unusual. Under the current Government, open data policy has focused mainly on releasing categories of data that have no established revenue streams attached to them. Consequently the economic promises made for open data have been slow to materialise in the UK. If government departments really want to drive economic benefits from re-use of public sector information, they will eventually have to grasp the nettle and wind down their commercial licensing operations. Charging for public data sucks the dynamism out of information markets, by creating barriers to entry for SMEs and restricting the flexibility with which data can be used — particularly in apps and on the web.

Price Paid Data is the first open data release identifiably linked to the DSB/ODUG process for unlocking datasets at the request of users. ODUG members have been very active in raising the profile of arguments for open data, but progress has been rather slow on raw release. Now that the first dataset has emerged from the request pipeline, the ODUG can point to this as a signal achievement.

Right. Now let’s talk about unlocking Land Registry’s cadastral data …

* Update: I’ve been told open data release of Price Paid Data has been agreed without use of ODUG/DSB funds.

Maggie Thatcher is unpopular in Liverpool

Tue, 09 Apr 2013 19:23:00 +0100

This is a map of people who signed a petition yesterday against a state funeral for Margaret Thatcher:

22,659 people signed the petition yesterday. This map is based on the postcode districts of 22,585 of those responses. (74 responses gave invalid postcode information and are excluded).

I have applied a standard deviation. Red dots indicate the centroids of postcode districts with 23 or more responses. Amber dots indicate the centroids of postcode districts with from 8 to 22 responses.

The postcode district counts are from the little-known but quite useful e-petitions open data API. Here’s a copy of the data I used, as a csv file. This data and the map are for yesterday’s responses only.

The petition itself remains live if you want to sign it.

Following is a list of the postcode districts with 100 or more responses:

L36 Liverpool (Huyton, Roby, Tarbock): 248
L4 Liverpool (Anfield, Kirkdale, Walton): 223
L9 Liverpool (Aintree, Fazakerley, Orrell Park, Walton): 193
L13 Liverpool (Clubmoor, Old Swan, Stoneycroft, Tuebrook): 171
L12 Liverpool (Croxteth Park, West Derby): 168
L20 Bootle (Bootle, Orrell, Kirkdale): 164
L11 Liverpool (Clubmoor, Croxteth, Gillmoss, Norris Green): 150
BN2 Brighton (Brighton, Bevendean, Brighton Marina, Kemptown, Moulsecoomb, Ovingdean, Rottingdean, Saltdean, Woodingdean): 140
L25 Liverpool (Belle Vale, Gateacre, Hunts Cross, Woolton, Halewood): 133
L21 Liverpool (Ford, Litherland, Seaforth): 129
BN1 Brighton (Brighton, Coldean, Falmer, Hollingbury, Patcham, Preston, Stanmer, Withdean): 118
L14 Liverpool (Broadgreen, Dovecot, Knotty Ash, Page Moss): 117
L35 Prescot (Prescot, Whiston, Rainhill): 115
L17 Liverpool (Aigburth, St Michael’s Hamlet, Sefton Park):113
WA7 Runcorn: 110

As you can see, based on this data opposition to a state funeral is strongest in the Merseyside area. More than 15% of responses were from the L (Liverpool) postal area. Margaret Thatcher was never very popular in Liverpool.

This is the distribution in Merseyside:

This is the distribution in London:

This is the distribution in Glasgow:

_____________________________________________________

Attribution statement for the maps:

Contains Ordnance Survey data © Crown copyright and database right 2013. Contains Royal Mail data © Royal Mail copyright and database right 2013. Contains National Statistics data © Crown copyright and database right 2013. Contains some NI data that I got from a bloke down the pub.

Shakespeare Review into Public Sector Information: Draft Recommendations

Thu, 04 Apr 2013 14:49:00 +0100

In June 2012 the UK Government announced that Stephan Shakespeare, Chair of the Data Strategy Board, would lead an independent review of public sector information. Terms of reference and additional information are available on the Gov.uk website.

Market research firm YouGov has recently conducted a survey regarding open data, in support of the PSI Review. (Stephan Shakespeare is CEO of YouGov.) An analysis of the high-level results of the survey was released earlier this week.

Today I had the opportunity to participate in a follow-up survey and comment on a draft of ten basic recommendations that Shakespeare plans to put forward in his PSI Review. The Review is scheduled for publication in May.

[ Update: You can now take the follow-up survey too! ]

Below, presented verbatim and without comment, are the ten draft basic recommendations:

Recommendation 1

Every department of government should make an immediate commitment to publish their single most important dataset quickly, to a high standard agreed to maximise linkability, ease of use and free access. They should also commit to maintaining that dataset and keeping it regularly updated.

Most important dataset to be defined as the one that is used most often by the department itself to carry out its work (or the one most requested by outside users, if that demand can be demonstrated and is significantly different); these datasets taken together will be the ‘National Core Reference Data’.

The situation currently is that most departments have done a reasonable job at making some datasets available; my proposal says it must be their most important datasets, the ones that define their core work; and currently they are not necessarily published and maintained to a consistent format and standard, and that should now happen.

Recommendation 2

Alongside this high-quality core data, departments should commit to publishing all their datasets (in anonymised form) as quickly as possible without concerns about quality - that is, if there is a clash between data quality and speed to publication, they should follow the ‘publish early and ugly’ principle because data scientists are well accustomed to getting value out of imperfect data.

Currently many datasets are held back because it is felt they are not ready because they are not of sufficiently high quality, and that resources prevent their speedy improvement. But data users say that lower quality is not as much of a problem as is non-publishing.

Recommendation 3

Recommendations 1 and 2 taken together define a twin-track policy for a simultaneous ‘high quality core’ AND a ‘publish early and ugly’ policy. This twin-track policy will maximise the benefit within practical constraints (with the further recommendation that departments take pride in adding as many datasets as possible and as quickly as possible from track 1 and track 2).

This approach reduces the excuses for poor or slow delivery; it says ‘get it all out and then improve’.

Recommendation 4

Building on existing activities, there should be an immediate programme of investment in basic data science through our academic institutions, covering both genuinely unfettered ‘basic research’ and research of ‘practical immediate value’ to the national data strategy. We cannot rely only on markets and government departments to maximise the potential of this relatively new and fast-developing field in which we are positioned to be a world leader.

At the moment, America invests massively more than us and continuously reaps the benefits in world-leading business applications of science and technology; yet Britain is capable of being first in this field, given our strength in data science and the fact we have large, coherent datasets. For example, nowhere in the world has such good health data, due to the scale of the NHS as a single provider. There is huge potential here for building social and economic value if we are willing to invest smartly.

Recommendation 5

We should have a clear pragmatic policy on privacy and confidentiality that increases protections for citizens while also increasing the availability of data; we can do this by putting in place guidelines for publication that, if correctly followed, pushes responsibility for (mis)use on the end (mis)user, strengthening application of punitive consequences. Especially sensitive datasets should be accessible only to those who can demonstrate sufficient expertise in the area and whose activity with the data is traceable.

We currently have an unrealistic degree of expectation of any data holder to perfectly protect all our data, which has led to a situation where data scientists are presumed ‘guilty unless proven innocent’ - an attitude that inhibits innovation. Following ‘Best practice’ guidelines should be enough, so long as we are willing to prosecute those who misuse personal data. Otherwise we will miss out on the enormous benefits of Public Sector Information, including open government data.

Recommendation 6

We should have a mechanism for driving the implementation of the national data strategy throughout the public sector, and its oversight. This should include clarity about what data is/can be available, with a feedback loop for its improvement; it will be continuously accessible to citizen and business- user influence. The idea is to be an exemplar of the democratic crowd-sourcing of decisions.

We have several committees, boards, overseers and champions of data; but no easily understood, easily accessed, easily influenced mechanism for making things happen. This is ironic given it’s all about ‘information’. We should create a single channel for driving Public Sector Information, including open government data through the system.

Recommendation 7

We should develop a model of a ‘mixed economy’ of public data so that everyone can benefit from some forms of two-way sharing between the public and the commercial sectors. Data that is derived from the activity of citizens must be seen as being at least co-owned by them and returning value to them, though the investment of business in collecting and processing the data should also be respected.

There are government initiatives such as Midata a government led project that works with businesses to give consumers better access to the electronic personal data that companies hold about them. The project recognises that data about citizens belongs to them and that they should have a way of claiming and using their ownership.

Furthermore, government should be able to make a collective claim on some data if one can make a strong case for public value which is not by other means returning to the public.

Recommendation 8

We should challenge the current quasi-commercial Trading Fund model (for Companies House, Land Registry, the Met Office and Ordnance Survey) in favour of a basic information utility or scientific institution model, in which Trading Funds should be responsible for transparency of data production (that is, collecting and publishing data in a way that can be seen to be reliable and authoritative) and only provide ‘added value’ services where the market is likely to fail.

Currently the Trading Funds do a reasonably good job of collecting, using and sharing data. But many think it would be even better if they could focus on transparent collection and distribution, and where appropriate scientific processing, rather than holding on to it for quasi profit-making purposes.

Recommendation 9

We should expect systematic and transparent use of data in the formulation, implementation and monitoring of government policy, and formally embed this in the democratic process.

Although government does publish some data as evidence for policy, for example in impact assessments, practice varies, and the wider consultation process is not generally considered to be effective. We should deepen and broaden the role of data in policy making.

Recommendation 10

We should continue to provide evidence for the economic and social value of Public Sector Information, including open government data to underpin a bold strategy of investment in an infrastructure of data to make the UK the world leader in this field, thereby gaining the greatest advantage in this new wave of the digital revolution.

Currently we can measure the costs of producing and publishing data but have no model for evaluating the economic or social benefits ‘downstream’, and so we may be undervaluing these activities, leading to underinvestment of resources.

Royal Mail cancels Pinpoint positional data capture project

Wed, 27 Mar 2013 06:50:00 +0000

Update Apr 23: cancellation of Pinpoint has now also been confirmed on the Royal Mail website.

According to a post on the Allies Computing website, Royal Mail have confirmed they will not proceed with their controversial Pinpoint positional data capture project.

Andrea Martin, Managing Director of Data Services for Royal Mail, is quoted:

“Royal Mail announced in the summer of 2012 a pilot initiative in East Anglia to map the co-ordinates of home and business. The pilot explored the potential for Royal Mail to support the location-based information marketplace. Following the completion and full review of the pilot, we have decided not to progress the initiative.”

I’ve posted previously about the origins and progress of the Pinpoint project. If successful Pinpoint would have added geographic coordinates to Royal Mail’s existing Postcode Address File (PAF) dataset.

It’s currently unclear why Pinpoint has been cancelled, though Royal Mail’s attempt to enter the market for geocoded address data had raised eyebrows within Britain’s geographic information community.

Pinpoint would have provided competition for Ordnance Survey’s well-regarded but rather pricey AddressBase products. However the future shape of the address data market is uncertain, with Royal Mail privatisation on the cards and both Royal Mail and Ordnance Survey under pressure to unlock their address data assets as part of an “open data” National Address Dataset.

Royal Mail had reportedly allocated an investment of £10 million or more to the Pinpoint initiative.

Ofcom’s Postcode Address File consultation - update

Fri, 15 Mar 2013 11:06:00 +0000

(Internal review response now received - see update below.)

A brief update on my post of last month, re the key figures redacted from Ofcom’s Postcode Address File consultation:

Royal Mail has also rejected my Freedom of Information request for the PAF cost stack figures, on the basis that they are commercially sensitive. RM maintains that on balance “the public interest in protecting the commercial interests of Royal Mail Group outweighs that of disclosure.”

Quelle surprise, right?

The Ofcom consultation closes next week. In related developments:

the Open Data Institute has submitted a response urging open data release of PAF as part of a free National Address Dataset,
the Open Data User Group has also submitted a (quite strongly worded) response,
BIS, which is currently the main Royal Mail shareholder, has responded to an Open Rights Group campaign e-mail on the same issue,
the PAF Advisory Board held an open meeting (slides), and
Royal Mail and BIS have announced a new Public Sector Licence for PAF, which rather looks like a spoiler for the Ofcom consultation and/or the open data campaign.

Further update (Mar 18):

I’ve now sent a letter to Royal Mail requesting a review of its FOI response, with an argument that it has insufficient grounds for relying on the commercial interests exemption and has also applied the public interest test incorrectly.

Further update (May 16):

Royal Mail has now replied to my request for an internal review of its FOI response. It’s quite long but Royal Mail do not seem to have presented any new arguments of substance.

I do hate to trouble the Information Commissioner, but …

Free school applications - Department for Education FOI responses

Tue, 19 Feb 2013 23:12:00 +0000

Following is a list of Freedom of Information responses in respect of free school applications, published today on the Department for Education website:

Name, local authority, previous school name and faith designation of applications to open a Free School - wave 3 (Date requested: 06 July 2012)

Name, Local Authority, Previous school name, and Faith designation of applications to open a Free School - waves 1 and 2 (Date requested: 22 June 2011)

Proposed name and local authority of Free School wave 2 applications (Date requested: 22 June 2011)

Proposed name, region and local authority of Free School wave 2 applications (university technical colleges, technical academies*, 16-19 Free Schools) (Date requested: 15 June 2011)

Publication of these responses follows decisions by the Information Commissioner and the Information Rights Tribunal. DfE has also published an open letter from Education Secretary Michael Gove to the Information Commissioner. [Update: the Information Commissioner has responded.]

Government releases list of proposed Free Schools to BHA (British Humanist Association)

Creationist sect among 100-plus faith applicants for free schools (Independent)

Private schools apply to go ‘free’ (Financial Times)

Michael Gove writes open letter to the Information Commissioner about free schools (Toby Young blog in Telegraph)

Ofcom's Postcode Address File consultation - key figures redacted

Fri, 15 Feb 2013 07:42:00 +0000

Ofcom has refused my FOI request (submitted in a tweet!) for an unredacted copy of the Postcode Address File consultation document, citing exemptions under Sections 43 and 44 of the Freedom of Information Act.

Section 43 exempts information from disclosure if it would prejudice commercial interests, and Section 44 enables Ofcom to rely on the general restriction on disclosure of information set out in Section 56 of the Postal Services Act 2011.

This refusal is unfortunate but not a surprise. I have no reason to suppose Ofcom has misinterpreted FOIA. The redactions in the consultation document were undoubtably made at the request of Royal Mail, which currently controls the Postcode Address File (PAF).

However it is difficult to understand how the Ofcom consultation process can be credible, without public disclosure of the PAF cost stack figures that have been redacted from the ‘non-confidential’ version of the consultation document.

As noted in the consultation document, the Open Data User Group (ODUG) has recently published a paper arguing for open data release of PAF as part of an Open National Address Dataset. ODUG is firming up an economic case for this proposal, for submission to the Data Strategy Board.

The arguments for open data release of PAF are not new; unlocking national address data has long been a central objective of the UK open data movement. However ODUG is also calling for a change in ownership of PAF, to safeguard the dataset against the Government’s privatisation plans for Royal Mail.

At the moment Royal Mail has an effective monopoly over creation of key elements of the national address dataset. The PAF cost stack figures are particular to those unique arrangements and it is unlikely that their disclosure would significantly benefit any Royal Mail competitor.

The redactions in the Ofcom consultation document therefore seem designed only to create a barrier for anyone trying either to build an economic case for open data release of PAF or to otherwise properly scrutinise the proposals that Ofcom have in mind for the future of PAF.

Open Data or Not? A hard look at the Scottish Data Zone Boundaries licence

Mon, 14 Jan 2013 19:10:28 +0000

Data licensing can be tricky.

There are data sets that are clearly open data and data sets that are clearly not.

Then there are those that have potential, if you look at them in the proper light …

The Indices of Multiple Deprivation (IMD) are a multi-domain measure of relative levels of deprivation within small geographic areas of the UK.

Indices for England, Scotland, Wales and Northern Ireland are produced separately by Government and have slightly difficult criteria, but it’s possible to make comparisons across the data for different countries.

IMD data sets are used widely in the public sector and for academic research. I’ve used IMD data myself as inputs into risk models for the insurance industry. The crime domain in particular is useful in predicting geographic variability of economic losses to theft, civil disorder, etc.

Last month the 2012 version of the Scottish IMD was released by the Scottish Government. Alasdair Rae has put together a useful website that visualises the SIMD 2012 data interactively on a map.

In England and Wales the small-area geography used in the IMD is the Lower Super Output Area. LSOA boundaries are unambiguously available as open data, and may be downloaded from the ONS website and elsewhere.

However in Scotland the unit of geography underlying the IMD is called a Data Zone. The Data Zone boundaries currently in use were produced in 2004 but built up from 2001 Census output areas.

There’s background information on Data Zones here and here. This is what they look like:

Pretty much all administrative geography for England, Scotland and Wales is now available as open data. So I was surprised to see the restrictive licensing terms presented on the download page for the Scottish Government’s Geography Data (which includes the Data Zone boundaries).

You can read the licence on the Scottish Neighbourhood Statistics website. (Click on Go to Data Download, then Download Geography. The SNS site doesn’t like direct links.)

The Geography Data licence starts like this:

We the Scottish Government grant you a non-exclusive non-transferable licence (without the right to sublicense) to copy and use the Data which is derived from Ordnance Survey data and as such is subject to the terms and conditions of the licence agreement between The Scottish Government and Ordnance Survey.

No transferability or sub-licensing, so right away that says it’s not an open licence. Without transferability it would be very difficult to put the Geography Data on a public website in a legally compliant manner.

The wording tells us clearly that the download includes Ordnance Survey derived data. However it’s ambiguous whether the Scottish Government is claiming any additional database rights.

The licence then says:

For any use of this material a Click-Use PSI Licence is also required.

with details of how to arrange that via the Office of Public Sector Information.

The problem is that the Click-Use Licence system was phased out from 2010, when the Open Government Licence (OGL) was introduced. The link provided on the SNS’s download page now resolves to a National Archives page about the OGL.

Click-Use Licences were typically issued for a five-year period, so in theory there could be re-users out there using the SNS Geography Data under those terms. However the OPSI is no longer issuing new Click-Use Licences.

Interpreted literally that means nobody who downloads the Geography Data subsequently can comply with the licensing terms on the download page.

It’s fairly obvious the licence on the download page is out of date. Does that mean we can substitute the terms of the Open Government Licence? That would enable us to do away with the restrictions on transferability and sub-licensing, both of which are supported by the OGL.

There is certainly a convention that the OGL simply replaces the Click-Use Licence, and we have statements to that effect from National Archives.

But I’ve never been entirely comfortable that the Government nailed that down properly, and there are substantial differences between the Click-Use Licence and the OGL. The Click-Use Licence does not specifically set out the scope of permitted re-use, and it seems to exclude mapping data. National Archives has also muddied the waters by introducing additional licences to the UK Government Licensing Framework.

Back to the Geography Data licence:

You may only use the Data for your own internal business use, that is use of Data for the internal administration and operation of your business and not for any commercial purpose, and not for financial profit or gain. Financial gain would include any profit whether direct or indirect, or benefit from the use or publication of the Data in any form.

This tells us that when the licence was written there was no intention to automatically allow full commercial re-use of the Geography Data, as would be permitted under the OGL.

So this addendum:

For any other use of the Data you will need to obtain permission from Ordnance Survey

The remainder of the licence is boilerplate to emphasise key terms from the Click-Use Licence.

Doesn’t look much like open data, does it?

This is why we shouldn’t always take data licences at face value. Licences are contracts, and subject to argument and interpretation.

In this case a literal reading of the licence would prevent anyone from actually re-using the data even for internal business purposes, because the Click-Use Licence is no longer available. It’s a reasonable assumption this is not the intention either of the Scottish Government or the Ordnance Survey. So there’s something wrong with the licence.

This is where it helps to understand the historical context, and the data specifications themselves. Reading the background information gives us some idea of which Ordnance Survey data was used to produce the Data Zones and, although it’s not conclusive, it looks as if it was simply older versions of data that is now included in Ordnance Survey’s open Boundary-Line product.

I did the sensible thing and sought an opinion from the Ordnance Survey. I had to persist a little but in due course received a detailed and helpful e-mail.

The key points from the Ordnance Survey e-mail are as follows:

Although we are unable to confirm which Ordnance Survey data has been used to create this data, we are happy to treat this data as being part of OS Boundary-Line and covered by the Open Data Licence.

As there is additional data present, we would always recommend that you seek permission of the map’s owner (in this case SNS) to ensure that there is no additional copyright enforced by them. It would appear that, as the link provided within SNS’ download licence (which refers to “Click-Use PSI Licence) is now for the Open Government Licence, the data is covered by the Open Government Licence …

On the strength of the above I would personally be confident in treating the SNS Geography Data as open data. I think Ordnance Survey permission to apply the OS OpenData Licence was the only real barrier.

It is significant that the Ordnance Survey is willing to apply the OS OpenData Licence to older versions of Boundary-Line, and presumably also to older versions of the other OS OpenData products. That makes practical sense but I don’t think it is stated explicitly on the OS website.

As regards “additional data present”, it would certainly be better to have a re-written licence on the download page itself to remove any remaining ambiguity. However based on the existing wording, and with due regard to the remarks in the OS’s e-mail, I think there is a perfectly defensible argument that the Scottish Government has left the full licensing decision to Ordnance Survey.

Draft Code of Practice (Datasets): Comments

Sat, 12 Jan 2013 14:45:00 +0000

The UK Ministry of Justice has drafted a Code of Practice (Datasets) to help public authorities implement changes to the Freedom of Information Act expected to take effect in April. These changes, introduced via the Protection of Freedoms Act, require public authorities to provide datasets in a re-usable format and with a licence for re-use, where reasonably practicable.

I wrote about these amendments back in February, when they were at the Bill stage. I took the view then that the proposals presented a threat to open data interests and were likely to encourage more charging for public sector information. There is nothing in the enacted provisions or in the draft Code of Practice to change my view.

Although there was no formal public consultation on the FOIA changes themselves, the MoJ recently ran a consultation on the Code of Practice (Datasets) via a commenting system on the Data.gov.uk website. That consultation closed this week and the comments have now been hidden. However you can still read the draft Code itself and the explanatory material.

These are the general comments that I submitted to the Code of Practice (Datasets) consultation:

I am generally not content with the Code of Practice (datasets), since I disagree with the underlying approach. I think it is a fundamental error to pursue licensing for re-use of data through the FOIA request process, and to conflate access with re-use. I view this approach as a tactic designed to undermine the more robust rights regime articulated in the PSI Regulations.

That said, I would like to endorse in its entirety the consultation response submitted by the Open Data Institute. In particular I strongly support the ODI’s recommendation that the Code should be used to re-enforce open data licensing (and the Open Government Licence) as the default approach to re-use of public sector information.

Following are some additional comments and suggestions:

Public authorities may receive requests for re-use of information where access is subject to the Environmental Information Regulations (EIR) rather than FOIA. The ‘right to data’ amendments in the Protection of Freedom Act do not explicitly apply to EIR requests. However in most other respects public authorities are accustomed to treating FOI and EIR requests in a similar manner. It would be sensible to state in the Code that the ‘right to data’ should be applied to EIR requests also, at least as a matter of informal best practice.

The Code of Practice should note, as a reminder to the public authority, that it may also receive requests for re-use of information that are not attached to an access request. The requestor may wish to re-use data that is already accessible, just not licensed.

The Code of Practice should reinforce the primacy of the access request over any additional considerations relating to re-use of the data. In particular the Code should remind public authorities of their responsibility to meet the statutory timescales for providing access to the data (subject to any exemptions), even if that means the data cannot immediately be provided in a re-usable format and/or a decision on licensing has not yet been made.

Lack of clarity in the Code of Practice on what constitutes a re-usable format is likely to create difficulties, particularly as this may affect costs attached to the request. For example, although Excel files are not universally considered to be an open format, it would be onerous for public authorities to establish charges simply for converting Excel files to another format such as csv.

Overall I found the draft Code of Practice to be rather slight as a guidance document, particularly as the target audience is likely to be FOI practitioners who may not have an in-depth knowledge of licensing and re-use issues. Although the draft was supposedly produced “in consultation with” the Information Commissioner’s Office it has less substance than comparable guidance on other FOI provisions available from the ICO.

The value of the consultation exercise is also rather questionable given that the Fee Regulations have not yet been published. If those regulations give public authorities carte blanche in setting fees for re-use of data, any other good practice arising from this guidance will be irrelevant.

The Open Data Institute’s submission makes a solid argument for revising the draft Code to encourage public authorities to publish open datasets. The MoJ could support open data by drafting Fee Regulations that re-enforce the expectation that public authorities will release datasets either without charge or at marginal cost. However I will be surprised if that happens. Under the current Government the Ministry of Justice has shown little support for open data or for public information rights in general.

The ODI’s suggested re-wording is good:

The Act and the Code are intended to increase regular publication of up-to-date datasets, in a re-usable format, and licensed to encourage their reuse. This Code requires public authorities to publish data that they manage in a reusable format and as open data. The Act does not require datasets to be maintained or updated if they would not otherwise be updated as part of the public authority’s function.

although I would personally like to see “maximise” in place of “encourage”. Public authorities will often say that their data charging policies encourage re-use, but it is more difficult to refute the argument that an open data approach maximises re-use.

I very much like the ODI’s idea that public authorities should be required to publish a cost/benefit analysis, if they want to charge for data instead of applying the Open Government Licence. (Of course this would reverse the onus contemplated by the Government in the terms of reference for the Open Data User Group, which expect applicants to produce a benefit case for open data release.)

I do think there is more work to be done on what we consider to be re-usable formats for the purpose of guidance to FOI practitioners. The vagueness of the Code on this point is likely to create difficulties. However I also differ in some respects with the ODI recommendations.

The statement in the draft Code that “a re-usable format is one that is machine readable” is rather dogmatic, and not a universally held view. That is certainly the way IT developers tend to think about re-use of data, and the statement is more likely to be true for large datasets. However many datasets are re-used directly for analysis (i.e. just read and considered) and do not necessary need to be pre-structured.

A particular problem in thinking about re-use in the context of FOI is that the FOI process is supposed to be “applicant blind”. Re-usability may in practice depend on who the applicant is and what they hope to do with the data.

The Code of Practice should encourage discussion based on the specifics of individual datasets. A single dataset may be suitable for a range of different re-uses and there will not always be an ideal format. When in doubt the public authority should try to strike a balance between making the data re-usable and preserving the data in its original context.

We have to be wary in particular of this knee-jerk aversion to “proprietary formats”. That phrase is used in the draft Code as equivalent to a format that does not support re-use. However there are some file formats that remain technically proprietary but are generally considered to support re-use, because the IP owners have undertaken not to assert rights that would limit their wide use.

See for example the Microsoft Open Specification Promise, which applies to the .xls file format. Similarly ESRI’s shapefile format for geographic data is widely treated as open. It is more important to consider interoperability than whether the format is proprietary.

I am firmly of the view that open data licensing should take priority over re-usable formats. If the MoJ is determined to shoehorn re-use into the FOI process, it may be necessary for open data activists to argue for the former at the expense of the latter. If we take a prescriptive approach to formats, some public authorities will feel obliged to spend an unnecessary amount of time restructuring and converting data for release — and they will want to charge for that time.

We will have to wait and see how the Code of Practice (Datasets) shapes up and how much of the feedback from the ODI and others the MoJ takes on board. It will be difficult to judge the seriousness of this exercise until the Fee Regulations have been published, and there may be a case for more substantive clarification from the ICO once the FOI amendments actually take effect.

Open Data User Group initial recommendations to Data Strategy Board

Fri, 14 Dec 2012 21:06:00 +0000

The UK’s Data Strategy Board has released minutes of a meeting held on 28 November. These minutes provide a first look at the list of datasets identified by the Open Data User Group as good potential candidates for open data release.

The Open Data User Group (ODUG) is a sub-committee of the DSB made up of volunteers, from a range of sectors and backgrounds but all with an interest in UK open data. Over the past several months the ODUG has been collecting data requests from the wider community via Data.gov.uk, in order to identify public data assets likely to benefit the UK if released for wider re-use under an open data licence.

According to the ODUG’s terms of reference it is focusing mainly (but not exclusively) on datasets held by the Public Data Group, a collection of publicly owned trading funds that includes Ordnance Survey, the Land Registry and the Met Office. The DSB has set aside a fund of up to £3.5m to “buy back” data from these and other public bodies.

The ODUG has submitted its initial recommendations in an (unpublished) paper to the Data Strategy Board. The DSB minutes are of course only a second-hand account of those recommendations. However it appears that the ODUG members have homed in on the following datasets as priorities:

A National Address Dataset, based on open data release of both the Royal Mail’s Postcode Address File (PAF) and Ordnance Survey’s AddressBase products. The ODUG had previously announced this as a priority. This recommendation has widespread popular support but there is strong institutional resistance. The DSB has given the proposal its backing in principle, although it is neutral on the ODUG’s additional recommendations about future ownership of the data.

Land Registry Price Paid information. Updates to this dataset from February 2012 onward are already open data, but the ODUG is developing a rationale for release of the historic data as well. The historic data goes back to 1995, so is the bulk of the dataset. This is one of the datasets for which I have submitted a data request to the ODUG.

Met Office hourly weather observations. Observation data for the past 24 hours is already available as open data via the Met Office’s DataPoint service. However the historic data has wide potential for re-use, particularly in applications for the utility and insurance sectors. There’s also a specific request for this data from Mike Godber of C3 Resources.

UK Vat Register information held by HMRC. I suspect this proposal is for release of a subset of non-confidential information from the Register, rather than all data. As noted in this data request from Paul Malyon (who is himself a member of the ODUG), the Register contains a wealth of useful but non-confidential information on unincorporated companies, which do not appear in Companies House records.

Public Rights of Way (PRoW) data held by local authorities and/or Ordnance Survey. The ODUG has asked the DSB to “seek transparency” on costs of either Ordnance Survey releasing a national PRoW data set or local authorities releasing it themselves. There have been numerous requests for this data, and I’ve written about its importance in a earlier post. All or most local authorities have used OS sources to produce their PRoW maps, so there is an issue with OS derived data. OS has previously undertaken to clear away barriers, but to date this has been slow going and only a handful of local authorities have released their data.

River Network. The ODUG has recommended asking Ordnance Survey and the Environment Agency to “create a generalised River Network map as part of medium and small scale location infrastructure” under an open licence. This is another idea that I’ve supported myself with a data request. Mapping of the river network is basic information infrastructure; it seems odd that we have good free public data on roads and railways but not on rivers. OS and EA have already invested in a Detailed River Network product for use by professional hydrologists, and I think they should be able to take some of the basic data (centrelines, nodes, name attributes) from that to produce a generalised map for everyday public use.

The DSG minutes indicate the above proposals are at various stages of development in terms of building rationales for open data release. We will have to wait and see how supportive the DSB members are to the ODUG recommendations. Do they have the appetite to challenge entrenched interests within the PDG and the Shareholder Executive, particularly in respect of address data?

The above list is of course only a first group of datasets. If the ODUG process does produce results, there will no doubt be many other candidates in the request pipeline.

I remain somewhat skeptical of the ODUG process as a well-designed approach to unlocking UK public sector data, mainly because it hinges on a narrow “business case” model that doesn’t adapt well to open data economics. I suspect the ODUG is also being hamstrung by lack of access to detailed information about the finances and commercial activities of the PDG trading funds.

Really the shoe should be on the other foot. Government should be pressing its departments and agencies to justify why they haven’t released their key data assets for re-use as open data, and to demonstrate why they think maintaining an artificial scarcity through charging is really a more beneficial approach.

However I’ve been impressed by the willingness of individual ODUG members to identify and understand the issues behind data requests. The initial short list of datasets that the ODUG has produced (at least as gleaned from the DSB minutes) is actually pretty good. With the help of the open data community the ODUG has come up with a list of datasets that have real potential to provide economic and social benefits if released under an open licence.

Scottish Parliament MSP Allowances: 2011-2012 Bulk Data

Fri, 30 Nov 2012 06:51:00 +0000

Update Dec 4: Please see the end of the post. I have corrected errors affecting a couple of hundred rows in the 2011-2012 data and released a revised csv. Additionally I’ve released a csv containing the bulk data for the 2010-2011 financial year.

On Tuesday the Scottish Parliament released the latest figures for MSPs’ allowances and expenses. Totals by category were included in a press release, and newspapers quickly published the list of overall spend for each MSP in 2011.

Further detail of individual expenses is available via a new search tool on the Scottish Parliament website:

The tool is rather clunky and looks very much like the functionality on the website of IPSA, which publishes expenses for MPs in England and Wales.

However unlike IPSA the Scottish Parliament has declined to provide access to the bulk data. The search tool allows users to drill down to a Results Details screen for individual claims …

… but there’s no way to download all of that detailed information in a csv file or similar machine-readable format.

STV journalist Michael MacLeod highlighted this problem on the Data.gov.uk website:

I spoke to the Scottish Parliament’s media officer Eric MacLeod on the telephone to ask whether the information was available in a machine readable format. I suggested that a CSV spreadsheet would be of use to journalists like myself wishing to help the public find out how much their MSPs cost them. I was told: “We don’t think that would be meaningful in any way.” I was also told that an FOI request for the information in a machine readable format was likely to be rejected.

The open data movement hasn’t really taken off in Scotland to the same extent as in England. However it’s difficult to believe that the Scottish Parliament is not well aware that there might be some public interest in further analysis and scrutiny of MSPs’ allowances and expenses.

So, for purposes of transparency and to re-enforce the principle that open data should really be released in a re-usable format, I have scraped all of the detailed Scottish Parliament MSP Allowances information for the 2011-2012 financial year and put it into a single csv file.

Anyone who wants the data can download it from my Google folder here:

~~https://docs.google.com/open?id=0B0AX7sP1au8Yc3FrQm10Ml9QZ2M~~

https://docs.google.com/open?id=0B0AX7sP1au8YNk1TdHI3Umt5Skk [ new link ]

(The folder is public but you may need a Google account to access it. If you have any difficulties downloading the data please contact me.)

The SPMSPA_2011-2012.csv file contains ~~33,260~~ rows of data from 17,736 Results Details screens. I’ve included a ReadMe.txt file with some notes on the structure of the data.

Update Dec 4

Good news: I’ve now also scraped the bulk data for the 2010-2011 financial year and released that in a separate csv file, SPMSPA_2010-2011.csv.

Bad news: I’ve deprecated the SPMSPA_2011-2012.csv file released on 30/11/2012 and released a replacement, SPMSPA_2011-2012_V2.csv.

Due to errors I made during the scraping process, a small number of transactions were duplicated and included in the 2011-2012 data in place of an equal number of omitted transactions. A total of 186 rows of data were duplicates, and a total of 294 rows of data were omitted. I have corrected those errors in the new file.

I’ve also discovered that where a transaction has been itemised in a table containing more than ten rows, only the first ten item rows were captured by the scraping process. That means that some travel-related allowances are not fully itemised. This issue affects slightly less than 800 transactions in the 2011-2012 data and slightly more than 800 transactions in the 2010-2011 data. The transaction totals are unaffected, i.e. they are correct. I have not fixed this issue; sorry.

There’s a new link to the data; see above.

Ten awesome open data resources (missing from Data.gov.uk)

Mon, 24 Sep 2012 12:02:42 +0100

Looking for UK open data? You probably know all about Data.gov.uk, the UK Government’s main portal for public data. Data.gov.uk lists more than 8,600 public data sets, most of them re-usable under the Open Government Licence.

Depending on your area of interest, you might also want to trawl through a more specialised data hub like the National Statistics website, the Department for Transport website or the NHS Information Centre.

But that’s far from everything. As I highlighted in my previous post we don’t have a proper inventory of UK public sector information. It’s mainly up to individual data holders whether they register their open data on Data.gov.uk.

That means there are still quite a few significant public data resources, open for re-use either explicitly or effectively, that you won’t find in any of the usual places.

Below is a personal list of ten UK open data resources, all currently missing from Data.gov.uk, that I’ve found useful and think deserve wider attention.

(Pic by Jonathan Gray, Creative Commons CC0 1.0 Universal Public Domain Dedication)

1. EduBase (Department for Education)

EduBase is the DfE’s up-to-date database of educational establishments across England and Wales. It contains records for every school: addresses, geographic coordinates, school characteristics, pupil census data, and more. Colleges and universities are in there as well.

If you register on the website you can download the complete database up to twice a year, for free. The data is licensed for re-use under the terms of the Open Government Licence.

2. Free Public Data Product (Companies House)

The Free Public Data Product was released by Companies House at the start of June 2012. It’s a monthly “snapshot” containing basic information on all live companies on the Companies House register. That includes addresses, dates of incorporation and SIC codes.

Companies House says the data is “provided free of charge and will not be supported”. There’s no specific mention of licence terms on the website, but the data is described as openly re-usable in BIS’s Open Data Strategy and elsewhere.

Companies House released this data very, very quietly and it’s not that surprising they haven’t registered it on Data.gov.uk. As a Trading Fund, Companies House are not exactly enthusiastic about open data. But you probably guessed that from the amount of effort they put into naming the product.

3. Oil and Gas Data and Maps (Department of Energy and Climate Change)

Oil and Gas Data and Maps is a collection of DECC data sets that includes maps of both onshore and offshore oil and gas exploration and production (raster and vector), plus well data, field data, stratigraphy and lithography data, and so on.

This data should be useful for anyone with an interest in the energy sector, including investors, insurers and risk managers.

Or if you’re environmentally minded, it’s a great resource for monitoring which companies are exploring and drilling where around the UK. I used some of the DECC spatial data in a web article I wrote last year on shale gas exploration.

Most of the data in this collection is open for re-use, but check the documentation with individual data sets.

4. Postbox Data (Royal Mail)

As you might expect Royal Mail maintains database records of all of their postboxes. This data is not available for download in bulk from the Royal Mail’s website. However Royal Mail have provided the full data several times in response to FOI requests.

Most recently in July Royal Mail provided a data set of postbox locations and collection times from the Central Collections Management Database, and additional data on meter and private boxes from their Final Plate Database.

The FOI correspondence is silent on licensing, but as a matter of existing practice Royal Mail don’t seem to be discouraging open re-use.

5. Wind Development Data (Department of Energy and Climate Change)

DECC also maintains a collection of data sets to support the renewables side of the energy industry.

The Aviation Safeguarding Maps page provides access to location data for aerodromes and airfields, and wind farm grid references. Data on the DECC is subject to the Open Government Licence unless otherwise stated. (You will note that the MOD Safeguarding Data on the same page is not open data.)

There’s also a Windspeed Database based on a 1 km square resolution. What I like most about this database is the DECC’s information warning. They’ve clearly inherited the data and have only a vague idea how it was produced.

6. Registered Places of Worship (General Register Office)

The General Register Office maintains a list of all registered places of worship in England and Wales, in compliance with the Places of Worship Registration Act 1855. The list includes addresses and religious denominations (excluding the Anglican Communion), and is useful for a number of geodemographic applications.

This is one of those data sets that’s safe for open re-use but you have to get hold of it first. The GRO doesn’t make a habit of publishing the list, but will release it under FOI.

The most recent list in general circulation is from an April 2010 request. Unfortunately it’s a PDF document. However if you submit a new request the GRO should provide the list in a more re-usable format, per new provisions in the FOI Act.

7. Flood Risk Objections (Environment Agency)

The Environment Agency holds an enormous amount of data related to flood risk, very little of which is open for re-use. One of the exceptions is a monthly spreadsheet of objections to planning applications on flood risk grounds.

This is quite a useful source of information for anyone considering buying (or financing) a home in a new development. Although local planning authorities are obliged to consult the Environment Agency about developments within the flood zone, the EA normally has no power to actually block a development.

Availability of the objections data means anyone with an interest in a new development can easily check whether an objection was raised, and if so investigate the planning history further to make sure the EA’s flood concerns were addressed.

At the moment the most recent monthly data at the above link is from October 2011. However I’ve made an inquiry and the EA have said the page will be updated with the missing information shortly.

8. Government Art Collection (Department for Culture, Media and Sport)

The Government Art Collection is a collection of over 10,000 works of art placed in British Government buildings in the UK and abroad. The DCMS maintains a searchable database with pictures, descriptions and locations of the art.

The information in the database is re-usable under the Open Government Licence. There is no link for bulk download of the data, but it’s a doddle to scrape.

9. 2011 Boundary Data (Census Dissemination Unit at Mimas)

The Census Dissemination Unit (CDU), based within Mimas at the University of Manchester, provides access and support for UK Census data and related data within the academic sector. Earlier this year CDU also opened up its services to the public.

2011 Boundary Data is a page of downloads of boundary data for England and Wales. Although the base boundary data is available from other sources, this is the best collection currently available for free to the public. The boundary data has been optimised for use at different resolutions and is available in various GIS formats including KML.

CDU also makes available reformatted versions of the 2011 Census data released by National Statistics, and Combined Census and Boundary Data.

10. HM Government e-petitions (Cabinet Office)

In May the Government Digital Service, a team within the Cabinet Office, released an API for the UK’s e-petitions system. Peter Herlihy of GDS blogged about it, as did Richard Parsons on the eDemocracy blog.

The API provides JSON feeds that include a master list of live petitions and anonymised statistics on supporters of each petition. Although the statistics are only at postcode district level, the data has substantial potential for re-use. Most obviously, the data can be used to analyse the geographic distribution of sentiment on contentious public issues.

Whatever happened to Information Asset Registers?

Tue, 11 Sep 2012 09:18:00 +0100

One of the challenges of trying to unlock public data for re-use is actually identifying data sets held by Government departments and agencies.

Whatever your area of interest, you have to know the data exists before you can argue for its release.

So what we need in the UK is a comprehensive inventory of structured data assets held by the public sector. Obviously.

Data.gov.uk website should serve that function, but it doesn’t. Data.gov.uk is just a big bucket for meta data. Public bodies can throw in pretty much what they like, or not.

There’s no requirement that public bodies register all of their data assets on Data.gov.uk. Even when they do register their data the standard of curation is highly variable.

The Open Data White Paper published by the Cabinet Office in June mentions a “workstrand” to establish a government data inventory. However it’s presented only as an additional feature that could be supported by the Data.gov.uk site.

What’s missing is any evidence of forward movement on a protocol. Government departments and agencies must be required to prepare or provide their own inventories of data assets. This is surely an essential step in the construction of a Government-wide inventory.

This is a matter for the cross-departmental Senior Officials’ Group on Transparency. However minutes of that group’s meetings make only a vague reference to plans for a workshop session on the subject.

The White Paper also suggests the Cabinet Office sees the proposed data inventory principally as a way to facilitate future publication and prioritisation of data sets. That creates a risk the inventory will be composed mainly of data sets departments are minded to make available for re-use, in support of their policy agendas.

That will only perpetuate the existing weakness of the Data.gov.uk approach. To be legitimate an inventory should disclose, in a politically neutral manner, all data assets held by public bodies.

Information Asset Registers

This brings us to Information Asset Registers and the question of which individual departments already hold and/or publish their own data inventories.

Information Asset Registers (IARs) were introduced by the previous Government in a 1999 white paper, The Future Management of Crown Copyright. The IAR concept was one of a number of initiatives aimed at “improving and encouraging access to the broad range of public sector information”.

(That’s right, kids. The Transparency Agenda wasn’t invented by Francis Maude and the Coalition. They just made it cool.)

IARs were promoted mainly by the Office for Public Sector Information (OPSI), prior to its absorption into National Archives.

In a nutshell the IAR is a mechanism to help an organisation understand and manage its information assets. It records the assets, their ownership, any business requirements and technical dependencies, etc.

An IAR might require its own database system or it might be a simple spreadsheet. That depends on the particular needs of the organisation.

National Archives continue to suggest the use of IARs as part of its guidance on digital continuity. However in its guidance on Information Principles, published late last year, the Cabinet Office clearly regards IARs only as a helpful concept.

Government departments are currently required to demonstrate nothing more than a “defined approach” for “consistently identifying, categorising and cataloguing information assets and their purpose”. There’s no expectation departments will necessary maintain a formal asset register and publish it.

Inforoute

Some years ago OPSI ran a project to build a central database of Information Asset Registers. The database was called Inforoute.

By 2008 Inforoute was providing searchable online access to information on data sets held by at least 18 Government department and agencies. The Inforoute project also coordinated and monitored progress on development of IARs across the public sector.

From outside Government it’s difficult to trace exactly what happened to the Inforoute project. Archived reports only tell half the story.

It’s possible the previous Government intended to fold Inforoute into the newly launched Data.gov.uk, but that the Coalition abandoned it. That’s what happened to other strands of work on public sector information initiated under Labour.

It may be symptomatic of how little progress has been made on IARs since then that, even today, several Government department websites still link to a non-functional archived version of the Inforoute interface.

Which departments publish IARs?

In June, 17 Government departments published their Open Data Strategies. Almost all mention the Government’s Information Principles and in particular Principle 1 - Information is a Valued Asset. Only a few mention Information Asset Registers.

The rest of this post consists of my notes on availability of Information Asset Registers or data inventories from those 17 Government departments. In particular I’ve reviewed the Open Data Strategies and information provided on department websites.

But first, some kittens. (Hey, you try to find an appropriate image to illustrate a post on Information Asset Registers.)

Department for Communities and Local Government (DCLG)

DCLG’s Open Data Strategy doesn’t mention an IAR. Its website briefly explains what an IAR is, and links to the defunct Inforoute database.

Archived information indicates that as of 2008 DCLG was one of the departments with IAR records accessible via Inforoute.

Result: DCLG had an IAR under the previous Government but does not maintain one now.

Department for Work and Pensions (DWP)

DWP’s Open Data Strategy doesn’t mention an IAR. Its website explains what an IAR is, and links to the defunct Inforoute database for further information.

DWP’s website also includes a page indicating it began an IAR in 2002 and that it is updated on an “ongoing” basis. However that page hasn’t changed for at least the past two years.

Result: It’s possible someone conscientious at DWP is keeping an IAR up-to-date, but I’d be surprised. There’s no published IAR available.

Department for International Development (DfID)

DfID’s Open Data Strategy states: “We have an Information Asset Register that assigns responsibilities for each asset to an Information Asset Owner.” No reference to an IAR on its live website, but archived versions mention DfID’s participation in Inforoute.

Result: DfID is maintaining an IAR, probably as a normal part of its IT security strategy, but doesn’t publish it.

Home Office

The Home Office’s Open Data Strategy says: “Our publicly available Information Asset Register is published on the Home Office website, and accessible via data.gov.uk. Information assets are reviewed by the relevant Information Asset Owners, in collaboration with others, to assess opportunities to provide re-usable data to support transparency, whilst considering the operational risk of doing so.”

And the IAR is indeed published on Data.gov.uk as well as on the Home Office website.

The Home Office’s IAR is rather short on detail, and some entries are for either database systems or collections of data sets rather than specific information assets. However it’s good enough to provide a starting point for investigation.

Result: the Home Office maintains an IAR and publishes it.

Department of Energy and Climate Change (DECC)

DECC’s Open Data Strategy says it has “compiled an information asset register to identify datasets across the Department” as part of its Knowledge and Information Strategy.

The only mention of an IAR on the DECC website is what must be a very out of date reference to Inforoute, as “the new gateway to information held by all UK Government departments”.

Result: If we take the DECC at its word it has an IAR of some sort, although unpublished.

It’s worth noting that the Coal Authority, an agency of the DECC, maintains its own Information Asset Register. This is available for download from the DECC website. It was last updated in 2010.

Foreign and Commonweath Office (FCO)

FCO’s Open Data Strategy states: “The FCO aims to complete the process of developing a corporate Information Asset Register (IAR) by the end of March 2013. The IAR will act as an information risk management tool, and will cover both paper and electronic assets and will give details of, for example, where the assets are held, what they contain, who owns them and who has access to them.”

Result: FCO doesn’t have an IAR but is compiling one. Its unclear whether it intends to publish it.

HM Revenue & Customs (HMRC)

HMRC’s Open Data Strategy states:

“As part of our ongoing commitment to transparency we will continue to analyse and understand the extent of the information assets we hold, through developing information models, asset registers, data dictionaries and repositories in order to guide and, where necessary, impose information management.

“In May 2011 we were one of the first Departments across Whitehall to publish an initial inventory of their datasets. This project allowed us to start investigating the life cycle of our data and take stock of our information, analysing ownership, security marking/disclosure and assessing publication opportunities/risks.”

This is a reference to the data catalogue released with HMRC’s Transparency Implementation Plan. This is available as an Excel workbook on its website, and is reasonably equivalent to an IAR.

Result: HMRC has an IAR and has published it.

Ministry of Defence (MOD)

MOD’s Open Data Strategy makes no mention of an IAR. However its website indicates it has maintained an IAR since 2002, updated most recently in June 2012. Format is given as “website”, but the only link provided is to the defunct Inforoute website.

Result: MOD has an IAR and is maintaining it. It may think it has published it, but in practice it’s not readily available. MOD would probably provide a copy on request though.

Ministry of Justice (MOJ)

There’s nothing in MOJ’s Open Data Strategy or on its website to confirm it maintains an IAR.

Prisons and some other bodies reporting to MOJ, such as the AJTC, do clearly maintain IARs. It would be odd if the MOJ itself did not maintain an IAR or something similar, particularly given its responsibility for National Archives (which provides guidance on information policy). IAR records for MOJ were available via Inforoute under the previous Government.

Result: No evidence found of a current IAR, and certainly not a published IAR.

HM Treasury

Treasury’s Open Data Strategy says that “… the Treasury Group has followed the mandatory requirement by identifying a series of information assets and assigning responsibility of these information assets to individual owners within the department. These assets are reviewed on an annual basis together with an information asset register and all these are made available to the business with training material for the information asset owners via a dedicated intranet site.”

No mention on website.

Result: HM Treasury maintains an IAR but does not publish it externally.

Department for Business, Innovation and Skills (BIS)

A message on the BIS website currently states: “The BIS Information Asset Register is being compiled and will be published in due course.”

No mention in BIS’s Open Data Strategy.

Result: BIS doesn’t have an IAR, but is preparing one and intends to publish it.

Department for Education (DfE)

DfE’s Open Data Strategy says it is putting in place a “new Information Asset Owner support system which will include online storage and management of Information Asset Registers”.

No mention of IARs on the DfE’s website.

Result: Very unclear. DfE may or may not currently maintain one or more IARs internally, but there’s nothing published.

Department for Environment, Food and Rural Affairs (Defra)

There is quite a good IAR, last updated in October 2009, on the archived version of the Defra website. However there is nothing on the live site.

Defra’s Open Data Strategy does not mention an IAR. It does highlight the Environment Agency’s Information for Re-use Register. That document is not the same as an IAR, but it is a useful catalogue of data products available for re-use from the EA.

Ironically, although the EA’s Re-use Register itself is regularly updated, the same webpage continues to state that the OPSI is “coordinating the compilation of a Government-wide Information Asset Register”. Similarly the Rural Payments Agency section of Defra’s website directs IAR inquiries to the defunct Inforoute search interface.

Result: Defra doesn’t maintain a current IAR.

Department for Transport (DfT)

DfT maintains an IAR, updates it annually, and publishes it on Data.gov.uk. It’s clear and comprehensive. DfT’s Open Data Strategy explains the role of its IAR in supporting the Government’s Information Principles.

Result: Excellent. All departments should be copying DfT’s approach to public sector information.

Department for Culture, Olympics, Media and Sport (DCMS)

DCMS’s Open Data Strategy mentions in passing that information assets are catalogued as part of its information security procedures. DCMS’s IAR is also discussed in an April 2012 Information Management Assessment from National Archives. However the IAR is not available on DCMS’s website.

Result: DCMS maintains an IAR but does not publish it.

Cabinet Office

Neither the Cabinet Office’s Open Data Strategy nor its live website mentions an IAR. An archived version of the website indicates that in 2008 the Cabinet Office had an IAR with 48 entries.

In its Open Data Strategy the Cabinet Office states that it publishes its data “in line with the Public Data Principles set out by the Transparency Board, including registering it on www.data.gov.uk.” It’s plausible that, as the department responsible for Data.gov.uk, the records on that site are a more complete representation of the Cabinet Office’s information assets than they would be for another department.

Result: No evidence that the Cabinet Office maintains an IAR.

Department of Health (DH)

DH publishes an Information Asset Register on its website. Although the homepage mentions the defunct Inforoute database, there is evidence that the IAR on the DH site itself is being actively maintained as some records have been updated in 2012.

DH’s IAR is not mentioned in its Open Data Strategy.

Result: DH maintains and publishes an IAR.

____________________________________________________

Summary and Conclusions

Of the 17 Government departments considered above, there is evidence that nine currently maintain Information Asset Registers. It’s possible some others maintain IARs on the quiet as part of their information security procedures. But only four actively publish them.

Some of those departments have more significant information assets than others. However three of the five key delivery departments mentioned in the Prime Minister’s transparency letter of July 2011 (Health, Education, Justice, Work and Pensions, Transport) do not publish an IAR.

The Cabinet Office and the MOJ, the two departments with particular responsibility for the broader public information agenda, do not publish IARs.

Six departments continue to link to Inforoute on their websites, although that service has been out of operation since well before the change of Government in 2010.

Not very good is it?

In practice the above suggests a couple of points.

If the Cabinet Office is serious about compiling a proper national inventory of public data it will probably take quite a lot of work to fill in the gaps. At the moment there is no way of establishing the full extent of public data assets. That means we have no idea what proportion of total data assets is currently registered on Data.gov.uk.

And those of us interested in opening up more types of public data will have to continue to rely on ad hoc investigation to discover new information assets. Proactive disclosure by Government has some considerable way to go.

Public Rights of Way in England and Wales: more progress needed on open data release of national vector mapping

Wed, 29 Aug 2012 06:38:00 +0100

Why is there no open national map of Public Rights of Way in England and Wales? Or to put the question another way, why isn’t the information that local councils maintain on Public Rights of Way freely available for re-use?

This post is specifically about open data release of vector mapping data for Public Rights of Way (PRoW) maintained by local surveying authorities. The statutory basis and policy background to PRoW information is quite complicated so I’m only going to sketch it in. However there are some links at the end of the post if you want to read into the subject further.

Surveying authorities

In England and Wales public rights of way are routes on which the public has a legally protected right to pass, on foot and sometimes by other modes of travel as well.

Surveying authorities (usually county councils or unitary authorities) are required by law to maintain a Definitive Map of public rights of way in their local areas. Each right of way also has a written description called a Definitive Statement.

Definitive Maps have always been available to the public for inspection at local offices of surveying authorities. However these days most authorities also put their PRoW information on the web. Some deliver this information via interactive mapping applications, others simply as raster maps or in PDF documents.

Whether on paper or online, surveying authorities usually present their Definitive Map information on Ordnance Survey background mapping.

Keeping Definitive Maps up to data is an area of considerable activity for surveying authorities. There is an ongoing effort to capture all unrecorded footpaths and bridleways created before 1949, because the Countryside and Rights of Way Act 2000 mandates that these cannot be recorded after 2025. DeFRA has recently consulted on new proposals to modernise the recording of rights of way.

Ordnance Survey use of PRoW data

Ordnance Survey began to add rights of way information to its maps back in 1960. Rights of way are now consistently shown on Ordnance Survey 1:25,000 (Explorer) maps and 1:50,000 (Landranger) maps, and in OS digital raster mapping products of equivalent or larger scale.

Although surveying authorities provide their PRoW information to Ordnance Survey for use on its maps, the Definitive Map and Definitive Statements held locally remain the authoritative record for statutory purposes.

When the previous Government prevailed upon Ordnance Survey to release its OS OpenData suite of products in April 2010, one of the main grumbles raised by digital mapping enthusiasts was the failure to include any Public Rights of Way data.

There was an excellent discussion of this issue on the Ordnance Survey’s blog at the time, in which Ordnance Survey staff did a very good job explaining the related problems.

Intellectual property rights and the derived data issue

In a nutshell there were two barriers. First, the main IP rights for Definitive Map data are held by the local surveying authorities, because they originate it and have statutory responsibility to maintain it. Second, most (and quite possibly all) surveying authorities prepare their Definitive Map by drawing over features on Ordnance Survey maps. This makes some of the PRoW information “derived data” that relies on sub-licensing of Ordnance Survey’s IP rights.

The upshot was that back in 2010 neither Ordnance Survey nor the surveying authorities were in a position to release PRoW mapping data for open re-use. However there was at least an awareness of public interest in open re-use of the data.

The 2011 Autumn Statement

Last year the Government’s Autumn Statement included a number of open data measures promoted by the Cabinet Office. One of these was a note that “Ordnance Survey has committed to amend its derived data restrictions on Local Authorities’ Public Rights of Way data, enabling this to be released more easily as Open Data.”

This implicitly put the onus on local authorities to release the data, once Ordnance Survey had cleared the way. This was one of two policy approaches the Cabinet Office could have taken. The other would have been to secure a collective commitment from local authorities to allow Ordnance Survey to release its own processed version of their data.

Ordnance Survey itself hasn’t made any public statement on how it has implemented the Autumn Statement commitment. However I gather from online discussions that it has been telling any local council inhibited by the derived data issue that they are now free to release their Definitive Map data for re-use under the OS OpenData Licence.

The OS OpenData Licence is the same licence that applies to Ordnance Survey’s own OpenData products. It basically incorporates the Open Government Licence (OGL) with additional attribution requirements to acknowledge Ordnance Survey as source of some of the data.

Recent open data releases of PRoW vector data

That’s basically where we are today. In principle the derived data issue is no longer a barrier. Any surveying authority that holds its PRoW Definitive Map as digitised vector data can simply provide that data as a download on its website.

The problem is that very few have done so. Honourable exceptions are Hampshire County Council and Devon County Council, both of which provide their data as downloads in KML and ESRI Shapefile formats. Hampshire specifies the OS OpenData Licence; clarification from Devon on licensing is pending but it’s reasonable to assume they will take the same line. Additionally Worcestershire County Council’s vector data is re-usable via WMS, also under the OS OpenData Licence.

UK volunteers who work on OpenStreetMap are keen to add PRoW vector map to their grand project, so long as the licensing can be properly cleared and documented. The OpenStreetMap Wiki includes a table of surveying authorities with notes on their view services and on availability for re-use of the Definitive Maps and Definitive Statements. (I am grateful to Rob Nickerson for bringing this to my attention.)

However OSM has exacting standards, and represents only one potential type of re-use for the PRoW vector data. Clearly an approach based on engaging with surveying authorities individually is going to be slow going.

Next steps - where’s the follow-through?

If we want a comprehensive release of Public Rights of Way open data within the life of the current Government it’s going to require a push from the centre.

As I’ve remarked before, Francis Maude et al in the Cabinet Office seem to be better at making policy announcements on open data than they are on the mechanics of delivery. However I would hope PRoW data is an issue that would fit well on the agenda of the new Open Data User Group.

As PRoW data is not separately productised by Ordnance Survey, and surveying authorities don’t seem to be deriving significant revenue from it, there should be no vested interests to undermine the business case for wider open data release.

Ideally I would like to see Ordnance Survey release the vector data it holds for Public Rights of Way either as a new product in its OS OpenData suite or as an enhancement to OS VectorMap District. This version of the data might not be “authoritative” to the standard of surveying authorities; but then the point of open data release is to maximise re-use of a data set for additional purposes rather than use for its original purpose.

The real practical question is whether Ordnance Survey actually holds PRoW data with full coverage in vector format. Back in April 2010 there were reportedly still some gaps.

My hope is that Ordnance Survey already maintains a discrete PRoW data set for its own production purposes. There is always a limit to how much additional preparatory work we can reasonably expect a public body to do in support of an open data release.

At the same time PRoW data is an issue that the Local Public Data Panel could perhaps take up. The LPDP could encourage more local authorities to release the data themselves, perhaps with best practice guidance. It could also organise local consents for Ordnance Survey to release the national data set, with suitable attribution of surveying authority copyrights.

_____________________________________________________

Update (Aug 29): PSMA Exemption Process

Steven Campbell, Chair of the PSMA User Group, has pointed out that the Public Sector Mapping Agreement includes a formal exemption process, under which PSMA members can ask the Ordnance Survey for permission to release derived data under either OS OpenData or Free to Use terms.

There’s a detailed guide to PSMA arrangements on the Ordnance Survey website. Most if not all surveying authorities should be PSMA members.

It seems to me that the Autumn Statement commitment is effectively equivalent to an informal but blanket application of the exemption process to Public Rights of Way data. If we could persuade Ordnance Survey to confirm that to PSMA members, ideally with a statement online that everybody could see, it would go some considerable way to clarifying the licensing position.

_____________________________________________________

Want to know more about Public Rights of Way?

Natural England publishes a guide to Definitive Maps and changes to public rights of way, which is free to download.

There’s a membership organisation for professionals involved in the management of public rights of way and access, IPROW. Their Good Practice Guide is full of information.

The PRoW information on the previous Government’s version of the DeFRA website lacks the more recent changes in policy and legislation but remains useful as background.

This Ordnance Survey Blog post from last year is also good: Everything you need to know about Rights of Way.

Areas for improvement in available flood data for England and Wales (Adaptation Sub-Committee report)

Wed, 11 Jul 2012 09:50:00 +0100

This morning the Adaptation Sub-Committee of the Committee on Climate Change, an expert body that advises on the UK Government’s programme for adaptation to climate change, published a report to Parliament on the state of preparedness in England and Wales for two of the largest climate risks, flooding and water scarcity (or drought).

This is probably the best report I’ve read on UK flood policy under the current Coalition Government. (You can read the media coverage for the highlights.) The report is robustly critical of weaknesses in Government policy and provides solid analysis of the consequences for increased flood risk in England and Wales. In fact it’s rather the sort of policy analysis we used to get from the Environment Agency, before Defra put a muzzle on the EA in late 2010.

One section of the report was of particular interest to me. Chapter 4 includes an annex that lists the data sources available to the Sub-Committee, along with “areas for data improvement”.

Below is a list of the flooding data that the Sub-Committee’s experts considered should be available but found not to be. I think much of this list will resonate for anyone who has worked with flood data at national level for insurance or risk assessment purposes.

Type of development: can only distinguish between residential and commercial. It would be useful to understand number of upper floor flats within floodplain.
Number of nonresidential properties: substantial discrepancies between different datasets.
Data on surface water flood risk are less advanced than for river/ coastal flood risk.
Infrastructure in areas of flood risk: lack of time series.
Surfacing: no clear record of proportion of multiple land use (such as residential gardens) that is impermeable and how this is changing over time due to urban creep.
No nationally available data on uptake of resilience and safety measures in actual developments (both properties and infrastructure).
Benefits of flood defence investment: lack of spatially aggregated data on change in flood likelihood for properties following completion of flood defence projects.
No data available on condition of non-EA flood defence assets or time series on condition of EA assets.
No national data on uptake of SuDS in new development or retrofitting of existing properties.
Data on uptake of flood warnings is not categorised by flood risk area, only for whole of the floodplain.
Data on availability of insurance is not accessible.
No data on disruption to critical services from flooding.

Update (13 July 2012):

The Sub-committee has drawn my attention to the supporting data and research released with Wednesday’s adaptation report.

This includes a results report and technical report from HR Wallingford on development of spatial indicators to monitor changes in exposure and vulnerability to flooding and the uptake of adaptation actions to manage flood risk in England.

Also of particular interest is a report from Royal Haskoning that assesses the economic case for property-level measures for flood risk in England.

Needless to say, this means the list of “areas for improvement” above really only scratches the surface on the subject of available flood data for England and Wales. I’m very impressed by the level of useful detail in the technical reports and by the amount of work and analysis that has gone into producing the adaptation report itself.

Freedom of Information and Charging for Public Sector Data

Fri, 29 Jun 2012 15:00:00 +0100

Prior to yesterday’s release of the Cabinet Office’s Open Data White Paper there was speculation, mainly in an article first published Wednesday evening on the Guardian website, that the UK Government would also use this occasion to announce a review of charging for information requested under the Freedom of Information Act.

It’s unclear whether the Guardian’s original draft was just based on a misunderstanding, or whether the Cabinet Office picked up on the initial negative responses and decided it would really rather not face the wrath of both open data campaigners and freedom of information activists on the same day.

Either way the Guardian article was substantially revised with comments from a Cabinet Office spokeswoman, and the original headline ‘Public services data to be published - but at a price’ was replaced by the much more congenial ‘Mass of government data on public services to be published’.

Although the Open Data White Paper was rather a damp squib for those of us hoping for some commitments on release of core reference data, it was apparently well-received by those concerned mainly with public sector transparency. The success of the launch event was also helped by a colourful revamp of the Government’s public data portal Data.gov.uk.

Had the Government decided to float a FOI charging proposal yesterday, the social media reaction would have eclipsed any positive coverage of the open data paper.

FOI charging therefore remains an outstanding and highly contentious issue for another day. Probably another day soon; we know that the Ministry of Justice, which for some reason has lead responsibility for the freedom of information brief, is keen on charging.

It should be said however that if charging for FOI responses and charging for re-use of public sector data are two ideas that are easy to confuse, the Government has only itself to blame. Recent legislative changes seem to have been designed to deliberately conflate requests for access to information with licensing to re-use that information, when the subject of the information request is a publicly-owned dataset.

Following changes to the Freedom of Information Act brought in last month (via Section 102 of the Protection of Freedoms Act), the Government has effectively combined the procedures for requesting FOI access to a dataset and requesting a licence to re-use that dataset.

Annex B in yesterday’s Open Data White Paper establishes clearly that the standard route for requesting re-use of any public dataset that is not already available is now to submit a FOIA request.

The FOIA changes include a new power to charge fees in relation to release of datasets for re-use. Although the original intention may not have been to allow charging for access alone, the actual language of the FOIA changes is suspiciously ambiguous. The provisions seem to require public bodies to prepare the data for re-use anyway, i.e. to do the work for which they could charge a fee, even if the requestor only wants to access the data.

The practical effect is that public bodies will have the option to charge for responding to a FOIA request whenever that request is for a public dataset, unless or until they receive clearer guidance to the contrary from Government.

Public bodies have not yet had formal guidance from Government on interpretation of these FOIA changes or on the appropriate shape of any charging policies. That may fall within the scope of new responsibilities given to the Information Commissioner’s Office. The ICO is expected to introduce a revised model publication scheme in April 2013.

Open data campaigners will hope that any such guidance re-establishes the presumption that public bodies should charge at most the marginal cost of preparing the data for re-use. Under the previous Government public bodies were (in principle at least) required to justify any charging policy that was not based on a marginal cost pricing approach.

P.S. I wrote a previous analysis of these problems back in February, when the Protection of Freedoms Bill was under scrutiny in Parliament.

Further information on Royal Mail's Positional Data Capture project

Wed, 16 May 2012 18:01:00 +0100

Subsequent to my post in March, a few additional details have emerged on Royal Mail’s Positional Data Capture project. The objective of this project is to collect location information for all Royal Mail delivery address points in the UK.

Last week a Communication Workers Union (CWU) representative posted a Deployment Guidelines document on the RoyalMailChat online message board. (Free registration required to view the full message.)

Following is an edited extract of key points:

Branches may be aware of the previous trial regarding the Positional Data Capture (PDC) initiative which took place in London, Cardiff and Lampeter last year. PDC is part of the new products and services strategy and will seek to establish Royal Mail as a major player in the positional data market; currently worth £100m a year and growing. This information is commercially attractive to Royal Mail who intends to sell the data captured to a host of organisations and companies who have shown an interest in obtaining this information.

Royal Mail, as the Universal service Provider for the UK, has a significant advantage over any other delivery or logistics company in that it has access to 28.2 million address points and has data systems such as A plus which capture every post coded delivery point in the UK. This information will be used to overlay positional data information, such as the address point and the delivery access point using satellite technology GPRS.

National Objectives

To capture all address points and access points circa 28.2m

To achieve data accuracy of c95% accurate to within <5m

To achieve 60% positive experience from the volunteers involved in PDC covering training and use of the technology.

To provide a commercially viable product and enhance Royal Mail’s portfolio of products and services

Locations

All Royal Mail Delivery Office locations will be covered.

Duration

The deployment will be divided into 2 phases, timeline highlighted below.

Anglia = 02/07/12 – 24/09/12

Thames Valley, South East, South West, London, Wales, Midlands, North West England, North East England, Scotland and Northern Ireland = 03/09/12 – 30/06/13

The programme outlined above is still provisional and is subject to change.

In addition to the above information, Nigel Pindar (Solutions Architect on the PDC project from May 2011 to March 2012) has posted the following note to his LinkedIn profile. (I’ve added the vendor links.)

Positional Data Capture (PDC) is a new Royal Mail product-offering, providing hitherto unachievable levels of GPS data accuracy for the UK’s 28.8 million addresses. It uses GPRS or WiFi to upload data from mobile devices to a PHP/Java, Windows, Apache & MySQL based Web Portal hosted on a PaaS infrastructure provided by BT.

This is a £10m project which will enable RMG to move into a completely new market, providing GPS data to a variety of new customers.

I authored both the Logical Design and Physical Design documentation, successfully taking them through RMG’s gated project management process and achieving sign-off on them.

I am RMG’s solutions/technical lead liaising with vendors; Trimble (the application vendor), BT, CSC, AFD and Accenture; ensuring that the solution components they’re supplying will integrate successfully with each other.

I’m assuming the above description applies only to the technology and vendors involved in the project to capture the data, and not to any proposed customer solution. It remains unclear what services or products Royal Mail plans to offer to the location intelligence market once it has successfully captured and cleansed the data.

It will be interesting to see whether Royal Mail decides to focus simply on production of address data sets for use in third-party software, i.e. in direct competition with the National Address Gazetteer (NAG) and Ordnance Survey’s AddressBase products, or whether it also intends to offer the data as part of its own software/service proposition.

I’m very skeptical that this project will produce a national geo-referenced address data set with higher positional accuracy than the existing NAG and AddressBase products. Any large data set produced from a mass labour effort is likely to be of variable quality. However it’s certainly plausible that the Royal Mail data will be competitive in much of the market for this type of information.

A particular selling point for Royal Mail will be the inclusion of address location data for Northern Ireland, since the existing available products only cover Great Britain. On the other hand Royal Mail is only collecting location data for postal delivery points, and missing out the 1.5 million or so “objects without postal addresses” included in the Plus and Premium versions of AddressBase. My current best guess is that Royal Mail will present its positional data as an added-value alternative for existing PAF customers and as a lower-price alternative for commercial licensees of the basic version of AddressBase.