a blog by Owen Boswarva

2 Apr

Post: 2 April 2014

Back in February I wrote a post about UK open data funding programmes. I had some outstanding questions, mainly about the £1.5 million in “Release of Data Fund” allocations announced on 21 February, so I followed up with a Freedom of Information request to the Cabinet Office.

In the FOI request I asked for the individual amounts allocated to each proposal, details of who received the money, and also for a copy of any documents that set out the most recent criteria for eligibility, selection or prioritisation of bids for allocation of amounts from the fund.

Cabinet Office has now responded to my FOI request with a note of the context and process around the Release of Data Fund, and placed a list of the individual allocations on its website.

Yesterday the Cabinet Office and BIS released a joint press release about the Release of Data Fund and the separate Breakthrough Fund, along with an application form for organisations who want to apply for funding from the 2014/15 rounds. There’s also a Cabinet Office blog post.


So have we learned anything new?

My earlier post seems to have been broadly correct as an outline of the various sources of open data funding.

Neither the February announcement nor the statement yesterday represent new funding for open data. However it is useful to have official clarification of what’s available, and a single point of entry into the application processes for the two main funds.

Cabinet Office has confirmed that the Release of Data Fund is the previous DSB fund under a new name, but it’s clear there has been a substantial drift in the stated purpose of that fund (as discussed in my earlier post).

Although there are still some differences, the purposes of the Release of Data Fund now closely resemble those of the Breakthrough Fund administered by BIS. For the 2014/15 funding rounds Cabinet Office has basically adopted (with some refinements) the application process that had already been established for the Breakthrough Fund.

How were the 2013/14 Release of Data Fund allocations awarded?

This is the problematic area. It’s apparent from the Cabinet Office note that the process of collecting bids for the 2013/14 funding round was informal and unpublicised.

Cabinet Office says the ODUG (an advisory group of volunteers selected by Cabinet Office) collected bids starting in January, which suggests a very short window of opportunity. According to a Yorkshire Post story on the Leeds Data Mill allocation, “civil servants invited founder Mark Barrett to pitch for the funding.”

I don’t know enough about public sector procurement to judge whether this process broke any rules — some of the allocations are internal to government, and the others may be below the relevant financial threshold for competitive tendering. In any case Cabinet Office has now put in place a public application process for future funding rounds, so there’s probably little to gain by getting Margaret Hodge involved.

However the optics are pretty bad. Given the uninspired nature of the most of the bids approved for 2013/14, I am sceptical whether ODUG tried very hard to canvass the open data community for worthwhile projects or ideas. It rather looks as if Cabinet Office just wanted to make a big funding announcement for Open Data Day.

How open is the 2014/15 Release of Data Fund application process?

Making an application form available online is obviously preferable to the unpublicised process used to allocate the first £1.5 million from the Release of Data Fund. However I have several concerns:

The deadlines for applications are very short: 14 May 2014 and 16 July 2014. Worse, those deadlines are buried in the application form and have not been highlighted on the landing page.

Eligible projects have to be finished by March 2015. I struggle to find any rationale for this, other than the obvious political benefit to the Government of being able to announce completed projects prior to the General Election.

It’s also unclear how the increasingly neglected data request process on fits in with the new Release of Data Fund application process. The ODUG involvement in both seems to be similar. But now that the original ambition to “buy back” important datasets from the trading funds has been largely abandoned, is there any point in contributing to business cases for release of those datasets?

The Release of Data Fund, like the Breakthrough Fund, may be useful for helping public authorities over the smaller hurdles. However Cabinet Office still seems to lack a strategy for addressing the real barrier to open release of important public datasets: the trading fund model and other existing revenue-driven licensing operations within government.

Image credits: money texture derived from an OKFN Open Buttons image, CC BY 3.0; CGI render by me.

17 Mar

Post: 17 March 2014

This morning the Public Administration Select Committee (PASC), a cross-party group of MPs chaired by Bernard Jenkin, published its report on Statistics and Open Data.

This report is the product of an inquiry launched in July 2013. Witnesses gave oral evidence in three sessions; you can read the transcripts and written evidence as well.

The PASC press release leads with criticism of the Government’s decision to let the Postcode Address File (PAF) out of public hands by including it in the recent sale of Royal Mail. The report says:

"The PAF should have been retained as a public data set, as a national asset, available free to all, for the benefit of the public and for the widest benefit of the UK economy. Its disposal for a short-term gain will impede economic innovation and growth."

"Public access to public sector data must never be sold or given away again."

An open national address dataset is the closest thing the UK’s open data community has to a unifying cause célèbre, so this is a message that will find a receptive audience.

However PASC has also dug quite deeply into the core problems with the UK Government’s open data policy. While I don’t agree with every aspect of the analysis, I was pleasantly surprised by the extent to which PASC has taken on board the evidence submitted to its inquiry.

The report is broadly supportive of the arguments for open data, and makes some helpful recommendations for improving government policy. Below I have picked out what I think are some key points.

Open data fundamentals

PASC has embraced the principle of ‘open data by default’:

"There should be a presumption that restrictions on government data releases should be abolished. It may be necessary to exempt certain data sets from this presumption, but this should be on a case-by-case basis, to provide for such imperatives as the preservation of national security or the protection of personal privacy."

and seems to have been persuaded by the arguments against charging for public data:

"It is short-sighted in the extreme for Government to seek to maximise fee income from data while those fees penalise in particular small companies that can prove the most innovative, and which could establish the UK as global leader in this new economic sector."

"Charging for some data may occasionally be appropriate, but this should become the exception rather than the rule."

It has recognised that disruption of existing information markets is not a good excuse for avoiding open release:

"It is also acknowledged that wider access to more data and information will be disruptive to the structure of existing markets, leading to some firms winning and some firms losing. But our evidence suggests that, in all probability, consumers will gain."

PASC also endorses Stephan Shakespeare’s proposal that the Government should adopt a ‘twin-track’ approach to data release, as:

"a practical and realistic way of maintaining the momentum on open data, which recognises that ‘the perfect should not be the enemy of the good: a simultaneous "publish early even if imperfect" imperative AND a commitment to a "high quality core"’."

The ‘right to data’

PASC says there is an inherent ‘right to data’ and that the Government should bring forward legislation:

"The Government needs to recognise that the public has the inherent ‘right to data’, like Freedom of Information. The Government should clarify its policy and bring forward the necessary legislation, without delay."

Unfortunately the Committee does not make it clear what it considers the ‘right to data’ to be. I am not sure PASC has understood the distinction between the ‘right’ to access data in a reusable format (provided for in last year’s changes to the Freedom of Information Act), and the idea of an information rights framework for open data itself, which is what several witnesses argued for.

I have written previously about the new dataset provisions in FOI and why they don’t go nearly far enough. It would have been more helpful if PASC had expressed clear support for a legal right to open data (by default at least). However it has at least recognised that there is a need for additional legislation of some kind.

Public sector procurement and outsourcing

PASC recognises the complications for open data created by contracting and outsourcing of public services:

"Open data principles should be applied not only to government departments but also to the private companies with which they make contracts.

"We recommend that companies contracting with the Government to provide contracted or outsourced goods and services should be required to make all data open on the same terms as the sponsoring department. This stipulation should be included in a universal standard contract clause …"

This is along the right lines, though my own preference would be to specify a requirement that IP ownership of data created from the contract will be retained by the sponsoring department. That approach would support future decisions to release open data, rather than just following from practice within the sponsoring department at the time of contract negotiations.

Who is responsible?

PASC highlights “a lack of coordination on open data at Ministerial and official level, though this is improving.” Using the PAF decision as an example, it says:

"The Cabinet Office leads on the policy, but its mechanisms to hold Departments to account are weak …"

"Despite the enthusiastic rhetoric emanating from the Cabinet Office, our evidence indeed indicated something more serious - a lack of understanding of open data among most Ministers and apparently most officials."

and also notes:

"There is an unwieldy plethora of open data bodies which tends to slow both decision-making and consultation."

The report casts doubt on the early focus of the current Government’s open data strategy:

"There is no sign of the promised emergence of an army of armchair auditors. There is little or no evidence that the Cabinet Office is succeeding in encouraging greater public engagement in using data to hold the public sector to account."

PASC says open data needs to be treated “as a major government programme in its own right”, and recommends that:

"The Minister for the Cabinet Office should be given explicit responsibility for all aspects of open data policy, including the commercial aspects.”

This is an attractive idea but I’m not sure it’s plausible. The Cabinet Office’s lack of influence over delivery departments is long-standing and entrenched. Formalising responsibility for open data policy will not overcome that problem.

I cannot see BIS or the Treasury relinquishing their influence over the commercial aspects of open data policy, particularly as those aspects are less about open data itself and more about conflicts with other policy agendas such as privatisation.

I did like this remark, though:

"The Transparency Strategy Board is too large to be effective in driving progress. A small group from that Board should work as a Programme Implementation Board."

Perhaps we should call that smaller group the Data Strategy Board? (Regular observers of UK open data policy will understand the joke.) and measuring open data progress gets a bit of a kicking in the PASC report; not so much for the development side but for the underlying concepts and the Government’s tendency to use the site to meet the needs of Whitehall rather than those of data users:

"It is often pointed out that more than 13,000 datasets can now be found on, but it is unclear how many of these represent simple republishing of data already published on other government sites. Some data sets are small and others large. And it is possible for departments to get more data out by publishing it in smaller bundles or updating it more frequently, in such a way that there is little or no extra public benefit. In these circumstances, measuring progress on this important agenda is difficult if not impossible."

This makes refreshing reading, given the number of times Ministers and senior civil servants (and more often then not journalists as well) have cited the number of datasets listed on as if that were a meaningful metric for something.

PASC recommends a few alternatives. One is:

"We invite the Government to publish a clear list of open data, indicating when each data series became open in each case."

This is a lovely idea but hopelessly impractical at this point. The Cabinet Office has had (and continues to have) great difficulty pulling together a cross-government inventory of public data assets. I doubt it will have much appetite for persuading departments to unearth the publication history of all the open data now listed on

PASC also recommends the adoption of not one but two five-star rating systems:

"We recommend above that the Government adopt the ‘five-star’ system along the lines proposed by Involve, for open data engagement. A second ‘five-star’ rating system, developed by Full Fact for assessing the usability of government statistics, would support the efforts of statisticians to play a more active role in open data. This system should also be adopted by the Cabinet Office in assessing departmental progress on open data.”

These two systems are presumably in addition to the original five-star deployment system for linked open data devised by Tim Berners-Lee back in 2006, and the variant of that system already implemented (with questionable success) on

My concern is that more of this kind of stuff will just feed the Cabinet Office’s managerialist tendencies …

This PASC recommendation is rather better:

"The Cabinet Office must give a much higher priority to ensuring that more interesting and relevant data is made open, and that the release mechanisms encourage people to use it and, where appropriate, hold Government and local authorities to account. Beginning in April 2014, targets should be set for the release of totally new government datasets not the republishing of existing ones."

Simple when you think about it, isn’t it? We should judge the progress of UK open data policy on the steady release of new and useful public datasets.

Photo credit: Big Ben by Carlesmari, CC BY 3.0

12 Mar

Post: 12 March 2014

These are the current best primary sources for bulk address data for GP practices in the UK, as far as I am aware:

England and Wales

GPs, GP Practices, Nurses and Pharmacies (HSCIC)

Format: CSV

Licence: OGL

England only

GPs (NHS Choices)

Format: CSV

Licence: OGL

Wales only

GP Practice Analysis (NHS Wales Shared Services Partnership)

Format: XLS

Licence: unclear


Practices and Their Populations (ISD Scotland)

Format: XLS

Licence: unclear; seems to require specific permission

Northern Ireland

GP Practice Lists (NHC Business Services Organisation)

Format: XLS and PDF

Licence: OGL


Photo credit: The Jericho Health Centre, Oxford by Kaihsu Tai, GNU Free Documentation License

3 Mar

Update: 9 April 2014

The Integrity Guide has now also been added to the GOV.UK page.

Post: 3 March 2014

I have obtained a copy of the Flood Warning Data Integrity Guide, a technical document intended for anyone who uses the Environment Agency’s live flood warning data (either the live XML feed or Flood Alert/Flood Warning Polygons).

You can download the Integrity Guide (in a zip file with the EA’s Standard Notice) here:

Flood Warning Data Integrity Guide

The feed and polygons were recently (and temporarily) made available as open data by the Environment Agency.

The Integrity Guide was released by the EA in response to a Freedom of Information request.


Photo credit: Flooding in Tewkesbury by Cheltenham Borough Council, CC BY-NC-ND 2.0

2 Mar

Post: 2 March 2014

Last week Guardian technology journalist Charles Arthur published an article about the prospects for open data release of Environment Agency flood mapping data:

Environment Agency poised to open flood data to public

There are some comments from me in the article as well as “below the line”.

I’ve created a Storify of Twitter comments reacting to the article.

For more on this subject see my post from last month.


Photo credit: Floods in Hull June 2007 by Maggie Hannan, CC BY 2.0

Page 1 of 14