ONS Digital

A little while ago I blogged about our entry in to the Civil Service Innovation Challenge. Well yesterday I attended Civil Service Live in London and successfully won our semi-final against a team from BIS and then today I returned and was unsuccessful in the final. The winning team was the impressive ‘Child Abuse Image Database (CAID)‘ from the Home Office which was an impressive and important project wonderfully presented so congrats to that team and enjoy Silicon Valley!

My talk went a little haywire so here are the words I intended to say in the order I intended to say them.

In May 2013 Tim Harford dedicated an entire column in the Financial Times to pointing out just how bad the ONS website was. He wasn’t alone – multiple media outlets and even Parliament were openly unhappy and concerned with the failures of the website.

All the criticism opened the door a crack to try something new.

According to figures in our business case open and accessible ONS statistics could be worth £19.6 million to UK Plc not to mention the importance of some of the statistics to running the economy.

But one of the many problems with the current ONS website is that it is actually really quite difficult to find the statistics. This is because the data is often buried in Excel spreadsheets or PDFs with complex, if accurate, titles and when they are surfaced in the HTML it is unstructured and inconsistent. This is because the organisation has never really moved beyond traditional print publishing thinking – we are ON the web but we are not OF the web.

When we started thinking about what we could/should do next I kept coming back to 2 initiatives I was inspired by back in the naughties – the short-lived Open BBC catalogue and the launch of the Guardian’s OpenAPI.

What these ideas had in common was making the underlying, structured data available in machine readable formats. It wasn’t about creating a separate API for users but rather that ‘the website is the API’ and that everything else builds up from that – including your own UI – it was the embodiment of that horrible saying – ‘eating your own dog food’.

This seemed the right fit for us – the needs of our users were changing and we needed to provide a more flexible platform to support them now and in the future.

The question was how?

There was not much innovation going on elsewhere in the statistical community internationally we could learn from. So we turned to the open data community for help. To start with we worked with the Open Data Institute to prototype some ideas in code to see what might be possible. This work was invaluable and gave us a place to start but also identified some significant challenges we were going to face.

For a solution to one of those challenges we worked with data science specialists ScraperWiki to build an open source tool called DataBaker that allowed us to take all of the multitude of spreadsheets the ONS produces and transform them into something genuinely machine readable (one of the greatest fallacies is that hitting saving as CSV somehow creates something useful!)

This all leads to the JSON that underpins the whole new ONS website and is available to all users simply by appending /data to any page.

This has allowed us to provide functionality throughout the site that is customisable by users, developers, data scientists and us. Interactive charts and headline figures are auto-updated as soon as the data refreshes, we can provide feeds for all sorts of configurations of our data, our Release Calendar can be easily filtered and schedules can be subscribed to in your calendar of choice.

So how can Singularity University help us? It can inspire us to identify new uses for our data – helping us to prepare the organisation for the next generation of user needs – whether that be smart cities, machine learning, the API economy or new models of data privacy for users. It can arm us with the stories that will allow us to meet our aim of being a truly Data Intense, Design Simple organisation.