Information about the upcoming work on Customise My Data (CMD)

Categories: Data

Introduction

At ONS we have a beta service publishing some of our data in a more open format. This service, which is called Customise My Data (CMD), allows users to filter our datasets for just the data they are interested in. 

For example, let’s say you’re interested to know the number of people aged 50 and over in Cardiff. You can use our population filter to generate a custom spreadsheet with just that data filtered for you. CMD allows users to download this filtered data. More advanced users can get this same data in a more structured format to build automated apps and pipelines.

This autumn we are conducting a major piece of work to improve the stability of CMD. This will mean republishing all data held on the system.

This blog post sets out what is happening, why we’re doing this and what it means for our users.

What is happening?

We are changing the graph database that underpins our system. A graph database plots the relationship between different points in a network. 

To return to our example, a graph database stores the fact that Cardiff is within Wales and that Wales is within the United Kingdom. This is essential for users to navigate through our filters without getting lost.

Unfortunately the only way we can migrate from one graph database to another is to delete all data held on CMD and reload it.

Why are you doing this?

We’re doing this for several reasons:

  1. To save public money.
  2. The new system will require less maintenance.
  3. The new system will be more stable.
  4. It improves scalability (this means we can handle more users coming to the system).

As a user of CMD data, what does this mean for me?

Unfortunately it means that the service will be unavailable for a period of time while we delete and reload the data. 

We will aim to reload the data as quickly as possible to minimise downtime. We will prioritise the datasets we know users rely on the most heavily. It will mean that certain datasets will become available again sooner than others.

It will also mean that some of the URLs containing data will change because we are resetting the version numbers to one. Please see the list below to find the correct links for all the datasets that we will republish. 

These links will redirect to the latest version of the data on CMD. If you follow one of the links below and find it does not work, it is because we haven’t republished that dataset yet. Please check back again at a later time. Users who are comfortable reading JSON data can find a real-time list of all available datasets via our API.

It’s important to note that everything that is published on CMD is also published separately on our main website. While this piece of work is underway, all our statistics will still be available to access via our main website. You can visit our release calendar to see what ONS is publishing this week.

Thank you for bearing with us while we complete this work. If you have any further questions or comments please email customise.my.data@ons.gov.uk. 

Update:

This work has now been completed. Thank you for your patience.

List of datasets:

Tags: , ,