Making life hard to make things easier

This is our particular take on principle four – Do the hard work to make it simple and it centres around probably the most controversial decision we made – at least in a particular corner of technically astute observers.

The questions is always variation on the following;

Why the heck did you build your own content management system rather than use Drupal/Wordpress/Joomla/Sitecore/Reddot/Liferay (the list is endless and you can delete as appropriate.)

This wasn’t a decision we took lightly and we looked hard at a number of the major open source CMSs out there but for what it is worth this was the thinking behind the decision – I’m not sure there is a right answer but I stand by what we chose.

We came to the project with some clear objectives and any publishing application was going to need to support them –

  • the ability to consistently publish within 60 seconds at 09.30
  • a platform that was optimised for continuous integration..
  • ..and automated testing
  • a secure platform where we had control as much as possible of the technical security
  • cloud ready
  • sustainable codebase..
  • ..and one we could recruit developers to support
  • an application that supported data visualisation as the primary visual language
  • a public website that was as performant as possible

Applications like Drupal (which I’ll use as the example as it was the closest to being used) come with a great many benefits as so much of the thinking around the standard use cases for web publishing has been done and tools built to support them. There is a large community and a ready pool of developers to get things working quickly and successfully.

The issue for us – and this is where you have to make a call and not everybody will agree – is that our main use cases for our application tended to be edge cases for the wider Drupal (and all the others) community. If we were to use something ‘off the shelf’ the team would have had to strip out a lot of un-needed / un-wanted functionality and add a fair amount of custom features – leaving only the skeleton of the original software and that created concerns about how maintainable it would be and started to dilute the advantages of using an existing product.

The feeling was we would quickly end up carrying weight (and risk) from a load of features we didn’t need and be committed to a product where the roadmap didn’t align to our needs which could potentially end up with us stranded on an old version due to the amount of customisation we would need.

Alongside this the teams we were talking to with the closest use cases to us (in particularly the GDS and Guardian development teams and later the Telegraph as well) had gone the DIY route.

The preference of the development team was to go with something lightweight, open source but bespoke – taking lessons from the GDS work but building something specific to our needs. It was about this time we started to talk about this as;

..a bespoke configuration of existing open source components

A decision needed to be taken and with the evidence at hand it was clear that there were strengths and weaknesses either way but given the preferences of the team, the work of GDS and the other organisations we spoke to with complex publishing operations I approved the direction to create ‘Florence & Friends’ which made up our platform.

It was certainly not a straightforward decision and created a LOT of pain for the team in the early days – there is a great deal of ‘water carrying’ going on in the background of modern CMSs and that all needed to be replicated but the freedom to be laser focused just on our user needs and to boil everything down to the MVP to get stats published allowed us to be build something and get it launched on schedule.

While currently far from perfect it has provided a real foundation for our ongoing ambitions while supporting the daily publishing schedule and successfully separating business process from web publishing.

The big question when you look back at difficult decisions like this is ‘would you do it again?’. Honestly there were times during the Beta when if you had asked me that the answer would have been a resounding ‘heck no!’ but with a little space now and looking at what was achieved I continue to stand by the decision and firmly believe it has provided us with something that will work better for our needs in the medium and long term in a way the other options wouldn’t have (though they may well have provided a less painful short term!)

2 comments on “Making life hard to make things easier”

  1. How has this new beta project impacted the future thinking for the ONS services in general? Knowing a number of people in various departments, everyone wants to be able to publish data online, but that doesn’t always match up with customer requirements, or field agents needs either.

    Is there a great deal of communication between the many, many departments who are conducting custom reporting for clients, the great work field agents do, and all the other diverse facets of the organisation?

    This is a project I’d loved to have worked on, and remember attempting to make contact with ONS digital years ago to discuss what was happening with making more data public, and free, and whether there was new projects like this happening.

    It’s great to see it is moving forward, I’d still love to work on it, but just knowing more about how it’s happening is good to know.

    1. Rather than this project impacting the organisation I think it was a product of a wider changing approach. ONS has become considerably more user focused in recent years and has worked hard to make more (larger) datasets available (it was always free and open licensed). There is still a long way to go though even on the public facing side of things. Our API (add /data to any page) needs extending and we need to increase support for geospatial and multi-variate data on the site. This is all coming – and the updated roadmap will soon be public as well.

Leave a Reply to Andy Parker Cancel reply

Your email address will not be published. Required fields are marked *