In part one we looked at our hosting platform and how we build, deploy and manage services in our test and production environments. And in part two we looked at our backend systems, and how we import, store, publish and export data.
In the third and final part, we’ll look at our frontend systems, and how we build internal tooling and web services for publishing and accessing our content.
Part 3: Our frontend systems
We have a few independent frontend products which work in very different ways.
The ONS website
The ONS website is mostly rendered server-side using a mixture of Go and Java – most of it is in two frontend components known as babbage and dp-frontend-renderer. We also have dp-frontend-router which is responsible for handling and routing all inbound web traffic.
Babbage is a monolithic service which contains most of our handlers/controllers, templates, and lots of other supporting code for making API calls, handling search and supporting our publishing process. We’re no longer actively developing on Babbage, so any new or updated website functionality is built using our new architectural approach (essentially, microservices with clearly defined purpose and responsibilities).
Our new frontend controllers typically belong to their own service – for example, our new “Filter a dataset” service is powered by dp-frontend-filter-dataset-controller – while the templates belong to the dp-frontend-renderer component. This has resulted in much cleaner code, and gives full ownership of specific functionality to a clearly defined service.
The developer site
The developer site is (or will be) built as static HTML pages and hosted on Amazon S3.
The current (prototype) developer site is built with DapperDox, but we’ve found that difficult to customise and extend for what we need, and it means running, monitoring and maintaining server side components for something that really isn’t very dynamic.
Our new beta developer site (built as part of the “Filter a dataset” service) uses Go templates to generate static HTML pages directly from our OpenAPI specs, and is published to Amazon S3 and hosted using CloudFront. We’ll be using progressive enhancement to add additional functionality where possible.
This significantly reduces our running costs, is far simpler to understand and maintain, makes updating it and running multiple versions much easier, and hopefully gives a nicer user experience. We’ve been user testing it, and had some really positive feedback, but we know we’ve still got more work to do before it becomes our production developer site.
Our content management system
Our CMS (Florence) is a client-side React app with some server-side proxying (in Go) to talk to our APIs (with a legacy bit built with jQuery).
Florence is used internally by our publishing, editorial and data visualisation teams for preparing content for publication, and by other parts of the organisation to preview and sign off publications.
It also means our users are typically in the same building as us, which makes it much easier to identify, diagnose and fix bugs, and gives us plenty of opportunities for user research and testing.
We’re planning on breaking this apart slowly, for example the functionality required for “Filter a dataset” has been built using React and lives outside the workspace, and we expect to do something similar when we take on the bulletin redesign work next year.
Our developer dashboard
We also have a “developer dashboard” which lives on GitHub, is entirely client-side, and uses Google Firebase for near-realtime updates from our Concourse servers. It actually uses AWS Lambda to fetch data from Concourse and write it to Firebase, but we’d like to integrate it directly into our CI pipelines.
It provides information on our daily releases and build pipelines, and was built as a 10% time project by our frontend engineers.
Although we haven’t had much time to focus on this, it lives permanently on wall displays in our office and already provides a huge amount of value.
What we’ve been doing
We’ve still got some code left over from the website alpha and beta projects from before the website launch. This is becoming a maintenance burden, so we’re working hard to refactor and replace as much as we can. That’s meant a considerable amount of refactoring work, building new services and APIs, and generally trying to clean things up a bit at a time.
But there’s still more surprises hiding in our frontend codebases that we haven’t found or had time to fix yet – things like HTML templates making backend API calls, search API clients which return formatted HTML rather than structured data, and monolithic web services which use unknown and undocumented web frameworks that don’t offer the flexibility or reliability we’d expect.
Over the past year we’ve been working on the “Filter a dataset” service and refactoring Florence to introduce React and remove a lot of the old legacy code. That means we haven’t been able to focus as much on the website or developer site as we’d have liked to, but lots of other discovery and alpha work has been going on to explore some of the future work we’d like to do (and some of it has already been applied to the existing website, like the redesigned bulletin headers).
At the moment these are all owned by a small team of frontend engineers, which means regular context switching, and increases the cognitive demand on our engineers when making changes which span multiple systems.
Some of the tools and services we use in frontend engineering:
- HTML5 and CSS3 – standard markup and styling languages
- Sass – CSS preprocessor
- React – a client-side web application framework
- Go – for some of our server side code, routing and templating
- BrowserStack – virtualised browser testing environment
- Puppeteer – browser-based acceptance testing in Chrome
- Selenium – browser-based acceptance testing in most browsers
Over the next couple of years we’ll be building on the work we’ve already done – continuing to refactor Florence, extending the beta developer site and getting the “Filter a dataset” service live. We’ll also be adding geography support to the ONS website and taking on the bulletin redesign work.
Our goal is to have our web and CMS products owned and maintained by separate teams. We’ve put a lot of thought into this, and we feel it will let us give each product the focus it deserves, and gives us more time for regular maintenance, bug fixes and feature delivery.
To achieve this, we’re also planning on scaling up our frontend engineering team – we’ve recently hired a new engineer who’ll join us in March, and we’re hoping there’s more roles to come in future!