Weeknote 25
Well in the words of the great Ron Burgundy “that escalated quickly“.
Last week anyone who visitied, or attempted to visit, the ONS website would have been beset with frustrations. It was a very stressful few days and amongst the most difficult of my career. I can only apologise to anyone who was affected by the problems.
The nub of it was that a side effect of some changes we made over the course of the weekend created some significant instability in the site and in the end the only course of action was to ‘roll back’ the site to an earlier, stable version.
This was not a decision taken lightly, but in the end it was the best course of action open to us. The technical and digtial teams worked incredibly hard last week, particularly late in to the evening on both Monday and Thursday to resolve the problems. In the end we did get to a point where we were able to publish successfully on Friday and [touch wood] things have remained stable the last few days.
The ONS is a pretty risk averse organisation and the level of testing that went in to these changes reflected that, so such major problem really did catch us out. A team had also been in over the weekend to make sure the changes hadn’t had any ill effects and at that point things looked ok – though obviously something wasn’t quite right.
We have also erred on the side of caution and removed some of the recent improvements we have added, so we can fully audit and investigate the causes of the problems. This means for the moment things like the interactive times series content are unavailable.
We are still investigating the root causes and making plans for how we move forward now. As these things become clearer, I’ll share them on this blog. For now I can only apologise again for any inconvenience this caused you and for internal readers thank everyone in Titchfield and Newport who went above and beyond it getting things working again.