ONS Digital

Hello, I’m John Topley and I’m the Technology Lead for the Response Management (RM) product being developed at ONS. Response Management is part of our Survey Data Collection (SDC) platform. You might already have read on this blog about the good work the eQ team are doing on the electronic questionnaire product that is also part of SDC.

Response Management is the internal software that covers overall management of the survey across all survey modes such as online and paper. It manages the survey sample, tracks responses and initiates required follow-up actions during the collection period. During the next census, Response Management will manage over 35 million responses from households alone. However, it’s important to note that Response Management is being designed and built in a generic way so it can handle all of ONS’s surveys and not just the census.

The tenth GDS design principle states:

Make things open: it makes things better

Accordingly we have just open-sourced the work the RM development team have been doing under the ONS Digital account on GitHub. So far there are five open-source GitHub repositories that comprise the Response Management system, with more to follow.

The diagram below gives an overview of the Survey Data Collection platform and Response Management’s place within it:

The individual components of Response Management are described below.

• Response Management UI
  This repository contains the Response Operations application that is the back office user interface to Response Management. This is a Ruby Sinatra application. We’re using Sinatra because it allows us to develop and iterate the front-end very quickly. Another reason is that we don’t need the full-stack framework provided by Ruby on Rails because the user interface is a thin layer that simply makes RESTful requests to the Java web services underneath.

The Response Operations UI is rather straightforward (some might say old-school) in that it’s not a Single Page Application, it doesn’t use AngularJS or any other complex front-end JavaScript framework; it doesn’t even use AJAX. Nonetheless as the Post It from a recent retrospective shows, it’s a hit with our internal users (SEL is Survey Enquiry Line – a team within ONS’s Telephone Unit):

• Response Management Service
  This repository contains the Java microservices that are the heart of the Response Management system. These services communicate with each other over HTTP and JMS as appropriate. Each service has a dedicated database user and schema i.e. there is no database-level integration. The main services are listed below:

Case Service
The Case service is a RESTful web service implemented using Spring Boot. It manages cases and associated address frame data. A case represents an expected response from an address. Every address in the survey sample must have at least one associated case. A case has a case type such as Household or Individual.

Each case can have multiple questionnaires associated with it, but it must have at least one. Each questionnaire has a question set and a unique Internet Access Code (IAC). Interesting things that happen during the life cycle of a case are recorded as case events. Case life cycle transitions are published as JMS messages for interested parties to subscribe to. The Action service described below is one such party.

Action Service
The Action service is a RESTful web service implemented using Spring Boot. An action represents an operation that is required for a case. For example, posting out a paper form or arranging a field visit. Actions are grouped into action plans via a set of rules that define, in terms of a day, when a certain action should be taken. Action plans are applied to cases to create actions.

Once an action is created it is distributed to a remote handler service to be completed. An example of a remote handler service is the gateway that interfaces with the Field Work Management Tool (FWMT) to schedule field visits. Another example is the service that generates CSV files containing address data and IACs that a print provider mail merges to create the letters inviting participation in a survey. The Action service can receive feedback from remote handlers to change the state of an action.

• Common framework code for Response Management
  This repository contains common framework code for the Response Management Spring Boot applications, as well as reusable test framework code.
• Common configuration for Response Management
  This repository contains common configuration for Response Management. At the moment it just contains our Checkstyle configuration which has to be available on a publically-accessible URL so Travis CI can build our projects.
• IAC Service
  The IAC Service is a Java microservice for generating and validating the Internet Access Codes respondents use to access questionnaires. It’s another Spring Boot application. This repository will always remain closed-source due to its sensitive nature.
• SDX Gateway
  The SDX Gateway service will be a Spring Boot application that provides a single RESTful endpoint so Survey Data Exchange (SDX) can notify Response Management when a questionnaire has been receipted. RM will use this information to update the status of the associated case.

In addition to the components mentioned above, we have also open-sourced the non-Response Management code below:

• Respondent Home UI
  This Ruby Sinatra application is the user interface for the Respondent Home product. It allows users to validate their Internet Access Code and forwards them to the eQ Survey Runner upon successful validation where they can complete their questionnaire.
• JWE RubyGem
  This repository contains an RFC 7516-compliant JSON Web Encryption (JWE) token generator. It’s used by Respondent Home above to authenticate the respondent to the eQ Survey Runner.
• LDAP RubyGem
  This repository contains the code for a RubyGem (library) that can be used to authenticate against an LDAP directory. It’s used by the Response Operations application that is the back office interface to RM.
• Base Chef Cookbooks
  We build and deploy Response Management using the Chef automation platform. This repository contains general purpose Chef cookbooks that are non-project specific and thus potentially suitable for use on any development project.

I’m excited to share the work we’ve been doing. Response Management has already been used to successfully run the 2016 Census Test. We are now hard at work building out the functionality for the much more challenging 2017 Census Test, as well as tests Social Survey Division colleagues are running for their surveys such as the Labour Force Survey.