ONS Digital

The following blog post is written by Andrew Phelps from our Social Surveys Transformation team.

Getting ONS social surveys online

An important feature of the transformation of our household and social data collection is a move towards using multiple data sources, with the balance moving increasingly to non-survey data sources and away from traditional surveys. Surveys will of course be required in some capacity, in particular to answer the “why” policy questions, but in the future we do expect non-survey data to be the primary source to meet our requirements.

When we do need surveys the expectation is that they will be mixed mode but “digital by default”, that is, online first. There has been a huge amount of activity in the last 12 months to understand the impact of moving social surveys online, using a prototype transformed Labour Market Survey as a testing vehicle.

A lot of this development work has been qualitative in nature, focussing on respondent document and questionnaire design using an innovative respondent-centric design, which is discussed from page 47 of the Survey Methodology Bulletin No. 78.

Additionally, two major quantitative tests have been undertaken.

Test 1 – July 2017

We mailed out to 38,000 households using an experimental survey design to answer some important questions – what proportion of households would take part online? What mailing strategies and materials would help maximise take-up? Maximising online take-up is very important in order to maximise the proportion of data collected in a relatively inexpensive collection mode.

We were encouraged that overall around 20% took part online (with no monetary incentive), and that if we sent invitation letters on a Wednesday in a brown envelope (rather than white), and sent three mailings about the survey, this could be increased to around 22 to 23%.

More detailed results of this test can be found in a report authored by Ipsos MORI (who managed the survey mailings and hosted the questionnaire online).

Test 2 – September 2017

Next challenge – what happens if we introduce an incentive? What is most effective in maximising response, and more importantly, what is cost-effective? We mailed out to a fresh 40,000 households, again with an experimental design fully described in the report for this test authored by Ipsos MORI. This test demonstrated that a heavily incentivised online survey could get a more than 30% response rate, but (against expectations) we found that sending a non-monetary incentive in the form of a tote bag to households in advance was the most cost-effective incentive.

So, a lot of testing activity, but we now have a clear picture of how to maximise online response, and how to cost-effectively use incentives. This will be vital learning as we move towards understanding more about how to integrate a face-to-face collection mode with the online mode, and how to retain households in later waves of social surveys.

Other testing and learning

Along the way we have learnt a lot about who takes part in surveys online, and bust a few myths in the process. For example, some of us thought introducing an online mode would mean we would have a more balanced sample in terms of age, with proportionately more young people doing it online. Not so! We have learnt that young people are a challenge to get responses from, regardless of mode. And actually, those aged 50 to 70 years are the most likely to respond online.

We have also been able to do a small test to see what proportion would do a “Wave 2” survey, and 6 in 10 did, which we’re really pleased with and this bodes well for future longitudinal testing.

Overall, gathering these metrics has been of vital importance to inform further development. We could have just relied on our perceptions of the best thing to do, but when perceptions and facts are out of line we know that survey design is not going to be as good as it could be.

The next challenge is to develop the prototype survey further and properly scope out the impact of a mixed-mode survey – what response rate might we get, and is the data that we get biased? More evidence needed!