The problem of non-response in population surveys on the topic of HIV and sexuality: a comparative study

Background: Throughout the Western world, response rates are falling in population surveys. The aim of the study was to explore for the presence of non-response bias in two Swedish research projects on HIV [‘HIV in Sweden’ (HIS11)] and sexuality [‘Sex and health’ (UngKAB09)]. Methods: The study used four data sets (two from each project), each of which had been generated using different methods. By comparing means and percentages on 15 items across the data sets, we explore the potential presence of non-response bias due to interest in the survey topic (topic salience), and discuss the suitability of two increasingly common methods for recruiting respondents: the pre-recruited probability web panel and the self-selected web survey. Results: While a higher proportion of the respondents in the HIS11 substudies had been tested for HIV and were inclined to perceive themselves as being at high risk of HIV infection, the respondents in the UngKAB09 substudies were on average more sexually active. Further, while there was little variation in the results between the HIS11 substudies (postal/web survey and pre-recruited web panel), there were some fairly large differences in the results between the UngKAB09 substudies (web surveys, one of which was based on a self-selected sample). Conclusion: The study concludes that (i) there are signs of non-response bias that may be due to topic salience, (ii) while care must be taken when using self-selection sampling methods, the pre-recruited probability web panel might provide a cost- and time-effective alternative for recruiting respondents in future population surveys.

Introduction

As part of the measures introduced to tackle the emerging AIDS epidemic in Sweden in the mid-1980s, a study was initiated in 1986 on the public’s knowledge, attitudes and behaviour in relation to HIV [‘HIV in Sweden’ (HIS)].¹ This population survey was based on a random national sample of 3900 individuals living in Sweden. The study has thereafter been repeated every third or fourth year, providing data that can be used as a basis for political decision-making in the area of HIV prevention, and for analysing possible changes in HIV-related attitudes and behaviour over time.

The HIS surveys conducted in the 1980s had response rates of around 70%—rates that are quite acceptable. Over subsequent decades, however, response rates declined, and the 2007 study included only 50% of the original sample of 6000 respondents.² Four years later, the response rate was as low as 25%.³ A similar trend has been experienced by other repeated sexuality surveys in the Nordic countries.^4–6 The problem of low response rates is not unique to sexuality studies, however. The recent steep decline in response rates has been observed by researchers from many different disciplines and countries,^7–9 and has resulted in a range of studies focused on specifying the nature of, and identifying possible ways of countering, this new ‘nonresponse challenge’.^10,11

Although most researchers would agree that low response rates pose new challenges to survey research, the magnitude of the problem is a matter of dispute. While some would argue that we have come to ‘the end of an era’ for population surveys, others maintain that no superior alternative to the population survey is currently available, and that we must learn to live with low response rates and to deal with the complex issues associated with them.¹² One such issue is that there is no necessary relationship between the rate of non-response and the degree of non-response bias.¹³ In fact, the potential bias resulting from a certain rate of non-response is likely to vary between studies. Second, because the degree of bias is dependent on the relationship between the likelihood of response and the variables that are analysed, there may be more variation in bias between the items within a survey than between surveys.¹⁴ Third, although non-response can lead to bias in many important statistics,¹⁵ the largest threat is that of bias in the estimated means, percentages and totals—statistics that are pivotal in surveys focused on public health and prevention.

There are currently two well-established approaches to dealing with low response rates. The first involves limiting non-response, and includes various techniques used to increase respondent participation during the data collection (e.g. advance information letters, incentives, multimodal data collection methods, reminders).^16,17 The second approach considers the employment of auxiliary information about the non-respondents (e.g. administrative records or paradata) to assess the degree of non-response bias and to reduce potential bias of this kind by using response propensity weights in the statistical analyses.^18,19

This article is not directly linked with either of these two approaches, but presents a somewhat different contribution to the discussion of non-response and its implications, by systematically comparing a number of estimates of the respondents’ knowledge, attitudes and behaviour in relation to HIV and sexuality from four data sets that were collected and investigated using different sampling and data collection methods. The study proceeds from the most recent (2011) ‘HIV in Sweden’ survey (HIS11), and compares some of the results from this project with those of another research project on sexuality and health (UngKAB09),²⁰ which was conducted using samples from largely the same population <2 years earlier. Although response rates were low for both research projects (around 25%), the projects share the strength of being based on two samples from the same population that were recruited and investigated using different methods. While the first substudy in both projects was based on a traditional postal and/or web survey of a probability sample from the general population, the second HIS11 substudy was based on a probability sample from a pre-recruited web panel register, which aimed to be representative, while UngKAB09 used a web survey administered at online communities, thus involving a self-selected non-representative sample of respondents. For both projects, the principal reasons for using multiple samples and methods were to provide the opportunity to cross-validate findings using two data sets, and to explore new and alternative approaches to data collection on the subject of HIV and sexuality.

The study presented in this article aims to explore for the presence of non-response bias in the data generated in the HIS11 and UngKAB09 research projects by comparing estimates of the respondents’ knowledge, attitudes and behaviour concerning HIV and sexuality. Because none of the data sets can be used as a ‘gold standard’, against which the remaining sets of data might be assessed, we use other rules of comparison. First, by contrasting the results from the two research projects (each represented by two sub-studies), we investigate the potential presence of one of the most common forms of non-response bias, i.e. bias due to certain individuals being more inclined to respond to surveys that cover topics in which they have a particular interest (topic salience)—in this case HIV (HIS11) and sexuality and health (UngKAB09).²¹ Second, by comparing the results from the two substudies within each project, we are able to discuss, from the perspective of non-response bias, the suitability of two respondent-recruitment methods that are increasingly being used in population surveys, namely, the pre-recruited probability web panel and the self-selected web survey.

Methods

The article is based on data from two large research projects that were conducted in Sweden at around the same time.^3,20 As mentioned above, the projects used two samples each, recruited from the same population. The HIS11 population survey included all individuals aged 16–49 years living in Sweden in 2011, while the UngKAB09 population included all youths and young adults aged 15–29 years living in Sweden in 2009. For the sake of comparison, the analyses presented below are based on reduced data sets that include all respondents aged between 16 and 29 years.

The HIS11 survey was based on two substudies: (i) a multimodal post- and web survey of an age-stratified random national sample selected via the SPAR register (a national register of the Swedish population), and (ii) a pre-recruited web panel survey based on a stratified random national sample drawn from an existing web panel register owned by the private data collection company Norstat.²² In the first of these substudies, only 25% of the original sample of 6000 individuals responded to the survey. The respondent sample for the second substudy consisted of 1500 web panellists from a large register comprising a random sample of the Swedish population. The panellists in the register were initially recruited by telephone interviews (with a response rate of 69%). They receive incentives for study participation (e.g. lottery tickets) and stay in the register for an average time of 18 months. The panel for this study was drawn by means of age-stratified random sampling, and also reflected the proportions of individuals in the Swedish population on the variables sex and region. In this second HIS11 substudy, the response rate was 23%.

The two substudies included in the UngKAB09 survey were as follows: (iii) a web survey of an age-stratified random national sample recruited via the SPAR register, and (iv) a self-selected web survey. The first substudy included a large sample of 24 000 individuals, of which 24% responded to the survey. Respondents for the second substudy were recruited via various types of Internet communities. These websites were selected by a panel of young people who identified a number of Internet meeting places that they judged to be ‘representative sites’ for youths and young adults. Before releasing the study on the sites, it was carefully marketed. Following a data collection period of 2 months, ∼10 000 young people had responded to the questionnaire. For an overview of the basic characteristics of the two projects, with their respective substudies, see table 1.

In the results section below, we present results from univariate analyses of the items that were identical across the two questionnaires. Although we present statistics on a number of socio-demographic respondent characteristics, our main focus is directed at the 15 items associated with HIV and sexuality. The results section does not include any significance tests (as our aggregated sample of 16 410 individuals would mean that even the smallest difference would be statistically significant). Instead, we focus on the broader patterns of similarities and differences in percentages and means across the substudies.

Results

The following three tables present results from univariate analyses of the respondents’ socio-demographic characteristics and their knowledge, attitudes and behaviour related to HIV and sexuality.

Comparison of socio-demographic respondent characteristics

Table 2 presents socio-demographic respondent characteristics for the four substudies.

Comparing the socio-demographic composition of the four samples, we find differences in the age distribution across the data sets, and the two HIS11 substudies include a smaller proportion of male respondents than the UngKAB09 substudies (35% compared with 44–51%). Because the variation in response rates across the age groups within each study was small (see table 1), the differences between the substudies in the age distribution are almost fully explained by the fact that the probability samples were stratified according to age, and that the HIS11 and the UngKAB09 studies used different stratification principles (see table 1). Due to the variation in the age distribution among the respondents in the four samples, all the figures presented in subsequent tables have been weighted in accordance with age.

Comparison of respondents’ knowledge and attitudes related to HIV and sexuality

Table 3 presents the respondents’ knowledge and attitudes related to HIV and sexuality. As regards the knowledge items, the numbers are similar across the four substudies. If we look at the two items on which there is some small variation, the results are slightly contradictory. While a larger proportion of the UngKAB09 respondents correctly recognize the benefits of using a condom to prevent HIV transmission (94–96% compared with 90–92%), a larger proportion of the UngKAB09 respondents also believe that HIV can be transmitted by kissing (19% compared with 15–16%). Looking to the attitude items, two results are particularly noteworthy. First, by comparison with the UngKAB09 respondents, the HIS11 participants agree somewhat more often with the statement that sexual intercourse should occur only in stable relationships (mean = 3.9–4.0 compared with 4.5–4.6). Second, a larger proportion of the HIS11 respondents perceive themselves to be at risk of becoming infected with HIV (3% compared with 1%) or chlamydia (8–11% compared with 4–6%).

Comparison of respondents’ behaviour related to HIV and sexuality

Table 4 presents the respondents’ HIV- and sexuality-related behaviour. Table 4 shows that, on the whole, the respondents in the UngKAB09 substudies are more sexually active than the HIS11 respondents. Among the former, we observe a higher average number of sex partners during the past year (mean = 2.2–3.1 compared with 1.3–1.6), a larger proportion of individuals with more than five sexual partners during the same period (7–14% compared with 3–4%) and a larger proportion of individuals who experienced sex on their first date (26–41% compared with 17–20%). Further, if we compare the results from the two UngKAB09 substudies, we see that the self-selected respondents are far more sexually active than those selected by probability sampling. Interestingly, although the UngKAB09 respondents are more sexually active than the HIS11 respondents overall, they are, on average, equally inclined to use a condom during sexual intercourse (23–25%). As regards the questions about using the Internet for flirting or to search for love, the prevalence of these activities was much higher among the self-selected respondents who had accessed the questionnaire at online communities than it was among the respondents in the three substudies based on probability sampling (64% compared with 36–39%; 47% compared with 22%).

Discussion

The study’s aim was to explore for the presence of non-response bias in the data from two Swedish research projects on the topics of HIV and sexuality (HIS11 and UngKAB09). While all the results are based on a descriptive comparison of estimates from four data sets, none of which can be used as a ‘gold standard’, a number of results nonetheless support the contention that salience bias, i.e. bias due to some individuals being more inclined to respond to surveys on topics in which they have a particular interest, may be present in the data to some extent. For example, while a larger proportion of the respondents in the ‘HIV in Sweden’ surveys (HIS11) had been tested for HIV during the past 12 months, and were inclined to perceive themselves as being at high risk of becoming infected with either HIV or chlamydia, the respondents in the ‘Sex and health’ surveys (UngKAB09) were more sexually active, in terms of both the average number of partners during the past 12 months and having experienced sex on the first date. In addition, the UngKAB09 respondents disagreed more often with the statement that sexual intercourse should occur only in stable relationships.

Further support for the view that topic salience plays a role in explaining survey participation is provided by results from the HIS11 non-response study, where 20% of a random sample of 450 non-respondents reported having declined to participate due to a lack of interest in the topic (the non-response study was conducted via telephone interviews and included questions about socio-demographic characteristics and reasons for not having participated in the survey). However, it is also worth noting that for the remaining non-respondents, factors not associated with the survey topic—such as lack of time or energy—constituted the reasons given for not responding. Nonetheless, if both the HIS11 and the UngKAB09 projects suffer to some extent from a systematic non-participation of individuals who are less interested in the survey topics, another tentative conclusion from this study would be that careful assessment is needed of which items may and may not be affected by non-response bias as a consequence of this. For example, although we need to be careful in drawing overly far-reaching conclusions regarding the validity of the estimates that were similar across the sub-studies, it is a fact that several of the estimates for the items relating to the respondents’ knowledge about the transmission of HIV were almost identical across the four data sets.

One potential future strategy for avoiding non-response due to a lack of interest in the topic of HIV might be to start integrating this specialized survey into more general public health surveys. Such a strategy might also counteract the potentially refined effects of salience bias, which are linked to the actual meanings and implications of topic relevance or interest. In fact, it has been shown that while topics that are associated with positive feelings and memories may trigger survey participation, those that are equally relevant to a particular respondent, but which are associated with negative feelings and memories, may decrease the likelihood of participation.²³

The comparison of estimates from the two HIS11 substudies showed that the results from the pre-recruited web panel survey were largely similar to those of the traditional postal/web survey, with only a few small variations being found between the substudies. Although the observed similarities across the two sub-studies cannot be used as evidence for the contention that the results constitute valid estimates of the population parameters (as both substudies may suffer from the same forms of non-response bias), one conclusion from this study is that the pre-recruited web panel based on probability sampling may constitute a cost- and time-effective alternative to traditional respondent-sampling methods. One advantage of using a sample from a web-based register is that it might be possible to acquire additional relevant information about the non-respondents—information that could subsequently be used for response propensity weighting in the statistical analyses.²⁴

As regards the comparison of estimates from the two UngKAB09 substudies, some substantial variation in estimates was noted between the substudy based on a probability sample and the substudy based on a self-selected sample—the respondents in the latter comprised a much more sexually active group of young people, and a much larger proportion of these respondents had experience of using the Internet for flirting or to search for love. One conclusion based on this finding, and which corroborates those of numerous previous studies, is that although self-selection sampling may provide an easy and inexpensive procedure for recruiting many respondents in a short time, such methods are likely to produce biased samples and must therefore be used with care in population surveys.^25–27 Taking this into account, it is worth noting that web surveys based on self-selected samples have proved to be particularly suitable for studies that examine so-called ‘hidden populations’, such as men who have sex with men.²⁸ At the same time, if the aim is to sample hard-to-reach populations, it might be preferable to use respondent-driven sampling—a method that resembles the non-probability snowball sampling method, but which makes use of network theory, physics, statistics and mathematics to reduce the biases associated with snowball sampling.²⁹

Two limitations of this study are that it is explorative rather than experimental in design, and that the conclusions are based on observations of similarities and differences across four data sets, which might all be biased due to non-response and the use of self-selection sampling. Despite this, our results and conclusions regarding the potential presence of non-response bias in two large, recent research projects on the topic of HIV and sexuality provide important insights for researchers who are willing to work hard in order to meet the non-response challenge.

Funding

This project was supported by the Swedish Institute for Communicable Disease Control (grant codes 359/2010 and 01-12380/2008), Stockholm, Sweden.

Conflicts of interest: None declared.

Acknowledgements

The authors would like to thank Margareta Forsberg, PhD, and Jonna Abelsson, BSc, for the development of the UngKAB09 questionnaire and for the collection of the UngKAB09 data.

References