The effect of multiple reminders on response patterns in a Danish health survey

Abstract

Background: Reminders are routinely applied in surveys to increase response rates and reduce the possibility of bias. This study examines the effect of multiple reminders on the response rate, non-response bias, prevalence estimates and exposure–outcome relations in a national self-administered health survey. Methods: Data derive from the Danish National Health Survey 2010, in which 298 550 individuals (16 years of age or older) were invited to participate in a cross-sectional survey using a mixed-mode approach (paper and web questionnaires). At least two reminders were sent to non-respondents, and 177 639 individuals completed the questionnaire (59.5%). Response patterns were compared between four groups of individuals (first mailing respondents, second mailing respondents, third mailing respondents and non-respondents). Results: Multiple reminders led to an increase in response rate from 36.7 to 59.5%; however, the inclusion of second and third mailing respondents did not change the overall characteristics of respondents compared with non-respondents. Furthermore, only small changes in prevalence estimates and exposure–outcome relationships were observed when including second and third mailing respondents compared with only first mailing respondents. Conclusions: Multiple reminders were an effective way to increase the response rate in a national Danish health survey. However, when differences do exist between respondents and non-respondents, the results suggest that second and third mailings are unlikely to eliminate these differences. Overall, multiple reminders seemed to have only minor effect on response patterns and study conclusions in the present study.

Introduction

The aim of most surveys is to obtain estimates that can be generalized to a population of interest. Nevertheless, most population surveys have some level of non-response and, hence, potentially biased survey estimates.^1,2 Survey estimates are widely used in public health surveillance, planning and prioritizing, and also when quantifying exposure–outcome relations. The validity of survey estimates is, therefore, important, and the response rate is considered a central indicator of data quality. For a number of years, response rates in health surveys have been diminishing, which can be problematic, as it means that a progressively smaller proportion of the population represents the majority.² Different strategies to improve response rates in self-administered surveys have, therefore, been suggested: e.g. reducing questionnaire length, offering incentives and using reminders.³ However, achieving a high response rate is often expensive and time-consuming, and some studies show that higher response rates do not necessarily provide different or more accurate results.^4,5 Furthermore, some studies show that even though prevalence estimates might be influenced by non-response, associations are not necessarily affected.^6–8 Overall, the effect of multiple reminders on response patterns has not been thoroughly investigated. Hence, the aim of this study is to examine the effect of multiple reminders on response rates, non-response bias, prevalence estimates and exposure–outcome relations in a large national self-administered health survey. Non-response bias is defined as the kind of bias that occurs from differences between responders and non-responders, i.e. data are not missing at random. In the current study, non-response bias is, first, assessed by comparing respondents and non-respondents with regard to socio-demographic factors and register-based health indicators, and, second, by comparing prevalence estimates and associations among first, second and third mailing respondents.

Methods

The Danish National Health Survey 2010

The data for this study derive from the Danish National Health Survey 2010 (DNHS-2010). The authors of the current study have been involved in the design and data collection for DNHS-2010. The survey was based on six mutually exclusive random subsamples: one in each of the five Danish regions and one national sample. The total sample consisted of 298 550 individuals aged 16 years or older and resident in Denmark on 1 January 2010. A standard questionnaire with 52 core questions was used in all six samples. In addition, a variable number of different questions were added to the standard questionnaire in each sample. The complex study design has previously been described in detail.⁹ The sample was randomly drawn from the Danish Civil Registration System (each individual has a unique personal identification number). All selected individuals received a letter of introduction that briefly described the purpose and content of the survey, and it was emphasized that participation was voluntary. The letter invited the selected individuals either to complete a web questionnaire or to fill out the enclosed paper questionnaire. A reminder procedure involving two postal reminders was used, with the second containing a new paper questionnaire, in five of the samples. In one sample (The Central Denmark Region), a reminder procedure involving three reminders was used, with the last two containing a new paper questionnaire. By the end of the survey, 177 639 individuals had completed the questionnaire. All invited individuals were subsequently categorized into four groups: individuals responding after the initial invitation (first mailing respondents), individuals responding after the first reminder (second mailing respondents), individuals responding after the second reminder (third mailing respondents) and, finally, non-respondents. In the sample using three reminders, the individuals responding after the last reminder were classified as third mailing respondents owing to the low number of respondents (n = 2517). A total of 249 respondents were excluded owing to missing information on the response date. The four groups were compared with respect to socio-demographic characteristics and register-based health indicators (e.g. prescribed drugs and contact with a general practitioner), and the three groups of respondents were compared with respect to survey-based prevalence estimates (e.g. good self-rated health and daily smoking) and exposure–outcome relations.

The standard questionnaire in the DNHS-2010 included, among others, questions on health-related quality of life, health behaviour, morbidity and social relations. For the purpose of this study, 12 central health indicators were chosen to examine the effect of multiple reminders on prevalence estimates and exposure–outcome relations. A detailed description of the indicators can be found in the Supplementary Appendix.

Register-based data

In Denmark, nationwide administrative registers are available for research purposes, and owing to the unique personal identification number assigned to each resident, it is possible to link information from several registers at the individual level. The linkage in the present study has been approved by the Danish Data Protection Agency (reference number: 2010-54-0864) and all local confidentiality and privacy requirements have been met. Information on the respondents’ sex, age and marital status was extracted from the Danish Civil Registration System.¹⁰ Information on citizenship, place of birth and parents’ place of birth was likewise extracted from the Danish Civil Registration System and was used to define three ethnic groups: Danish background, other Western background (those from EU-member countries, Andorra, Iceland, Liechtenstein, Monaco, Norway, San Marino, Switzerland, Vatican City, Canada, the USA, Australia and New Zealand) and non-Western background (those from all other countries). Level of education at the individual level was extracted from Danish education registers, which are generated from the educational institutions’ administrative records.¹¹ All invited individuals were also linked to the Danish National Patient Register,¹² which holds administrative data (e.g. the hospital ward and date and time of activity) and clinical data (diagnoses and surgical procedures) on all patients in Danish hospitals, as well as to the Danish National Health Service Register,¹³ which holds information on activities of health professionals contracted by the tax-funded public healthcare system. Finally, we linked all invited individuals to the Danish National Prescription Registry, which contains information on dispensed prescriptions.¹⁴

Statistical analyses

Differences in register-based health indicators between respondents and non-respondents were analysed using the χ² test. Multiple logistic regression analyses were used to examine exposure–outcome relationships, and results are presented as odds ratios (OR) with 95% confidence intervals. Weights that accounted for differences in selection probabilities were applied in all analyses. Thus, individuals living in municipalities with a lower selection probability were given a higher weight. Accordingly, individuals living in municipalities with a higher selection probability were given a lower weight. All invited individuals in a particular municipality were given the same weight. Hence, municipalities are weighted to their actual proportion in the Danish population. All analyses were carried out using SAS version 9.3.

Results

A total of 109 479 individuals were classified as first mailing respondents, 40 575 as second mailing respondents, 27 336 as third mailing respondents and 120 911 as non-respondents. The cumulative response rates after the first mailing, second mailing and third mailing were 36.7, 50.3 and 59.5%, respectively. A lower return rate for each mailing was observed: i.e. the first mailing yielded a return of 36.7%, the second mailing a return of 21.5% and the third mailing a return of 18.4%. The relative distribution of socio-demographic characteristics among respondents, non-respondents and all invited individuals is summarized in table 1. In general, younger individuals, males, unmarried individuals, individuals with a non-Western background and individuals with a basic school education were underrepresented among the respondents as compared with non-respondents. Likewise, third mailing respondents differed from first mailing respondents in terms of the same socio-demographic characteristics. The socio-demographic distribution among third mailing respondents was very similar to the distribution among all invited. However, the overall cumulative socio-demographic distribution did not change substantially with multiple reminders.

Table 2 displays the 3-month prevalence of register-based health indicators among respondents, non-respondents and all invited individuals. It illustrates that higher proportions among respondents than non-respondents have been prescribed drugs, had contact with a general practitioner and had contact with a practising specialist physician. On the other hand, higher proportions among non-respondents than respondents have been admitted to a hospital. Similar differences were observed when comparing third mailing respondents with first mailing respondents. With respect to survey-based prevalence estimates, table 3 illustrates that first mailing respondents reported somewhat better self-rated health and health-related behaviour than second and third mailing respondents did. For instance, first mailing respondents were more likely to report excellent, very good or good self-rated health, and a daily intake of fruit, and less likely to be daily smokers and sedentary in leisure time compared with third mailing respondents. Nevertheless, the differences observed when comparing prevalence estimates between first mailing respondents and all respondents were generally small. Hence, even though differences were found between first and third mailing respondents, the inclusion of second and third mailing respondents had only a minor effect on the overall prevalence for the 12 selected health indicators. Separate analyses were conducted for the fourth mailing group (n = 2517). The prevalences among the fourth mailing group were very similar to the prevalences among the third mailing group. No systematic differences between the groups were observed (data not shown).

To assess whether the difference in prevalence estimates between mailing groups could be explained by differences in the relative distribution of sex, age, education and marital status, the prevalence by mailing group was further stratified by these socio-demographic indicators for three health indicators: excellent, very good or good self-rated health; daily smoking; and long-standing illness. In all educational groups, third mailing respondents were worse off than first mailing respondents. For example, among individuals with basic schooling, the proportion with excellent, very good or good self-rated health was 81.6% among first mailing respondents and 75.6% among third mailing respondents. The corresponding proportions among individuals with a higher education were 92.2 and 89.0%, respectively (data not shown). Likewise, third mailing men and women were worse off than first mailing men and women, respectively. The same pattern was seen with regard to age and marital status. The examined exposure–outcome relationships by mailing group are presented in table 4, with OR adjusted for sex and age: e.g. first mailing respondents with a long-standing illness had 0.08 lower odds of having excellent, very good or good self-rated health than first mailing respondents without a long-standing illness. Among all respondents, the corresponding odds were 0.09. In general, the observed relationships were virtually identical when comparing all respondents with only first mailing respondents, and no uniform tendency in the OR became evident with repeated mailings.

Discussion

This study examines the effect of multiple reminders on the response rate, non-response bias, prevalence estimates and exposure–outcome relations in a national self-administered health survey among individuals aged 16 years or older in Denmark. We found that multiple reminders increased the response rate, but that the socio-demographic distribution and the 3-month prevalence of register-based health indicators did not change substantially with the number of reminders. Additionally, only small changes in prevalence estimates and in exposure–outcome relationships were observed on inclusion of second and third mailing respondents in comparison with only first mailing respondents. The findings can be generalized to other surveys with similar design; however, the effects of multiple reminders might differ in surveys with different study designs and study populations.

Response rate

A substantial increase in respondents was observed when multiple reminders were used. This is in line with the findings in a previous review in which the odds of response generally increased by more than a quarter when reminders were issued (via repeated mailings or telephone calls, for instance).³ Also, consistent with previous research, a lower return rate by each mailing was observed, questioning the justification of multiple reminders^6,15–18: e.g. Breen and colleagues¹⁹ concluded that even though reminder phone calls increased the number of responses, these benefits were not justified by the resources required. Hence, if multiple reminders are used to increase the number of respondents, a more cost-effective alternative might be to enlarge the initial sample size. However, a high response rate might be an end in itself, because the risk of non-response bias is usually reversely related to the response rate. A study may, therefore, be greeted with scepticism if the response rate is low and the validity may be questioned. But if the non-response is not associated with the indicator of interest, the non-response might not impact the representativeness of the study.

Differences in socio-demographic characteristics and register-based health indicators

A major strength of this study is the complete information on socio-demographic characteristics and health care utilization among both respondents and non-respondents. Hence, the ‘accurate’ socio-demographic distribution and health care utilization rate are known for all individuals invited to this study. Differences in socio-demographic characteristics were found between respondents and non-respondents, suggesting that response bias could be an issue. The uncovered differences were broadly in accordance with previous studies: i.e. women respond to a higher degree than men, the middle-aged respond more than the young and elderly, individuals with a higher level of education respond more than those with a lower level, the married respond more than the unmarried and those with a native background respond more than those with foreign backgrounds.^{8,16,17,20–27} Regarding health care utilization, higher rates among respondents than among non-respondents were observed for most types of care, with the only exception being hospitalization, which is often associated with more severe health problems. The severity of the health problems may, therefore, be a reason for non-response in itself. This is supported by a previous Danish study showing that the hospitalization rate was only higher among non-respondents than among respondents immediately before and during the data collection period. Overall, the results of the present study are in line with other studies examining the same types of care.^24,28 It has been suggested that the framing of the survey as a health survey could especially attract regular users of care.²⁴ Thus, individuals already in contact with the health care system might be more prone to participate in a health survey, as they consider it relevant for their situation. This might explain the higher utilization rates among first mailing respondents and respondents in general than among non-respondents. Also, use of health care can be viewed as a sign of being more interested and concerned about health and, hence, acting more rapidly on symptoms, and individuals with high utilization might in some part be the ‘worried well’.

The inclusion of second and third mailing respondents did not change the overall characteristics of respondents compared with non-respondents in terms of socio-demographic characteristics and register-based health indicators. Hence, when differences do exist between respondents and non-respondents, the results suggest that second and third mailings are unlikely to remove many of these differences: i.e. they are unlikely to eliminate non-response bias. This is mainly because the amount of data provided by the second and third mailings will be small in comparison with the first mailing. This is supported by previous findings showing that although the cumulative distribution of characteristics among respondents changed slightly towards the distribution among all invited individuals concurrently with multiple reminders, it was still closer to the distribution for the initial respondents than to the distribution among all invited individuals by the end of the study.^6,15,29 The argument that multiple reminders make the distribution among respondents more like the distribution among all invited individuals might, therefore, be questioned. Hence, questionnaires in different languages, incentives and the contents of introduction letters or other tailored methods might be more efficient methods to reach non-respondents.

Prevalence estimates

The proportion of respondents reporting excellent, very good or good self-rated health was higher among first mailing respondents than among third mailing respondents. Likewise, the Oslo Health Study showed that among the oldest age groups, a higher proportion among immediately attending respondents reported excellent or good health compared with those attending after one or two reminders.²⁹ Others found a low proportion of respondents with poor self-rated health among initial respondents.^2,8,30 Also, a higher proportion of daily smokers were found among third mailing respondents than among first mailing respondents. This is in line with previous findings,^6,7,16,31 such as those of Brogger and colleagues, who found an increase in the overall current smoking rate by 3.8 percentage points (from 30.1 to 33.9%) in relation to increased contact efforts. Furthermore, we found that third mailing respondents were more likely to be obese and sedentary in leisure time than were first mailing respondents. This is supported by another study showing that respondents participating after one or two reminders were more likely to be obese than those participating immediately.²⁹ However, Steffen and colleagues³¹ found that getting more exercise per week, but not body mass index, was related to early response. Overall, the results support the general notion that late respondents have a higher probability of poor health compared with early respondents.^{2,8,16,30,32,33} Furthermore, the results indicate that the higher probability of poor health among third mailing respondents compared with first mailing respondents cannot be explained by differences in the educational distribution, as in each educational group, third mailing respondents had poorer health than first mailing respondents. Rather, the systematic differences between first mailing respondents and third mailing respondents and the slightly higher hospitalization rate among non-respondents indicate that first mailing respondents are the ‘worried well’: i.e. healthy individuals who see their general practitioner regularly and follow healthy lifestyle practices. However, the response bias observed when comparing first mailing respondents with third mailing respondents is unlikely to affect study conclusions owing to restricted impact. The most pronounced bias lies in the differences between respondents and non-respondents, which is unlikely to be eliminated by multiple reminders. Whether multiple reminders are justified will depend on how much the researcher weights the observed changes in prevalence and the increased risk of non-response bias associated with a low response rate compared with the extra cost related to multiple reminders.

Exposure–outcome relationships

Only small changes in exposure–outcome relationships were observed when comparing all respondents with only first mailing respondents, which is in line with previous findings concluding that a broad range of exposure–outcome relationships remained consistent regardless of the underlying response rate.^6–8 Hence, the present study and previous research provide support for the notion that results based on internal comparison mostly remain generalizable even when study subjects are drawn from a selective group. With respect to exposure–outcome relationships, multiple reminders have very little effect and might, therefore, not be justified.

Limitations

The study design do not allow us to assess the impact of regional differences on response pattern in, e.g. the optional questions, survey advertising and cooperation with local municipalities. Another limitation is the non-randomized design, i.e. there is no direct comparison group to whom no reminder is sent, and hence, individuals might self-select to receive a reminder by not replying. Nevertheless, the number of responses received by the time of the second and third reminder was practically zero and it is unlikely that the number of responses would have increased markedly without reminders. However, some level of misclassification cannot be ruled out, as some first mail responders might be classified as second mail responders if they responded around the same time as the reminders were distributed.

Conclusions

Multiple reminders are an effective way to increase the total response rate in a national health survey. However, multiple reminders had only a minor effect on response patterns and study conclusions. The utility of the response rate for predicting non-response bias is, therefore, questionable and alternative indicators for predicting non-response bias—those involving sampling frame data, auxiliary data and survey data, for instance—might be more useful. If the aim of multiple reminders is both to reduce non-response bias and reach a high response rate, other means than multiple reminders should be considered as alternatives or complements when planning a survey. Tailored study design methods such as questionnaires in different languages, incentives and the contents of introduction letters are some of the means that should be brought into consideration. If the aim of multiple reminders is solely to increase the number of respondents, the researcher could instead increase the sample size. The larger sample size would achieve about the same total number of responses and, hence, study power, but in a shorter time and at a lower cost. In general, it might be helpful for the researcher to reflect on the desired total number of responses, the value of a high response rate and the efficiency of different methods to reduce non-response bias when deciding which strategy to adopt and how the available resources might be best spent.

Funding

This work was supported by the National Institute of Public Health, University of Southern Denmark and The Ministry of Health.

Conflicts of interest: None declared.

Acknowledgements

Inger Helt Poulsen, Region of Zealand, Denmark, has participated in the DNHS-2010 work group and has put data from the Region of Zealand at the authors’ disposal for the present study.

References