5. National Household Survey non-response bias indicators

5.1 Introduction

Because of the voluntary nature of the NHS, Statistics Canada put a great deal of effort into maximizing household participation, particularly using mixed collection methods and the wave collection methodology (see Chapter 1 on Data collection). Despite these efforts and a response rate similar to that of other voluntary surveys conducted by Statistics Canada (68.6%), almost one in three households did not participate. At the processing stage, total non-response was treated using a reweighting methodology. The purpose of reweighting is mainly to reduce the non-response bias in the estimates.

Non-response has two possible effects on the data. First, it contributes to an increase in the total variance of the estimates, since the actual sample size is smaller than the planned size. Second, it can introduce a bias in the estimates when non-respondents differ from respondents in the characteristics measured.^Footnote1 This is referred to as non-response bias. Consequently, to assess data quality, it is recommended that efforts be made to determine whether there is a potential non-response bias and, if possible, to derive an indicator of its magnitude.

A non-response bias indicator is a tool attempting to identify the presence of a potential residual bias, i.e., after reweighting. Non-response bias occurs when the characteristics of interest of respondents differ from those of non-respondents. Unlike the sampling variance, it does not necessarily decline with an increase in sample size.

A positive non-response bias indicator indicates that the estimator overestimates the parameter of interest on average, and a negative non-response bias indicator indicates an underestimate. The magnitude of the non-response bias indicator indicates the extent of the potential bias in the estimate. Thus, a high value for the non-response bias indicator may indicate substantial differences between the characteristics of respondents and the characteristics of non-respondents.

5.2 Methodology of non-response bias indicators

The indicators of bias due to unit non-response were produced from a linked file of the 2011 and 2006 censuses. Using family name, address and date of birth, 73% of 2011 Census respondents were linked to their responses from the 2006 Census. This included responses from the 2006 long-form for a large portion of the 2011 NHS sample, including both respondents and non-respondents to the NHS.

The weights of the resulting matched sample were calibrated to estimates of the number of persons in the population common to both the 2006 and 2011 censuses. This calibration was done separately for 2011 NHS matched respondents and matched non-respondents. A second set of weights was calculated for the matched respondents. These weights used the 2011 non-response adjustment factors and were calibrated to the estimates of the total number of persons in the population common to both the 2006 and 2011 censuses. Using the two sets of weights, benchmark totals were obtained from the entire matched sample (i.e., 2011 NHS respondents and non-respondents) using the 2006 Census long-form questionnaire data and adjusted totals were obtained from the matched respondents sample again using the 2006 Census long-form questionnaire data.

The difference between an adjusted total and a benchmark total was used as an indicator of the bias due to total non-response in the 2011 NHS. Algebraically, for category $k$ of the 2006 variable $x_j$ this bias indicator can be expressed as $\hat{B}({\hat{X}}_{jk}^a)={\hat{X}}_{jk}^a-{\hat{X}}_{jk}$ with the benchmark ${\hat{X}}_{jk}= \underset{}{{\displaystyle {\displaystyle {\sum }_{i\in s_m}{w}_i{x}_{ijk}}}}$ and the adjusted total ${\hat{X}}_{jk}^a=\underset{}{{\displaystyle {\displaystyle {\sum }_{i\in s_{m,r}}{a}_i{w}_i{x}_{jk}}}}$, where, for unit $i,a_i$ is a total non-response adjustment, $w_i$ is a calibrated weight, $x_{ijk}$ is the value of variable $j$ for category $k,s_m$ is the matched sample and $s_{m,r}$ is the set of 2011 respondents in the matched sample.

5.3 Limitations of the NHS non-response bias indicators

There are several limitations to these indicators. The indicator reflects total non-response bias and thus does not account for item non-response. It is an indicator for the population in common between 2006 and 2011. In other words, the potential bias for births, recent immigrants (last 5 years) and non-permanent residents (last 5 years) is neglected. The indicator is based on a matched sample, so unmatched units that were part of the common population are not represented. The weight adjustment that accounts for the unmatched portion is likely imperfect. The indicator is based on the 2006 variables which are proxies to the 2011 variables. It lacks precision for geographic areas with a small matched sample, often the smaller geographic areas. Finally, its reliability is also affected by non-response to the 2006 Census short and long forms as well as to the 2011 Census. In sum, the bias indicators are exactly as the name implies, indicators of bias and not measures or estimates of bias.

5.4 Some NHS non-response bias indicators

For illustrative purposes, the non-response bias indicators shown in Table 5.4.1 were calculated at the Canada level for a few characteristics where there is potentially a higher risk of bias.

• Immigration status (landed immigrant, non-permanent resident)
• Aboriginal identity (First Nations (North American Indian) single identity, Métis single identity, Inuk (Inuit) single identity)
• Visible minority (South Asian, Chinese, Black, Filipino, Latin American)
• Highest certificate, diploma or degree (no certificate, diploma or degree; secondary (high) school diploma or equivalent; trades certificate or college diploma^Footnote2; bachelor's degree).

Table 5.4.1
Bias indicators at the Canada level for the 'Immigration status,' 'Aboriginal identity,' 'Visible minority' and 'Highest certificate, diploma or degree' characteristics