Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.
4. National Household Survey estimation
Table of contents
- 4.1 Considerations in the choice of an estimation procedure
- 4.2 NHS universe
- 4.3 Design weights
- 4.4 Total non-response adjustment
- 4.5 Surprise respondents
- 4.6 Calibration
Any sampling process requires an associated estimation procedure for scaling sample data up to the population level and to ensure survey estimates are representative of the population. The choice of an estimation procedure is generally governed by both operational and theoretical constraints. From the operational viewpoint, the procedure must be feasible within the processing system of which it is a part, and from the theoretical viewpoint, the procedure should minimize the statistical error in the estimates it produces. Section 4.1 describes the operational and theoretical considerations relevant to the choice of estimation procedures. Sections 4.2 to 4.6 focus on the details of the estimation in the 2011 National Household Survey (NHS), including the definition of the NHS universe, design weights, total non-response weight adjustments, and final weight calibration processes.
4.1 Considerations in the choice of an estimation procedure
4.1.1 Operational considerations
Mathematically, an estimation procedure can be described by an algebraic formula, or estimator, that shows how the estimate for the population is calculated as a function of the observed sample values and other information from the sample design or external data sources. Most of the time, this estimator is a simple function of weights and of the variable of interest for the responding units. Using a unique set of weights to produce all estimates guarantees a certain level of consistency between the different estimates of the survey.
Therefore, the approach taken in the NHS (and in most sample surveys) is to split the estimation procedure into two steps: (a) the calculation of weights (known as the weighting procedure) and (b) the use of weights to produce estimates, such as the estimation of a particular characteristic by summing the weights of those persons or households having that characteristic. Most of the mathematical complexity is then contained in step (a) which is performed just once. Meanwhile, step (b) is reduced to a simple process, such as summing weights whenever tabulation is required. It should be noted that since the weight attached to each sample unit is the same for any tabulation involving that unit, consistency between different estimates based on sample data is assured.
4.1.2 Theoretical considerations
For a given sample design and a given estimation procedure, one can, from sampling theory, make a statement about the chances that a certain interval will contain the unknown population value being estimated. The primary criterion in the choice of an estimation procedure is the minimization of the width of such intervals for a given level of confidence so that these statements about the unknown population values are as precise as possible. The usual measure of precision for comparing estimation procedures is known as the standard error. Provided that certain conditions are met, intervals of plus or minus two standard errors from the estimate will contain the population value for approximately 95% of all possible samples.
As well as minimizing standard error, a second objective in the choice of estimation procedure for the NHS sample is to ensure, as far as possible, that sample estimates for census characteristics are consistent with the corresponding known census values. Fortunately, these two objectives are usually complementary in the sense that sampling error tends to be reduced by ensuring that sample estimates for certain basic characteristics are consistent with the corresponding population figures. However, while this is true in general, forcing NHS sample estimates for census characteristics to be consistent with corresponding census figures for very small subgroups can have a detrimental effect on the standard error of estimates for the sample characteristics themselves.
In cases where there is no information about the population being sampled other than that collected for sample units, and there has not been unit non-response, the estimation procedure would be restricted to weighting the sample units inversely to their probabilities of selection. For example, if a unit had a one-in-three chance of selection, then that selected unit would receive a weight of 3. When unit non-response is observed, the weight also has to be further adjusted using the probability of response of the unit. Also, in practice, one almost always has some supplementary knowledge about the population (e.g., its total size, and possibly its breakdown by a certain variable — perhaps by province and territory). Such information can be used to improve the estimation formula so as to produce estimates with a greater chance of lying close to the unknown population value. In the case of the NHS sample, a large amount of very detailed information about the population being sampled was available in the form of the census data at every geographic level. We can take advantage of this wealth of population information to improve the estimates made from the NHS sample. This will be discussed later in this report.
4.1.3 Additional considerations for the National Household Survey weighting
Just as in previous censuses, every household in the NHS was assigned a weight so that the characteristics of NHS respondents can be weighted and combined to produce estimates for the population. However, there were at least two major issues that made the process more complex in 2011. First, census and NHS collection were separate events. Having two sets of questionnaires being asked at two different times led to complications such as household linkage and data inconsistencies between the two sources. This was mentioned in Section 2.9. Secondly, the NHS had to deal with relatively high household non-response. Chapter 3 discussed the problem of total non-response and the design of the non-response follow-up (NRFU) process to improve response rates.
In a survey, the procedure of weighting the sample units inversely to their probabilities of selection (or probabilities of response in the presence of a total non-response adjustment) should result in small differences between the sample estimates and the census counts for large subgroups of the population. However there could be significant differences for smaller subgroups. These differences were usually made greater by the cases of total non-response in the NHS.
It is difficult to make the NHS sample estimates for census characteristics consistent with all the census counts at every geographic level. Differences between sample estimates and census counts become visible when a cross-tabulation of a sample variable and the corresponding census variable is produced. The tabulation of sample based estimates of totals for particular characteristics will not necessarily agree with the equivalent census counts tabulations for those characteristics.
Adjusting the weights, equal to the inverse of the probabilities of response, by small amounts to achieve perfect agreement between estimates and census counts for certain subgroups is known as 'calibration'. This procedure is further discussed in Section 4.6.
4.2 NHS universe
The census household universe can be broken into three: the private households, the collective households, and the households outside Canada. The NHS household universe corresponds to private households that were eligible for the 2011 Census. Unless specified otherwise, the term 'in-scope' will be used in this document to indicate that a household is part of the NHS universe (i.e., private households) while 'out of scope' refers to households not in the universe (i.e., collective and outside Canada households).
An exception to the NHS universe involves the private households of five census subdivisions (CSDs) corresponding to five Indian reserves. They were excluded from the universe because of a very low response rate in the NHS. If they had not been excluded, then surrounding areas within the same weighting areas (WA) would have been greatly affected by the non-response and calibration weight adjustments that were necessary to compensate for their low response rates. The exclusion of those five CSDs is an example of some of the differences that can be observed when comparing a census publication to an NHS publication. The number of private households and the population of those five CSDs can be found in Table 4.2.1.
Census subdivisions not in the NHS universe
|Province||Census division||Census subdivision||Number of private households||Population|
|Sources: Statistics Canada, 2011 Census and 2011 National Household Survey.|
|Ontario||Cochrane||Factory Island 1||384||1,414|
|Ontario||Kenora||Sandy Lake 88||459||1,861|
|Manitoba||Division No. 19||The Narrows 49||337||826|
|Saskatchewan||Division No. 14||Opaskwayak Cree Nation 27A (Carrot River)||72||286|
4.3 Design weights
Every dwelling of the first phase sample (i.e., selected for the NHS) was given a first phase basic weight equal to the inverse of the probability of selecting that dwelling in the first phase sample. Every subsampled dwelling in the second phase sample (i.e., selected for non-response follow-up [NRFU]) was also given a second phase basic weight equal to the inverse of the probability of selecting that dwelling in the subsample.
The design weights were generally calculated as follows:
- Dwellings that were not eligible for NRFU subsampling were assigned a design weight equal to their first phase sample weight. The vast majority of these corresponded to households identified as having responded to the NHS.
- Dwellings that were eligible for NRFU subsampling and were not selected for NRFU had their design weight set to 0.
- Dwellings that were subsampled for NRFU were assigned a design weight equal to the product of their first and second phase basic weights.
4.4 Total non-response adjustment
There are different types of non-respondent households in the NHS. Households that were identified as having responded by the time of subsampling were not eligible for subsample selection. However, it was discovered after careful evaluation of their questionnaires that some of them, despite having returned the questionnaire, had not in fact provided responses and so were actually non-respondents. Furthermore, many dwellings selected for NRFU did not contribute information despite attempts to obtain a response. The overall unweighted response rate was 68.6%, and the weighted response rate was 77.2%.
Various strategies for the treatment of total non-response that made use of auxiliary data available for both respondents and non-respondents were studied. The imputation approaches were attractive in the NHS context given that census data were available for the vast majority of non-respondents. In other words, unit non-response to the NHS can then be viewed as an item non-response problem. Unfortunately, the imputation approaches were not always successful because the large number of matching variables made it often impossible to find a perfect donor. A perfect donor would be a respondent household that has the same value as the non-respondent household for every matching census variable. This led to significant increases of occurrence in the data of combinations of census variables and non-census imputed variables that are rare in the population. After thorough analysis, these imputation strategies were discarded and it was decided that non-respondent households that were initially flagged as having responded and households in the subsample that did not respond would be assigned a weight of 0. The weights that they would have had if they had responded were transferred to their nearest neighbours. The method, a re-weighting approach, can be divided into the following three main steps.
The first step was to determine the auxiliary variables that best predicted the households' propensities to respond. Many auxiliary variables from the census and NHS, from linkages to 2010 tax files, the Indian Register, and 1980 to 2011 immigration files, and variables related to Indian reserves (where applicable), were considered in the construction of logistic regression models at the Canada level using forward variable selection. All the auxiliary variables were then assigned a relative weight according to their predictive power in the response model that was selected. Although some auxiliary variables did not enter the final models, every variable was assigned a minimum positive weight.
In the second step, the Canadian Census Edit and Imputation System (CANCEIS) (see CANCEIS version 5.2 Basic User Guide) was used to locate 20 nearby respondent households that best matched each non-respondent household. The chosen households had the same number of members and were usually in the same neighbourhood or a neighbourhood near the non-respondent household. The matching process used the auxiliary variables and their relative weights from the first step. A match score between 0 and 1 was given for each matching variable, where a score of 0 occurred if the values were the same and a value greater than 0 (usually 1) was given if they were different. These scores were multiplied by the CANCEIS imputation weight of the variable and summed over all variables to calculate a distance score between the non-respondent and the respondent. The 20 respondents with the lowest scores were identified.
The third step consisted of transferring the weight of the non-respondent household to each of the 20 respondent households identified in the second step. The amount of weight distributed was proportional to the inverse of the distance between the respondent household and non-respondent household. Therefore, respondent households that better matched the non-respondent household received a greater share of the weight that was being transferred.
4.5 Surprise respondents
Surprise respondents are households that were from dwellings selected in the first phase sample that had not responded before subsample selection, but which responded afterward without being part of the NRFU subsample. The surprise respondents were combined with respondent households in the second phase sample. Their design weight was set to 0 as described in Section 4.3. Instead of leaving these surprise respondents with their initial weight of 0, their weight was forced to 1 and the weight of some respondents in the subsample was reduced accordingly. This was done because a weight of 1 minimised bias while making sure the surprise respondents were self-represented.
Similar to the total non-response adjustment, CANCEIS was used to identify the set of neighbours consisting of the 20 closest respondent households in the second phase sample for each surprise respondent. A weight of 1/20 was then subtracted from each of these neighbour respondents and transferred to surprise respondents.
Once the NHS design weights were adjusted to account for total non-response and surprise respondents, they were adjusted one last time in a calibration process. The process slightly adjusted the weights in order to satisfy a certain number of constraints. While this ensures consistency between NHS estimated totals and census counts for many constraints, the effect of calibration is a reduced sampling variability in the survey estimates. During this final adjustment step, it was important to limit the range of the calibrated weights so that no unreasonable amount of weight was placed on any household or person. Therefore, weights were constrained to be between 1 and 100.
Appendix C lists all the constraints that were considered during the calibration process. They included the same constraints as in 2006 as well as additional constraints involving census and economic family variables and language variables. Characteristics available from both the census and the NHS for which consistency was attempted include information such as age, sex, marital status, common-law status, household size, dwelling type, mother tongue, home language, etc.
Calibration and constraint selection was carried out independently in geographic areas known as weighting areas (WA). There were 5,884 WAs in Canada, most of which consisted of a contiguous area of land within a census division (CD). On average a WA contained about 2,300 households in the census, and most WAs contained between 300 and 699 households that were either initial NHS respondents or households that were selected for NRFU. See Chapter 6 for additional information about the construction of WAs.
The first step in the calibration process was to categorize each of the constraints into one of three groups:
- Mandatory constraints – These constraints must be calibrated because they must have agreement between the census count and the NHS estimates at geographic levels which are usual aggregates of WAs (e.g., Canada, provincial and territorial). The number of persons in the WA and the number of households in the WA were the two mandatory constraints.
- Low response constraints – Constraints that would be applied to less than 30 responses should not be calibrated because they can make survey estimates unstable.
- All other constraints – These constraints were examined further to see if they should be calibrated.
The second step was to determine the constraints from the third group to be used in the calibration process in addition to the mandatory constraints. The constraints from the third group were added one by one beginning with those that were least linearly dependent with the low response constraints and the constraints already included. Constraints that were too linearly dependent were not included.
Calibration was then performed on the final set of constraints from step 2 by adjusting the household weights as little as possible so that the sum of the weighted estimates equalled the census counts for those constraints. In practice, the calibration process was performed using software developed by Statistics Canada called StatMx, which was programmed entirely in SAS language. See Verret (2013) for more details.
There are a few reasons why sample estimates may be different from census counts, particularly for small areas, even after the calibration step. The main ones are listed below.
- Constraints dropped during the calibration process: As described earlier, constraints could be dropped for having small counts, by being linearly dependent with other constraints that were chosen, or by being nearly linearly dependent with constraints having small counts. Constraints that were dropped were not controlled upon, and usually led to some differences between census counts and NHS estimates.
- Sub-WAs: The WA was the smallest geographic area for which the weighting system attempted to have agreement between the census counts and the NHS estimates for as many auxiliary variables as possible. Any place that is smaller than a WA, such as most DAs, is known as a sub-WA. These could have discrepancies between the census counts and the NHS estimates.
The point above is important in areas of high non-response. This is because WAs had to be larger than usual to contain 300 to 699 respondent households. As a result, some small communities or municipalities with low response rates were sub-WAs, meaning that consistency of their estimates with census counts cannot be guaranteed.