## Archived Content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

# 8. Variance

The error in an estimate is the difference between the estimate and the actual value of what is being estimated. Sampling error and household non-response error are two sources of error in the NHS. Sampling error stems from the fact that the estimates are based on observations from a sample and not from the entire population. Household non-response error occurs when households selected in the sample do not respond to the survey. Estimation methods also have an impact on the error. Some methods are more precise than others in measuring a particular characteristic of the population.

The error has two components: a random component, the variance; and a systematic component, the bias. The variance measures how much the estimate varies from the average that would result from hypothetical repetitions of the survey process. The variance can be estimated using data from respondents in the sample. The bias is the difference between the average estimate that would result from hypothetical repetitions of the survey process and the actual value of the characteristic being estimated. Non-response bias is covered in Chapter 5. The sampling and estimation methods used in the NHS generate negligible bias. This chapter deals with sampling variance and variance due to household non-response.

To estimate the variance, the first step is to derive estimators using mathematical formulas. Those formulas have to take into account the various sampling and weighting steps described in previous chapters. The formulas specify how data from respondents are used to produce variance estimates. To perform the derivations, certain assumptions and simplifications had to be made. First, it was assumed that the sample and subsample selected by systematic stratified sampling were selected by stratified simple random sampling without replacement. This assumption is necessary when systematic sampling is performed. The assumption is reasonable because the strata are small and their population can therefore be treated as homogeneous. The second assumption concerns the mechanism that generates household non-response. It was assumed to be equivalent to a Poisson sampling process in which a household's probability of response/selection is equal to the ratio of its weight assigned before non-response adjustment weight transfer to the weight assigned to it after the transfer. For example, a respondent household whose weight doubled following the transfer would have an estimated response probability of 50%. It was also assumed that this probability was known, not estimated. Making these assumptions about the non-response mechanisms means that the non-response bias in the variance estimate is ignored. The last simplification is that surprise respondents were excluded from the variance estimation process. This makes the process much easier and has little impact on the estimate since surprise respondents make up a small fraction of all respondents. The technical details of the variance estimation process are provided in Verret (2013).

Since the variance estimators based on the formulas described above tend to underestimate the actual variance, the final step was to correct them using simulations. To that end, an artificial population was first generated using sample data. Multiple samples were then selected independently (i.e., replicates), and the weighting methods were applied to each one. The calculated variance estimators were compared with the actual variability from replicate to replicate. This provides an upward adjustment that is applied to all the variance estimates calculated with the formulas.

The estimated variance can be used to construct several types of measures of an estimate's variability. For example, it can be used to calculate standard errors or coefficients of variation (CVs). The standard error is equal to the square root of the variance. The CV is the ratio of the estimate's standard error to the estimate itself. The smaller these measures of variability are, the more precise the estimate. The CV is a particularly interesting measure of variability in that it does not depend on the unit of measure of the characteristic being studied and can be expressed as a percentage. It is important to carefully distinguish between these measures of variability and other measures of quality that are not, strictly speaking, measures of variability. Examples of such measures are the bias indicators, response rate and global non-response rate of the NHS. The response rate is an indicator of the risk associated with household non-response error. The global non-response rate is an indicator of the risk of error due to household non-response and item non-response. Table 8.1 shows the CVs of various estimates at the Canada, provincial and territorial levels. The contents of this table can be consulted or downloaded at the following address: http://www12.statcan.gc.ca/nhs-enm/2011/ref/cv/index.cfm?Lang=E.

The sampling and estimation methods used in the 2011 NHS are different from the ones used for the 2006 Census long questionnaire. The magnitude and effect of non-response are very different in 2011. All that has an impact on the variability of the estimates. A methodological note comparing the CVs of the 2011 estimates with the CVs of the 2006 estimates for some variables is available at http://www12.statcan.gc.ca/nhs-enm/2011/ref/cv/cvnote.cfm?Lang=E.

Table 8.1
Coefficients of variation (%) of the 2011 National Household Survey – Canada, provinces and territories

Table summary
This table displays the results of Coefficients of variation (percentage) of the 2011 National Household Survey – Canada, provinces and territories. The information is grouped by Characteristic (appearing as row headers), Canada, Newfoundland and Labrador, Prince Edward Island, Nova Scotia, New Brunswick, Quebec and Ontario (appearing as column headers).
Characteristic Canada Newfoundland and Labrador Prince Edward Island Nova Scotia New Brunswick Quebec Ontario
Population characteristic
Immigration and citizenship
Total population in private households by citizenship 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Canadian citizens 0.01 0.06 0.18 0.06 0.06 0.02 0.03
Not Canadian citizens 0.23 6.06 5.87 2.40 3.01 0.50 0.35
Total population in private households by immigrant status 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Non-immigrants 0.02 0.07 0.24 0.09 0.08 0.03 0.04
Immigrants 0.08 3.55 3.88 1.41 1.87 0.21 0.10
Non-permanent residents 0.61 9.99 16.29 4.97 5.98 1.28 1.02
Total immigrant population in private households by place of birth 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Americas 0.29 6.74 7.96 2.94 3.11 0.60 0.41
Europe 0.17 4.79 6.15 2.08 2.98 0.46 0.22
Africa 0.50 16.08 41.14 7.51 9.04 0.72 0.83
Asia 0.13 7.95 6.87 3.07 4.43 0.50 0.17
Oceania and other 1.32 28.55 55.64 12.39 15.27 6.86 2.51
Visible minority
Total population in private households by visible minority 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Visible minority population 0.10 5.27 6.21 1.76 2.90 0.30 0.14
Not a visible minority 0.02 0.07 0.20 0.10 0.07 0.04 0.05
Aboriginal peoples
Total population in private households by Aboriginal identity 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Aboriginal identity 0.26 1.61 8.37 1.71 2.10 0.72 0.60
Non-Aboriginal identity 0.01 0.12 0.14 0.07 0.07 0.01 0.02
Mobility
Total – Mobility status 5 years ago 0.00 0.03 0.04 0.02 0.02 0.01 0.00
Non-movers 0.05 0.35 0.72 0.27 0.29 0.09 0.08
Movers 0.08 0.86 1.52 0.54 0.60 0.16 0.14
Education
Total population aged 15 years and over by highest certificate, diploma or degree 0.01 0.05 0.10 0.04 0.03 0.01 0.01
No certificate, diploma or degree 0.10 0.72 1.61 0.60 0.61 0.19 0.17
High school diploma or equivalent 0.09 0.86 1.44 0.59 0.59 0.20 0.14
Postsecondary certificate, diploma or degree 0.04 0.45 0.80 0.29 0.35 0.08 0.07
University certificate, diploma or degree at bachelor level or above 0.09 1.11 1.79 0.62 0.77 0.19 0.14
Labour
Total population aged 15 years and over by language used most often at work 0.03 0.29 0.41 0.20 0.20 0.06 0.05
English, single response 0.04 0.30 0.44 0.20 0.27 0.30 0.05
French, single response 0.08 8.67 7.41 2.63 0.66 0.08 0.75
Non-official languages, single response 0.51 6.10 15.65 7.38 7.02 1.20 0.83
Total population aged 15 years and over by labour force status 0.01 0.05 0.10 0.04 0.03 0.01 0.01
In the labour force 0.03 0.33 0.47 0.22 0.23 0.06 0.05
Employed 0.04 0.41 0.62 0.26 0.27 0.07 0.06
Unemployed 0.22 1.47 2.87 1.25 1.32 0.45 0.34
Not in the labour force 0.06 0.48 1.03 0.37 0.39 0.12 0.10
Housing
Total number of occupied private dwellings by condition of dwelling 0.00 0.00 0.01 0.00 0.00 0.00 0.00
Only regular maintenance or minor repairs needed 0.02 0.18 0.33 0.15 0.16 0.04 0.03
Major repairs needed 0.25 2.09 4.07 1.37 1.50 0.49 0.45
Total number of occupied private dwellings by number of rooms 0.00 0.00 0.01 0.00 0.00 0.00 0.00
Average number of rooms per dwelling 0.03 0.24 0.45 0.16 0.18 0.04 0.04
Total number of private households by tenure 0.00 0.00 0.01 0.00 0.00 0.00 0.00
Owner 0.03 0.26 0.47 0.19 0.19 0.07 0.05
Renter 0.07 0.90 1.31 0.47 0.60 0.11 0.13
Band housing 0.12 1.07 3.63 0.30 0.65 0.55 0.56
Shelter costs
Total number of owner and tenant households with household total income greater than zero, in non-farm, non-reserve private dwellings by shelter-cost-to-income ratio 0.01 0.02 0.12 0.04 0.04 0.01 0.01
Spending less than 30% of household total income on shelter costs 0.04 0.29 0.61 0.23 0.23 0.07 0.07
Spending 30% or more of household total income on shelter costs 0.12 1.32 2.46 0.79 0.99 0.24 0.19
Spending 30% to less than 100% of household total income on shelter costs 0.14 1.50 2.78 0.93 1.15 0.28 0.22
Income of individuals
Total income in 2010 of population aged 15 years and over 0.01 0.05 0.10 0.04 0.03 0.01 0.01
Average income (\$) 0.09 0.50 0.98 0.44 0.41 0.15 0.14
After-tax income in 2010 of population aged 15 years and over 0.01 0.05 0.10 0.04 0.03 0.01 0.01
Average after-tax income (\$) 0.07 0.42 0.84 0.33 0.34 0.13 0.11
Income of families
Family income in 2010 of economic families 0.01 0.09 0.20 0.07 0.07 0.02 0.02
Average family income (\$) 0.10 0.55 1.11 0.51 0.46 0.18 0.15
Average after-tax family income (\$) 0.08 0.46 0.95 0.38 0.38 0.15 0.13
Average family size 0.01 0.09 0.20 0.07 0.07 0.02 0.02
Income of households
Household income in 2010 of private households 0.00 0.00 0.01 0.00 0.00 0.00 0.00
Average household total income (\$) 0.09 0.51 1.01 0.45 0.42 0.15 0.14
Average after-tax household income (\$) 0.07 0.43 0.87 0.34 0.35 0.13 0.12
Table summary
This table displays the results of Coefficients of variation (percentage) of the 2011 National Household Survey – Canada, provinces and territories. The information is grouped by Characteristic (appearing as row headers), Manitoba, Saskatchewan, Alberta, British Columbia, Yukon, Northwest Territories and Nunavut (appearing as column headers).
Characteristic Manitoba Saskatchewan Alberta British Columbia Yukon Northwest Territories Nunavut
... not applicable
Source: Statistics Canada, 2011 National Household Survey.
Population characteristic
Immigration and citizenship
Total population in private households by citizenship 0.00 0.00 0.00 0.00 0.00 0.02 0.00
Canadian citizens 0.08 0.07 0.05 0.05 0.34 0.16 0.02
Not Canadian citizens 1.20 1.84 0.68 0.52 6.26 5.91 3.01
Total population in private households by immigrant status 0.00 0.00 0.00 0.00 0.00 0.02 0.00
Non-immigrants 0.11 0.09 0.07 0.07 0.50 0.22 0.04
Immigrants 0.56 1.19 0.29 0.18 3.95 2.94 1.74
Non-permanent residents 4.10 4.76 1.58 1.37 21.79 12.52 6.22
Total immigrant population in private households by place of birth 0.00 0.00 0.00 0.00 0.00 0.02 0.00
Americas 1.81 3.54 1.08 0.88 9.58 8.62 3.71
Europe 1.20 2.19 0.64 0.44 5.92 5.72 2.77
Africa 3.61 5.97 1.65 1.73 28.75 13.57 5.28
Asia 0.84 1.87 0.46 0.23 9.38 5.51 3.22
Oceania and other 10.08 11.08 3.56 1.83 22.02 24.68 6.46
Visible minority
Total population in private households by visible minority 0.00 0.00 0.00 0.00 0.00 0.02 0.00
Visible minority population 0.69 1.39 0.33 0.19 6.62 3.54 2.19
Not a visible minority 0.10 0.09 0.07 0.07 0.43 0.25 0.04
Aboriginal peoples
Total population in private households by Aboriginal identity 0.00 0.00 0.00 0.00 0.00 0.02 0.00
Aboriginal identity 0.62 0.71 0.72 0.64 2.72 0.67 0.11
Non-Aboriginal identity 0.12 0.13 0.05 0.04 0.82 0.72 0.67
Mobility
Total – Mobility status 5 years ago 0.02 0.02 0.01 0.01 0.00 0.06 0.04
Non-movers 0.26 0.29 0.17 0.15 1.56 0.89 0.47
Movers 0.42 0.47 0.21 0.20 1.89 0.94 0.62
Education
Total population aged 15 years and over by highest certificate, diploma or degree 0.03 0.04 0.02 0.02 0.00 0.09 0.06
No certificate, diploma or degree 0.46 0.55 0.33 0.30 2.91 0.86 0.32
High school diploma or equivalent 0.44 0.52 0.27 0.23 2.71 1.49 0.93
Postsecondary certificate, diploma or degree 0.27 0.32 0.14 0.12 1.23 0.70 0.50
University certificate, diploma or degree at bachelor level or above 0.54 0.70 0.29 0.24 2.75 1.73 0.99
Labour
Total population aged 15 years and over by language used most often at work 0.14 0.16 0.08 0.08 0.51 0.34 0.25
English, single response 0.15 0.17 0.08 0.09 0.56 0.36 0.35
French, single response 3.04 6.39 3.39 3.25 17.74 13.75 4.66
Non-official languages, single response 2.78 4.28 2.14 0.94 21.23 3.78 0.83
Total population aged 15 years and over by labour force status 0.03 0.04 0.02 0.02 0.00 0.09 0.06
In the labour force 0.16 0.19 0.09 0.09 0.74 0.37 0.28
Employed 0.18 0.21 0.10 0.10 0.91 0.46 0.33
Unemployed 1.27 1.48 0.74 0.59 4.80 1.88 1.14
Not in the labour force 0.33 0.42 0.24 0.16 2.50 1.13 0.50
Housing
Total number of occupied private dwellings by condition of dwelling 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Only regular maintenance or minor repairs needed 0.13 0.15 0.06 0.05 0.84 0.35 0.34
Major repairs needed 1.05 1.24 0.83 0.70 4.34 1.68 0.90
Total number of occupied private dwellings by number of rooms 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average number of rooms per dwelling 0.13 0.17 0.08 0.07 0.91 0.39 0.19
Total number of private households by tenure 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Owner 0.15 0.19 0.11 0.10 1.05 0.66 0.92
Renter 0.38 0.57 0.30 0.24 2.37 0.72 0.24
Band housing 0.16 0.21 0.16 0.65 2.49 3.78 Note ...: not applicable
Shelter costs
Total number of owner and tenant households with household total income greater than zero, in non-farm, non-reserve private dwellings by shelter-cost-to-income ratio 0.06 0.11 0.04 0.02 0.16 0.06 0.01
Spending less than 30% of household total income on shelter costs 0.20 0.25 0.13 0.13 1.00 0.51 0.14
Spending 30% or more of household total income on shelter costs 0.79 0.89 0.40 0.29 4.02 3.05 2.07
Spending 30% to less than 100% of household total income on shelter costs 0.91 1.01 0.47 0.35 4.59 3.44 2.33
Income of individuals
Total income in 2010 of population aged 15 years and over 0.03 0.04 0.02 0.02 0.00 0.09 0.06
Average income (\$) 0.33 0.46 0.33 0.23 1.48 0.65 0.38
After-tax income in 2010 of population aged 15 years and over 0.03 0.04 0.02 0.02 0.00 0.09 0.06
Average after-tax income (\$) 0.27 0.39 0.28 0.19 1.30 0.58 0.35
Income of families
Family income in 2010 of economic families 0.06 0.08 0.04 0.04 0.30 0.21 0.10
Average family income (\$) 0.38 0.52 0.35 0.27 1.80 0.70 0.43
Average after-tax family income (\$) 0.31 0.45 0.30 0.22 1.58 0.62 0.40
Average family size 0.06 0.08 0.04 0.04 0.35 0.21 0.10
Income of households
Household income in 2010 of private households 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average household total income (\$) 0.34 0.48 0.33 0.24 1.57 0.66 0.39
Average after-tax household income (\$) 0.28 0.40 0.29 0.20 1.38 0.59 0.36

Is something not working? Is there information outdated? Can't find what you're looking for?