Statistics Canada
Symbol of the Government of Canada
Warning View the most recent version.

Archived Content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Appendix B
Data quality, sampling and weighting, confidentiality and random rounding

Modified on December 11, 2007

Data quality

General

The 2006 Census was a large and complex undertaking and, while considerable effort was taken to ensure high standards throughout all collection and processing operations, the resulting estimates are inevitably subject to a certain degree of error. Users of census data should be aware that such error exists, and should have some appreciation of its main components, so that they can assess the usefulness of census data for their purposes and the risks involved in basing conclusions or decisions on these data.

Errors can arise at virtually every stage of the census process, from the preparation of materials through data processing, including the listing of dwellings and the collection of data. Some errors occur at random, and when the individual responses are aggregated for a sufficiently large group, such errors tend to cancel out. For errors of this nature, the larger the group, the more accurate the corresponding estimate. It is for this reason that users are advised to be cautious when using small area estimates. There are some errors, however, which might occur more systematically, and which result in 'biased' estimates. Because the bias from such errors is persistent no matter how large the group for which responses are aggregated, and because bias is particularly difficult to measure, systematic errors are a more serious problem for most data users than the random errors referred to previously.

For census data in general, the principal types of error are as follows:

  • coverage errors, which occur when dwellings or individuals are missed, incorrectly enumerated or counted more than once
  • non-response errors, which result when responses cannot be obtained from a certain number of households and/or individuals, because of extended absence or some other reason or when responses cannot be obtained from a certain number of questions in a complete questionnaire
  • response errors, which occur when the respondent, or sometimes the census representative, misunderstands a census question, and records an incorrect response or simply uses the wrong response box
  • processing errors, which can occur at various steps including coding, when 'write-in' responses are transformed into numerical codes; data capture, when responses are transferred from the census questionnaire in an electronic format, by optical character recognition methods or key-entry operators; and imputation, when a 'valid', but not necessarily correct, response is inserted into a record by the computer to replace missing or 'invalid' data ('valid' and 'invalid' referring to whether or not the response is consistent with other information on the record)
  • sampling errors, which apply only to the supplementary questions on the 'long form' asked of a one-fifth sample of households, and which arise from the fact that the responses to these questions, when weighted up to represent the whole population, inevitably differ somewhat from the responses which would have been obtained if these questions had been asked of all households.

The above types of error each have both random and systematic components. Usually, however, the systematic component of sampling error is very small in relation to its random component. For the other non-sampling errors, both random and systematic components may be significant.

Top of page

Coverage errors

Coverage errors affect the accuracy of the census counts, that is, the sizes of the various census universes: population, families, households and dwellings. While steps have been taken to correct certain identifiable errors, the final counts are still subject to some degree of error because persons or dwellings have been missed, incorrectly enumerated in the census or counted more than once.

Missed dwellings or persons result in undercoverage. Dwellings can be missed because of the misunderstanding of collection unit (CU) boundaries, or because either they do not look like dwellings or they appear uninhabitable. Persons can be missed when their dwelling is missed or is classified as vacant, or because the respondent misinterprets the instructions on whom to include on the questionnaire. Some individuals may be missed because they have no usual residence and did not spend census night in a dwelling.

Dwellings or persons incorrectly enumerated or double-counted result in overcoverage. Overcoverage of dwellings can occur when structures unfit for habitation are listed as dwellings (incorrectly enumerated), when there is a certain ambiguity regarding the collection unit (CU) boundaries or when units (for example, rooms) are listed separately instead of being treated as part of one dwelling (double-counted). Persons can be counted more than once because their dwelling is double counted or because the guidelines on whom to include on the questionnaire have been misunderstood. Occasionally, someone who is not in the census population universe, such as a foreign resident or a fictitious person, may, incorrectly, be enumerated in the census. On average, overcoverage is less likely to occur than undercoverage and, as a result, counts of dwellings and persons are likely to be slightly underestimated.

For the 2006 Census, three studies are used to measure coverage error. In the Dwelling Classification Study, dwellings listed as vacant were revisited to verify that they were vacant on Census Day, and dwellings whose households were listed as non-respondent were revisited to determine the number of usual residents and their characteristics. Adjustments have been made to the final census counts to account for households and persons missed because their dwelling was incorrectly classified as vacant. The census counts may also have been adjusted for dwellings whose households were classified as non-respondent. Despite these adjustments, the final counts still may be subject to some undercoverage. Undercoverage tends to be higher for certain segments of the population, such as young adults (especially young adult males) and recent immigrants. The Reverse Record Check Study is used to measure the residual undercoverage for Canada, and each province and territory. The Overcoverage Study is designed to investigate overcoverage errors. The results of the Reverse Record Check and the Overcoverage Study, when taken together, furnish an estimate of net undercoverage.

Other non-sampling errors

Top of page

While coverage errors affect the number of units in the various census universes, other errors affect the characteristics of those units.

Sometimes it is not possible to obtain a complete response from a household, even though the dwelling was identified as occupied and a questionnaire was mailed out or dropped off. The household members may have been away throughout the census period or, in rare instances, the householder may have refused to complete the form. More frequently, the questionnaire is returned but no response is provided to certain questions. Effort is devoted to ensure as complete a questionnaire as possible. Once the questionnaires are captured, edit analysis are performed to detect significant cases of partial non-response and follow-up interviews are attempted to get the missing information. Despite this, at the end of the collection stage, a small number of responses are still missing, i.e., non-response errors. Although missing responses are eliminated during processing by replacing each one of them by the corresponding response for a 'similar' record, there remain some potential imputation errors. This is particularly serious if the non-respondents differ in some respects from the respondents; this procedure will then introduce a non-response bias.

Even when a response is obtained, it may not be entirely accurate. The respondent may have misinterpreted the question or may have guessed the answer, especially when answering on behalf of another, possibly absent, household member. The respondent may also have entered the answer in the wrong place on the questionnaire. Such errors are referred to as response errors. While response errors usually arise from inaccurate information provided by respondents, they can also result from mistakes by the census representative who completed certain parts of the questionnaire, such as the structural type of dwelling, or who followed up to obtain a missing response.

Some of the census questions require a written response. During processing, these 'write-in' entries are given a numeric code. Coding errors can occur when the written response is ambiguous, incomplete, and difficult to read or when the code list is extensive (e.g., major field of study, place of work). A formal quality control (QC) operation is used to detect, rectify and reduce coding errors. Within each work unit, a sample of responses is independently coded a second time. The resolution of discrepancies between the first and second codings determines whether recoding of the work unit is necessary. Census coding is now entirely automated, resulting in a reduction of coding errors.

The information on the questionnaires is scanned and captured into a computer file. To monitor and to ensure that the number of data capture errors are within tolerable limits, a sample of fields are sampled and reprocessed. Analysis of the two captures is done. Unsatisfactory work is identified, corrected and appropriate feedback is done to the system in order to minimize their occurrence.

Once captured, the data are edited where they undergo a series of computer checks to identify missing or inconsistent responses. These are replaced during the imputation stage of processing where either a response consistent with the other respondents' data is inferred or a response from a similar donor is substituted. Imputation ensures a complete database where the data correspond to the census counts and facilitate multivariate analyses. Although errors may have been introduced during imputation, the methods used have been rigorously tested to minimize systematic errors.

Top of page

Various studies are being carried out to evaluate the quality of the responses obtained in the 2006 Census. For each question, non-response rates and edit failure rates have been calculated. These can be useful in identifying the potential for non-response errors and other types of errors. Also, tabulations from the 2006 Census have been or will be compared with corresponding estimates from previous censuses, from sample surveys (such as the Labour Force Survey) and from various administrative records (such as birth registrations and municipal assessment records). Such comparisons can indicate potential quality problems or at least discrepancies between the sources.

In addition to these aggregate-level comparisons, there are some micro-match studies in progress, in which census responses are compared with another source of information at the individual record level. For certain 'stable' characteristics (such as age, sex, mother tongue and place of birth), the responses obtained in the 2006 Census, for a sample of individuals, are being compared with those for the same individuals in the 2001 Census.

Sampling errors

Estimates obtained by weighting up responses collected on a sample basis are subject to error due to the fact that the distribution of characteristics within the sample will not usually be identical to the distribution of characteristics within the population from which the sample has been selected.

The potential error introduced by sampling will vary according to the relative scarcity of the characteristics in the population. For large cell values, the potential error due to sampling, as a proportion of the cell value, will be relatively small. For small cell values, this potential error, as a proportion of the cell value, will be relatively large.

The potential error due to sampling is usually expressed in terms of the so-called 'standard error'. This is the square root of the average, taken over all possible samples of the same size and design, of the squared deviation of the sample estimate from the value for the total population.

The following table provides approximate measures of the standard error due to sampling for census long form (2B) data. These measures are intended as a general guide only.

Approximate standard error due to sampling for 2006 Census sample data

Cell value Approximate standard error
50 or less 15
100 20
200 30
500 45
1,000 65
2,000 90
5,000 140
10,000 200
20,000 280
50,000 450
100,000 630
500,000 1,400

Users wishing to determine the approximate error due to sampling for any given cell of data, based upon the 20% sample, should choose the standard error value corresponding to the cell value that is closest to the value of the given cell in the census tabulation. When using the obtained standard error value, the user, in general, can be reasonably certain that, for the enumerated population, the true value (discounting all forms of error other than sampling) lies within plus or minus three times the standard error (e.g., for a cell value of 1,000, the range would be 1,000 ± [3 x 65] or 1,000 ± 195).

Top of page

The standard errors given in the table above will not apply to population, household, dwelling or family counts for the geographic area under consideration (see Sampling and weighting below). The effect of sampling for these cells can be determined by a comparison with a corresponding 100% data product.

The effect of the particular sample design and weighting procedure used in the 2006 Census will vary, however, from one characteristic to another and from one geographic area to another. The standard error values in the table may, therefore, understate or overstate the error due to sampling.

Sampling and weighting

The 2006 Census data were collected either from 100% of the households or on a sample basis with the data weighted to provide estimates for the entire population. The long form questionnaire (2B) information was collected on a 20% random sample basis of the households and weighted to compensate for sampling. All table headings are noted accordingly. Note that on Indian reserves and in remote areas all data were collected on a 100% basis.

For any given geographic area, the weighted population, household, dwelling or family total or subtotal may differ from that shown in reports containing data collected on a 100% basis. Such variations are due to sampling and to the fact that, unlike sample data, 100% data do not exclude institutional residents.

Confidentiality and random rounding

The figures shown in the tables have been subjected to a confidentiality procedure known as random rounding to prevent the possibility of associating statistical data with any identifiable individual. Under this method, all figures, including totals and margins, are randomly rounded either up or down to a multiple of '5', and in some cases '10'. While providing strong protection against disclosure, this technique does not add significant error to the census data. The user should be aware that totals and margins are rounded independently of the cell data so that some differences between these and the sum of rounded cell data may exist. Also, minor differences can be expected in corresponding totals and cell values among various census tabulations. Similarly, percentages, which are calculated on rounded figures, do not necessarily add up to 100%. Order statistics (median, quartiles, percentiles, etc.) and measures of dispersion such as the standard error are computed in the usual manner. When a statistic is defined as the quotient of two numbers (which is the case for averages, percentages, and proportions), the two numbers are rounded before the division is performed. For income, owner's payments, value of dwelling, hours worked, weeks worked and age, the sum is defined as the product of the average and the rounded weighted frequency. Otherwise, it is the weighted sum that is rounded. It should also be noted that small cell counts may suffer a significant distortion as a result of random rounding. Individual data cells containing small numbers may lose their precision as a result. Also, a statistic is suppressed if the number of actual records used in the calculation is less than 4 or if the sum of the weight of these records is less than 10. In addition, for values expressed in dollar units, the statistic is suppressed if the range of the values is too narrow or if all values are less than, in absolute value, a specified threshold. Finally, again for values expressed in dollar units, the statistic is suppressed if there is a dollar value too large compared to all the others.

Top of page

Users should be aware of possible data distortions when they are aggregating these rounded data. Imprecisions as a result of rounding tend to cancel each other out when data cells are re-aggregated. However, users can minimize these distortions by using, whenever possible, the appropriate subtotals when aggregating.

For those requiring maximum precision, the option exists to use custom tabulations. With custom products, aggregation is done using individual census database records. Random rounding occurs only after the data cells have been aggregated, thus minimizing any distortion.

In addition to random rounding, area suppression has been adopted to further protect the confidentiality of individual responses.

Area suppression is the deletion of all characteristic data for geographic areas with populations below a specified size. The extent to which data are suppressed depends upon the following factors:

  • if the data are tabulated from the 100% database, they are suppressed if the total population in the area is less than 40
  • if the data are tabulated from the 20% sample database, they are suppressed if the total non-institutional population in the area from either the 100% or 20% database is less than 40.

There are some exceptions to these rules:

  • income distributions and related statistics are suppressed if the population in the area, excluding institutional residents, is less than 250 from either the 100% or the 20% database, or if the number of private households is less than 40 from the 20% database
  • place-of-work distributions and related statistics are suppressed if the total number of employed persons in the area is less than 40, according to the sample database. If the data also include an income distribution, the threshold is raised to 250, again according to the sample database
  • tabulations covering both place of work and place of residence along with related statistics are suppressed, if the total number of employed persons in the area is less than 40 according to the sample database, or if the area's total population, excluding institutional residents, according to either the 100% or the sample database, is less than 40. If the tabulations also include an income distribution, the threshold is raised to 250 in all cases and the tabulations are suppressed if the number of private dwellings in the place of residence area is less than 40
  • if the data are tabulated from the 100% database and refer to six-character postal codes or to groups of either dissemination blocks or block-faces, they are suppressed if the total population in the area is less than 100
  • if the data are tabulated from the 20% sample database and refer to six-character postal codes or to groups of either dissemination blocks or block-faces, they are suppressed if the total non-institutional population in the area from either the 100% or 20% database is less than 100
  • if the data refer to groups of either dissemination blocks or block-faces, and cover place of work, they are suppressed if the total number of employed persons in the area is less than 100, according to the sample database
  • if the data refer to groups of either dissemination blocks or block-faces, and cover both place of work and place of residence, they are suppressed if the total number of employed persons in the area is less than 100, according to the sample database, or if the area's total population, excluding institutional residents, according to either 100% or the sample database, is less than 100.

In all cases, suppressed data are included in the appropriate higher aggregate subtotals and totals.

The suppression technique is being implemented for all products involving subprovincial data (i.e., Profile series, basic cross-tabulations, semi-custom and custom data products) collected on a 100% or 20% sample basis.

For further information, contact Statistics Canada's National Contact Centre.