Data quality

The 2006 Census was a large and complex undertaking and, while considerable effort was taken to ensure high standards throughout all collection and processing operations, the resulting estimates are inevitably subject to a certain degree of error. Users of census data should be aware that such error exists, and should have some appreciation of its main components, so that they can assess the usefulness of census data for their purposes and the risks involved in basing conclusions or decisions on these data.

Errors can arise at virtually every stage of the census process, from the preparation of materials through data processing, including the listing of dwellings and the collection of data. Some errors occur at random, and when the individual responses are aggregated for a sufficiently large group, such errors tend to cancel out. For errors of this nature, the larger the group, the more accurate the corresponding estimate. It is for this reason that users are advised to be cautious when using small area estimates. There are some errors, however, which might occur more systematically, and which result in 'biased' estimates. Because the bias from such errors is persistent no matter how large the group for which responses are aggregated, and because bias is particularly difficult to measure, systematic errors are a more serious problem for most data users than the random errors referred to previously.

For census data in general, the principal types of error are as follows:

• coverage errors, which occur when dwellings or individuals are missed, incorrectly enumerated or counted more than once
• non‑response errors, which result when responses cannot be obtained from a certain number of households and/or individuals, because of extended absence or some other reason or when responses cannot be obtained from a certain number of questions in a complete questionnaire
• response errors, which occur when the respondent, or sometimes the census representative, misunderstands a census question, and records an incorrect response or simply uses the wrong response box
• processing errors, which can occur at various steps including coding, when 'write‑in' responses are transformed into numerical codes; data capture, when responses are transferred from the census questionnaire in an electronic format, by optical character recognition methods or key‑entry operators; and imputation, when a 'valid', but not necessarily correct, response is inserted into a record by the computer to replace missing or 'invalid' data ('valid' and 'invalid' referring to whether or not the response is consistent with other information on the record)
• sampling errors, which apply only to the supplementary questions on the 'long form' asked of a one‑fifth sample of households, and which arise from the fact that the responses to these questions, when weighted up to represent the whole population, inevitably differ somewhat from the responses which would have been obtained if these questions had been asked of all households.

The above types of error each have both random and systematic components. Usually, however, the systematic component of sampling error is very small in relation to its random component. For the other non‑sampling errors, both random and systematic components may be significant.

Coverage errors

Coverage errors affect the accuracy of the census counts, that is, the sizes of the various census universes: population, families, households and dwellings. While steps have been taken to correct certain identifiable errors, the final counts are still subject to some degree of error because persons or dwellings have been missed, incorrectly enumerated in the census or counted more than once.

Missed dwellings or persons result in undercoverage. Dwellings can be missed because of the misunderstanding of collection unit (CU) boundaries, or because either they do not look like dwellings or they appear uninhabitable. Persons can be missed when their dwelling is missed or is classified as vacant, or because the respondent misinterprets the instructions on whom to include on the questionnaire. Some individuals may be missed because they have no usual residence and did not spend census night in a dwelling.

Dwellings or persons incorrectly enumerated or double-counted result in overcoverage. Overcoverage of dwellings can occur when structures unfit for habitation are listed as dwellings (incorrectly enumerated), when there is a certain ambiguity regarding the collection unit (CU) boundaries or when units (for example, rooms) are listed separately instead of being treated as part of one dwelling (double-counted). Persons can be counted more than once because their dwelling is double counted or because the guidelines on whom to include on the questionnaire have been misunderstood. Occasionally, someone who is not in the census population universe, such as a foreign resident or a fictitious person, may, incorrectly, be enumerated in the census. On average, overcoverage is less likely to occur than undercoverage and, as a result, counts of dwellings and persons are likely to be slightly underestimated.

For the 2006 Census, three studies are used to measure coverage error. In the Dwelling Classification Study, dwellings listed as vacant were revisited to verify that they were vacant on Census Day, and dwellings whose households were listed as non-respondent were revisited to determine the number of usual residents and their characteristics. Adjustments have been made to the final census counts to account for households and persons missed because their dwelling was incorrectly classified as vacant. The census counts may also have been adjusted for dwellings whose households were classified as non-respondent. Despite these adjustments, the final counts still may be subject to some undercoverage. Undercoverage tends to be higher for certain segments of the population, such as young adults (especially young adult males) and recent immigrants. The Reverse Record Check Study is used to measure the residual undercoverage for Canada, and each province and territory. The Overcoverage Study is designed to investigate overcoverage errors. The results of the Reverse Record Check and the Overcoverage Study, when taken together, furnish an estimate of net undercoverage.

Other non‑sampling errors

While coverage errors affect the number of units in the various census universes, other errors affect the characteristics of those units.

Sometimes it is not possible to obtain a complete response from a household, even though the dwelling was identified as occupied and a questionnaire was mailed out or dropped off. The household members may have been away throughout the census period or, in rare instances, the householder may have refused to complete the form. More frequently, the questionnaire is returned but no response is provided to certain questions. Effort is devoted to ensure as complete a questionnaire as possible. Once the questionnaires are captured, edit analysis are performed to detect significant cases of partial non-response and follow-up interviews are attempted to get the missing information. Despite this, at the end of the collection stage, a small number of responses are still missing, i.e., non‑response errors. Although missing responses are eliminated during processing by replacing each one of them by the corresponding response for a 'similar' record, there remain some potential imputation errors. This is particularly serious if the non‑respondents differ in some respects from the respondents; this procedure will then introduce a non‑response bias.

Even when a response is obtained, it may not be entirely accurate. The respondent may have misinterpreted the question or may have guessed the answer, especially when answering on behalf of another, possibly absent, household member. The respondent may also have entered the answer in the wrong place on the questionnaire. Such errors are referred to as response errors. While response errors usually arise from inaccurate information provided by respondents, they can also result from mistakes by the census representative who completed certain parts of the questionnaire, such as the structural type of dwelling, or who followed up to obtain a missing response.

Some of the census questions require a written response. During processing, these 'write‑in' entries are given a numeric code. Coding errors can occur when the written response is ambiguous, incomplete, and difficult to read or when the code list is extensive (e.g., major field of study, place of work). A formal quality control (QC) operation is used to detect, rectify and reduce coding errors. Within each work unit, a sample of responses is independently coded a second time. The resolution of discrepancies between the first and second codings determines whether recoding of the work unit is necessary. Census coding is now entirely automated, resulting in a reduction of coding errors.

The information on the questionnaires is scanned and captured into a computer file. To monitor and to ensure that the number of data capture errors are within tolerable limits, a sample of fields are sampled and reprocessed. Analysis of the two captures is done. Unsatisfactory work is identified, corrected and appropriate feedback is done to the system in order to minimize their occurrence.

Once captured, the data are edited where they undergo a series of computer checks to identify missing or inconsistent responses. These are replaced during the imputation stage of processing where either a response consistent with the other respondents’ data is inferred or a response from a similar donor is substituted. Imputation ensures a complete database where the data correspond to the census counts and facilitate multivariate analyses. Although errors may have been introduced during imputation, the methods used have been rigorously tested to minimize systematic errors.

Various studies are being carried out to evaluate the quality of the responses obtained in the 2006 Census. For each question, non-response rates and edit failure rates have been calculated. These can be useful in identifying the potential for non‑response errors and other types of errors. Also, tabulations from the 2006 Census have been or will be compared with corresponding estimates from previous censuses, from sample surveys (such as the Labour Force Survey) and from various administrative records (such as birth registrations and municipal assessment records). Such comparisons can indicate potential quality problems or at least discrepancies between the sources.

In addition to these aggregate‑level comparisons, there are some micro-match studies in progress, in which census responses are compared with another source of information at the individual record level. For certain 'stable' characteristics (such as age, sex, mother tongue and place of birth), the responses obtained in the 2006 Census, for a sample of individuals, are being compared with those for the same individuals in the 2001 Census.

For further information on the quality of census data, contact the Social Survey Methods Division at Statistics Canada, Ottawa, Ontario, Canada K1A 0T6, or by calling 613-951-4783.