Statistical Confidentiality

Definition

Statistical confidentiality refers to the protection of data that relate to single statistical units, obtained directly for statistical purposes or indirectly from administrative or other sources, against any breach of the right to confidentiality — that is, the prevention of unlawful disclosure. Statistical confidentiality is necessary to gain and keep the trust of both those required to provide data and those using the statistical information. It must be differentiated from other forms of confidentiality under which information is not provided to the public, such as, for example, national-security concerns [IRES, Ch. X, para. 10.6, PDF p. 143, 2018].

Basis in the Fundamental Principles

Principle 6 of the United Nations Fundamental Principles of Official Statistics provides the basis for managing statistical confidentiality; it states that “individual data collected by statistical agencies for statistical compilation, whether they refer to natural or legal persons, are to be strictly confidential and used exclusively for statistical purposes” [IRES, Ch. X, para. 10.7, PDF p. 143, 2018]. See Dissemination and the full Box 1.1 list of Fundamental Principles on Official Statistics.

National statistical law

Legal provisions governing statistical confidentiality at the national level are set forth in countries’ statistical laws or other supplementary governmental regulations. National definitions of confidentiality and rules for microdata access may differ, but they should be consistent with the fundamental principle of confidentiality [IRES, Ch. X, para. 10.8, PDF p. 143, 2018]. See Legal Framework and Institutional Arrangements for Chapter VII’s treatment of the legal framework within which this national statistical law sits, and IRES Chapter VII for further discussion of mandatory reporting.

Direct vs. indirect disclosure and the identifiability test

Statistical confidentiality is protected if the disseminated data do not allow statistical units to be identified either directly or indirectly, thereby disclosing individual information. Direct identification is possible if data of only one statistical unit are reported in a cell. Indirect identification (or residual disclosure) may take place if individual data can be derived from disseminated data — for example, because there are too few units in a cell, or because of the dominance of one or two units in a cell. To determine whether a statistical unit is identifiable, account must be taken of all means that might reasonably be used by a third party to identify it. The forthcoming Energy Statistics Compilers Manual will contain a separate section on best country practices in this respect [IRES, Ch. X, para. 10.9, PDF p. 143, 2018].

The two protection factors

General rules for protecting confidentiality normally require that two factors be taken into account when deciding on the confidentiality of data: (a) the number of units in a tabulation cell, and (b) the dominance of a unit’s or units’ contribution over the total value of a tabulation cell. Application of these general rules in each statistical domain is the responsibility of national statistical authorities [IRES, Ch. X, para. 10.10, PDF p. 143, 2018].

The three protection methods

As the first step in statistical disclosure control of tabular data, sensitive cells — those that tend to directly or indirectly reveal information about individual statistical units — need to be identified. Once identified, the most common practices for protecting against disclosure of confidential data include [IRES, Ch. X, para. 10.11, PDF pp. 143–144, 2018]:

(a) Aggregation. A confidential cell in a table is aggregated with another cell, and the information is then disseminated for the aggregate rather than for the two individual cells — for example, grouping (and disseminating) data on energy production at higher levels of SIEC that adequately ensure confidentiality (see SIEC Classification System for the hierarchy this draws on) [IRES, Ch. X, para. 10.11(a), PDF pp. 143–144, 2018];

(b) Suppression. Removing records from a database or table that contain confidential data, so that values in sensitive cells are not published while original values in other cells are (primary suppression). Suppressing only one cell in a table means totals for the higher levels to which it belongs cannot be calculated, so other cells must sometimes also be suppressed to guarantee protection of the primary cells (secondary suppression). If suppression is used, it is important to indicate in the metadata which cells have been suppressed because of confidentiality [IRES, Ch. X, para. 10.11(b), PDF p. 144, 2018];

(c) Other methods. Controlled rounding and perturbation are more sophisticated techniques: controlled rounding modifies the original value of each cell by rounding it up or down to a near multiple of a base number, while perturbation is a linear-programming variant of the controlled-rounding technique [IRES, Ch. X, para. 10.11(c), PDF p. 144, 2018].

Statistical disclosure control

Recommendation (tracker row X/10.12): statistical disclosure control techniques are the set of methods used to reduce the risk of disclosing information on individual units. While applied at the dissemination stage, they are pertinent to all stages of statistical production; techniques related to dissemination usually restrict the amount of data or modify the data release, attempting to achieve an optimal balance between confidentiality protection and the provision of detailed information. On the basis of available international guidelines and national requirements, countries are encouraged to develop their own statistical disclosure methods best suited to their specific circumstances [IRES, Ch. X, para. 10.12, PDF p. 144, 2018].

The energy-specific tension: confidentiality vs. relevance

An issue of balance between applying statistical confidentiality and the need for public information exists throughout official statistics. Balancing respect for confidentiality against the need to preserve and increase the relevance of statistics is difficult; legislation on statistical confidentiality has to be carefully considered where its rigorous application would make it impossible to provide sufficient or meaningful information to the public. In official energy statistics this issue is of particular importance, since in many countries the production and distribution of energy are dominated by a very limited number of economic units [IRES, Ch. X, para. 10.13, PDF p. 144, 2018]. See Data Quality for the general quality trade-offs (including against confidentiality) discussed in Chapter IX.

The setup of energy balances illustrates the challenge. If, for instance, the transformation block of an energy balance cannot be published due to confidentiality, the quality of that balance deteriorates significantly, since it will no longer be possible to show an internally logical energy balance of energy flows from production and imports/exports through transformation to final consumption. The question is how to make it possible to publish energy balances if there are few units in one part of the balance; the confidentiality issues have to be addressed [IRES, Ch. X, para. 10.14, PDF p. 144, 2018].

Applying confidentiality rules in energy statistics

Recommendation (tracker row X/10.15): while recognizing the importance of general rules on statistical confidentiality, countries should implement them so as to promote access to data while ensuring confidentiality, and thus ensure the highest possible relevance of energy statistics, taking into account their legal circumstances. It is recommended that:

(a) any information deemed confidential (and needing to be suppressed) be reported in full at the next higher level of energy product (or energy flow) aggregation that adequately protects confidentiality;

(b) data that are publicly available (e.g., from company reports, publicly available administrative sources) be fully incorporated and disseminated;

(c) permission to disseminate certain current data, with or without a certain time delay, be requested from the concerned data reporters;

(d) passive confidentiality be considered an option — occurring when data are made confidential only at the request of the concerned economic entity and the statistical authority finds the request justified based on the adopted confidentiality rules; and

(e) proposals be formulated for including in confidentiality rules the provision that data can be disseminated if this does not entail excessive damage to the concerned entity, with the rules determining “excessive damage” clearly defined and publicly available

[IRES, Ch. X, para. 10.15(a)–(e), PDF pp. 144–145, 2018].

Related

• Dissemination — the broader dissemination-policy framework within which scope-of-data (confidentiality) decisions are made (Section A)
• Legal Framework and Institutional Arrangements — Chapter VII’s mandatory-reporting and confidentiality-protection provisions that this concept operationalizes at the dissemination stage
• Data Quality — confidentiality as one of the considerations that can trade off against a quality dimension
• SIEC Classification System — the product hierarchy used for aggregation-based confidentiality protection
• IRES Chapter X — Dissemination — the Chapter X digest this page supports