Data Quality

Sound decision-making and policy formulation in the energy domain depends on high-quality statistical information about the supply and use of energy being available. While “quality” can carry different meanings depending on context, data quality is most commonly defined in terms of its “fitness for use” — how well the statistical outputs meet user needs. This is a relative definition: it allows for different perspectives on what constitutes quality depending on the purposes the outputs serve [IRES, Ch. IX, para. 9.3, PDF p. 128, 2018].

Data quality is not a property of final outputs alone. Energy data made available to users are the end product of a complex process spanning many stages — defining concepts and variables (energy products and flows), collecting data from various sources, processing/analysing/formatting data to meet user needs, and disseminating it, followed by an evaluation of the process and outputs to confirm objectives were met and to suggest improvements. Achieving overall data quality depends on ensuring quality at every stage of this process [IRES, Ch. IX, para. 9.1, PDF p. 128, 2018].

Quality assurance vs. quality assessment

Quality assurance comprises all planned and systematic activities that can be demonstrated to provide confidence that statistical products or services are adequate or fit for their intended uses by clients and stakeholders. It entails anticipating and avoiding problems, aiming to prevent, reduce or limit the occurrence of errors (e.g., in a survey) [IRES, Ch. IX, para. 9.4, PDF p. 128, 2018].

Quality assessment is a part of quality assurance that focuses specifically on assessing or determining the extent to which quality requirements have been fulfilled — that is, assessment is a subset of the broader assurance activity, not a separate undertaking [IRES, Ch. IX, para. 9.4, PDF p. 128, 2018].

Activities or measures for ensuring attention to data quality cover not only the final outputs, but also the organization producing them and the underlying processes leading to the outputs. Outputs/products are typically described in terms of quality dimensions such as relevance, accuracy, reliability, timeliness, punctuality, accessibility, clarity, coherence and comparability. An organization or agency exhibits high quality when it maintains a professionally independent, impartial and objective institutional environment, a commitment to quality, a guarantee of confidentiality and transparency, and adequate resources for producing outputs; for high-priority processes, sound statistical methodologies and cost-effective procedures that minimize reporting burden are paramount [IRES, Ch. IX, para. 9.5, PDF pp. 128–129, 2018].

Three lines of quality

Quality is approached along three lines:

1. Statistical product (or output) quality
2. Process quality
3. The quality or characteristics of the environment in which the office/agency operates

Chapter IX’s focus is on ensuring statistical product (or output) quality [IRES, Ch. IX, para. 9.6, PDF p. 129, 2018].

The nine quality dimensions

It is widely recognized that statistical quality is multidimensional — there is no single measure of data quality, and accuracy is no longer regarded as the sole absolute indicator of high-quality data. Data outputs are typically described, across the various quality assurance frameworks, in terms of the following dimensions, which reflect a broad perspective incorporated in most existing frameworks: relevance, accuracy, reliability, timeliness, punctuality, accessibility, clarity, coherence and comparability. These dimensions overlap and are interrelated; managing each one adequately is essential if the information produced is to be fit for use [IRES, Ch. IX, para. 9.14, PDF p. 131, 2018].

(a) Relevance. The degree to which statistical information meets or satisfies the current and/or emerging needs of key users — whether the required statistics are produced, and whether those produced are actually needed and shed light on issues of most importance to users. This requires identifying user groups and their data needs/expectations. Relevance also covers methodological soundness, particularly the extent to which concepts, definitions and classifications correspond to what users require. Relevance has three components: completeness, user needs and user satisfaction [IRES, Ch. IX, para. 9.14(a), PDF pp. 131–132, 2018].

Chapter IX Section C builds directly on these nine dimensions: quality measures and indicators are organized by dimension (Box 9.2 gives a full sample set for relevance, accuracy and reliability, timeliness and punctuality, coherence and comparability, and accessibility and clarity), and quality reports typically structure their content the same way — see Quality Measurement and Reporting [IRES, Ch. IX, para. 9.21, PDF p. 135, 2018].

(b) Accuracy and reliability. Accuracy reflects the degree to which information correctly estimates or describes the phenomena it was designed to measure — the closeness of estimates to true values. It has many facets and no single overall measure; it is usually characterized in terms of errors in statistical estimates, traditionally decomposed into bias (systematic error) and variance (random error). For energy estimates based on sample surveys, accuracy can be measured via coverage rates, sampling errors, non-response errors, response errors, processing errors, and measurement/model assumption errors. Reliability is an aspect of accuracy: whether statistics consistently measure over time the reality they are designed to represent. Regular monitoring of the nature and extent of revisions to energy statistics is a gauge of reliability [IRES, Ch. IX, para. 9.14(b), PDF p. 132, 2018].

(c) Timeliness and punctuality. Timeliness is the length of time between the end of the reference period to which information relates and its availability to users; timeliness targets derive from relevance considerations — the period for which information remains useful for its main purposes. Planned timeliness is a design decision, often trading off against accuracy and cost. Punctuality refers to whether data are delivered on the dates promised, advertised or announced (e.g., in an official release calendar). Mechanisms for managing both include announcing release dates well in advance, following up with non-responding data providers, releasing preliminary data followed by revised/final figures, using modern technology, and adhering to pre-announced release schedules (informing users of any divergences and the reasons) [IRES, Ch. IX, para. 9.14(c), PDF p. 132, 2018].

(d) Coherence and comparability. Coherence reflects the degree to which data are logically connected and mutually consistent — the degree to which they can be successfully brought together with other statistical information within a broad analytic framework over time. Comparability measures the impact of differences in applied statistical concepts, measurement tools and procedures when statistics are compared across geographical areas or over time. The use of standard concepts, definitions, classifications and target populations promotes coherence and comparability, as does a common methodology across surveys. Coherence and comparability break down into: coherence within a dataset (internal coherence, e.g. checking across products in an energy balance), coherence across datasets (e.g. checking that production and trade figures agree with economic and customs statistics, respectively), and comparability over time and across countries. The key mechanism for managing coherence and comparability of energy statistics is adherence to the methodological basis of the recommendations presented in IRES when data items are compiled, together with cooperation and knowledge exchange between statistical programmes; divergences from IRES’s recommended concepts, definitions, classifications and methodology, and breaks in series resulting from changes to them, should be explained [IRES, Ch. IX, para. 9.14(d), PDF p. 133, 2018].

(e) Accessibility and clarity. Accessibility is the ease with which users can learn of information’s existence and locate and import it into their own working environment, including the suitability of the access medium and its cost; an advance release calendar and a policy-governed provision for microdata access both promote accessibility. Clarity (sometimes called interpretability) is the extent to which easily comprehensible metadata are available where necessary for a full understanding of the statistics — covering underlying concepts and definitions, data origins, variables and classifications used, collection and processing methodology, and indications of statistical quality. User feedback (e.g., via user-satisfaction surveys) is the best way to assess clarity from the user’s perspective [IRES, Ch. IX, para. 9.14(e), PDF p. 133, 2018].

Interconnectedness and trade-offs

The nine dimensions are interconnected in a complex relationship — action taken on one dimension may affect others. The accuracy–timeliness trade-off is probably the most frequently occurring and most important: striving to improve timeliness by reducing collection and processing time may reduce accuracy. A representative case for energy statistics programmes is the trade-off between the most accurate possible estimate of total annual energy production/consumption and delivering that information while it is still of interest to users [IRES, Ch. IX, para. 9.15, PDF pp. 133–134, 2018].

Recommendation (tracker row IX/9.15): if a country cannot meet accuracy and timeliness requirements simultaneously for a given energy statistics dataset, it should produce provisional estimates — available soon after the end of the reference period but based on less comprehensive data content — and later supplement them with less-timely but more comprehensive revised/final estimates. Tracking the size and direction of revisions can then be used to assess the appropriateness of the chosen timeliness– accuracy trade-off [IRES, Ch. IX, para. 9.15, PDF pp. 133–134, 2018] — recommendations tracker row IX/9.15.

A further trade-off can arise between relevance and comparability over time: classification changes made in ongoing surveys to improve relevance can reduce comparability over time by creating breaks in series [IRES, Ch. IX, para. 9.15, PDF p. 134, 2018].

Other trade-offs. Beyond trade-offs between two output-quality dimensions, conflicts can also emerge between a quality dimension and other considerations — respondent burden, confidentiality, transparency, security or cost. For example, ensuring the efficiency or cost-effectiveness of a statistical programme may limit its flexibility to address important gaps and deficiencies, creating challenges for relevance. Such trade-offs require careful examination of all relevant factors and priorities, and decisions already made should be communicated to users along with the reasons for making them [IRES, Ch. IX, para. 9.16, PDF p. 134, 2018]. See Statistical Confidentiality for Chapter X’s dedicated treatment of the confidentiality-vs-relevance trade-off in energy statistics.

Related

• Quality Assurance Frameworks — the organizing frameworks (NQAF Template and peers) through which data quality is systematically managed
• Quality Measurement and Reporting — quality measures/indicators, Box 9.2, quality reports and quality reviews, all organized around the nine dimensions defined on this page
• Energy Balance and Statistical Difference — the statistical difference between the supply and use sides of an energy balance is itself a practical diagnostic of internal coherence, one of the nine dimensions defined above
• Official Statistics — high quality is identified as an indispensable feature of official statistics
• Scope of Energy Statistics — the statistical domain to which the quality concepts on this page apply
• IRES Chapter IX — Data Quality Assurance and Metadata — the Chapter IX digest this page supports