IRES Chapter IX — Data Quality Assurance and Metadata

Chapter IX (PDF pp. 128–141) discusses quality concepts and quality frameworks, defines and describes the dimensions of statistical quality and the trade-offs among them, discusses quality measures and indicators, quality reports, and types of quality reviews, and closes with a discussion of metadata [IRES, Ch. IX, para. 9.2, PDF p. 128, 2018].

This digest covers the full chapter (Sections A–D), paras 9.1–9.42, PDF pp. 128–141.

A. Introduction (paras 9.1–9.2)

Ensuring data quality is a core challenge of all statistical offices and other data-producing agencies; it must be addressed as an integral part of every statistical domain or programme. Energy data made available to users are the end product of a complex, multi-stage process — defining concepts and variables, collecting data, processing/analysing/formatting it, and disseminating it, followed by an evaluation of process and outputs. Achieving overall data quality depends on ensuring quality at every stage [IRES, Ch. IX, para. 9.1, PDF p. 128, 2018]. The chapter’s roadmap: quality concepts and frameworks, dimensions of quality and their trade-offs, quality measures/indicators, quality reports, quality reviews, and metadata [IRES, Ch. IX, para. 9.2, PDF p. 128, 2018].

B. Data quality, quality assurance and quality assurance frameworks (paras 9.3–9.16)

Full detail on this section is on Data Quality and Quality Assurance Frameworks. Summary:

• Data quality is most commonly defined as “fitness for use” — how well statistical outputs meet user needs, a relative definition that admits different perspectives depending on purpose [IRES, Ch. IX, para. 9.3, PDF p. 128, 2018].
• Quality assurance is the set of planned, systematic activities that provide confidence that statistical products/services are fit for intended use; quality assessment is the subset of assurance that determines the extent to which quality requirements have been met [IRES, Ch. IX, para. 9.4, PDF p. 128, 2018].
• Quality is approached along three lines: statistical product/output quality, process quality, and the quality of the producing environment; the chapter’s focus is on output quality [IRES, Ch. IX, para. 9.6, PDF p. 129, 2018].
• Systematic data quality management typically takes the form of a quality assurance framework. In 2012 the United Nations Statistical Commission endorsed the generic National Quality Assurance Framework (NQAF) Template, developed by the Expert Group on National Quality Assurance Frameworks [IRES, Ch. IX, para. 9.8, PDF p. 129, 2018]. The Template was built in close alignment with the European Statistics Code of Practice, the IMF Data Quality Assessment Framework, the Statistics Canada Quality Assurance Framework, and the Code of Good Statistical Practice for Latin America and the Caribbean [IRES, Ch. IX, para. 9.9, PDF p. 129, 2018].
• Box 9.1 reproduces the NQAF Template’s five sections and nineteen named NQAF elements in full — see Quality Assurance Frameworks [IRES, Ch. IX, Box 9.1, PDF p. 130, 2018].
• Benefits of quality assurance frameworks (transparency/credibility, quality culture, guided self-assessment, cross-agency exchange) [IRES, Ch. IX, paras 9.11–9.12, PDF p. 131, 2018]. Recommendation (tracker row IX/9.13): countries without a framework should review the above-mentioned ones and adopt or adapt one to their circumstances [IRES, Ch. IX, para. 9.13, PDF p. 131, 2018].
• Nine dimensions of quality, individually defined: relevance; accuracy and reliability; timeliness and punctuality; coherence and comparability (tied explicitly to adherence to IRES’s own concepts and definitions); accessibility and clarity [IRES, Ch. IX, para. 9.14(a)–(e), PDF pp. 131–133, 2018].
• Trade-offs: the accuracy–timeliness trade-off is the most frequent and important. Recommendation (tracker row IX/9.15): where a country cannot meet both simultaneously, it should produce provisional estimates followed by later, more comprehensive revised/final estimates [IRES, Ch. IX, para. 9.15, PDF pp. 133–134, 2018]. Further trade-offs can arise between relevance and comparability over time, and between a quality dimension and considerations such as respondent burden, confidentiality, transparency, security or cost [IRES, Ch. IX, paras 9.15–9.16, PDF p. 134, 2018].

C. Measuring and reporting on the quality of statistical outputs (paras 9.17–9.31)

Full detail on this section is on Quality Measurement and Reporting. Summary:

• Quality measures directly measure an aspect of quality (e.g., the publication time lag as a direct measure of timeliness); quality indicators are by-products of the statistical process that give insight into quality without measuring it directly (e.g., response rates as a proxy for non-response bias). Commodity balances comparing supply and consumption data — the statistical difference flow — are named as a quality indicator that flags potential problem areas [IRES, Ch. IX, paras 9.17–9.19, PDF p. 134, 2018].
• Recommendations (tracker rows IX/9.20, IX/9.21): countries are encouraged to develop or identify quality measures/indicators for their energy statistics outputs [IRES, Ch. IX, para. 9.20, PDF p. 135, 2018], and to select a practical (limited) set relevant to their specific outputs, covering each quality dimension [IRES, Ch. IX, para. 9.21, PDF p. 135, 2018]. Box 9.2 reproduces a full sample of indicators organized by quality dimension — see Quality Measurement and Reporting [IRES, Ch. IX, Box 9.2, PDF p. 136, 2018].
• Quality reports — a short “user-oriented” form vs. a detailed “producer-oriented” form (the latter typically produced every ~5 years or after major changes) [IRES, Ch. IX, paras 9.23–9.26, PDF pp. 135–137, 2018]. Recommendation (tracker row IX/9.27): countries are encouraged to regularly issue quality reports as part of their metadata, with cadence tied to survey frequency [IRES, Ch. IX, para. 9.27, PDF p. 137, 2018].
• Quality reviews — self-assessments, audits (internal/external) and peer reviews. Recommendation (tracker row IX/9.28): some form of quality review should be undertaken periodically, e.g. every four to five years, or more frequently after significant methodological/data-source changes [IRES, Ch. IX, paras 9.28–9.31, PDF pp. 137–138, 2018].

D. Metadata on energy statistics (paras 9.32–9.42)

Full detail on this section is on Metadata and Single Integrated Metadata Structure. Summary:

• Statistical data comprises microdata, macrodata and metadata; metadata are “data about data” — without them, statistical data are just numbers [IRES, Ch. IX, paras 9.32–9.33, PDF p. 138, 2018].
• Two main types: structural metadata (the “labels” essential for discovering/organizing/retrieving/processing data — column names, units, time period, commodity code) and reference metadata (conceptual, methodological and quality metadata describing content and quality, which can be presented separately from the data) [IRES, Ch. IX, paras 9.34–9.36, PDF p. 138, 2018].
• The Single Integrated Metadata Structure (SIMS), an ESS metadata and quality-reporting inventory, is named as a worked example; its 37-item inventory is reproduced in full in Box 9.3 — see Single Integrated Metadata Structure [IRES, Ch. IX, para. 9.37, PDF p. 139, 2018].
• Recommendation (tracker row IX/9.38): different levels of metadata detail should be made available to meet different user groups' requirements — the “metadata pyramid” layering concept [IRES, Ch. IX, paras 9.38–9.39, PDF p. 139, 2018].
• Metadata promote international comparability by helping countries adopt international standards and harmonize approaches [IRES, Ch. IX, para. 9.40, PDF p. 139, 2018].
• Statistical Data and Metadata Exchange (SDMX) is named as the technical standard for exchanging statistical data and metadata. Recommendation (tracker row IX/9.41): countries should develop capacity to disseminate data/metadata via web technology and SDMX standards [IRES, Ch. IX, para. 9.41, PDF p. 139, 2018].
• Recommendation (tracker row IX/9.42): countries should accord metadata high priority, keep it current, and adopt a coherent, structured cross-domain approach to metadata [IRES, Ch. IX, para. 9.42, PDF p. 139, 2018].

Page map

Section	Page
Data quality, quality assurance/assessment, three lines of quality, nine dimensions, trade-offs	Data Quality
Quality assurance frameworks, NQAF Template, Box 9.1 in full	Quality Assurance Frameworks
IMF Data Quality Assessment Framework (DQAF)	Data Quality Assessment Framework
Quality measures/indicators, Box 9.2 in full, quality reports, quality reviews	Quality Measurement and Reporting
Metadata concepts (structural/reference, metadata pyramid, SDMX)	Metadata
Single Integrated Metadata Structure (SIMS), Box 9.3 in full	Single Integrated Metadata Structure

---

Full text: Cleaned copy