Metadata

Microdata, macrodata and metadata

Types of statistical data include microdata, macrodata and metadata. Microdata are non-aggregated observations or measurements of characteristics of individual units; macrodata are data derived from microdata by grouping or aggregating them; and metadata are data that describe the microdata, macrodata or other metadata [IRES, Ch. IX, para. 9.32, PDF p. 138, 2018].

Greater emphasis has been given over the years to ensuring that statistics published by national statistical offices, international organizations and other data-producing agencies are accompanied by adequate metadata. Metadata — “data about data” (and statistical metadata, “data about statistical data”) — are a specific form of documentation that defines and describes data so users can locate and understand them, make an informed assessment of their strengths, limitations, usefulness and relevance, and use and share them. Without metadata, statistical data are just numbers [IRES, Ch. IX, para. 9.33, PDF p. 138, 2018].

Structural vs. reference metadata

Metadata are important tools supporting both the production and the final use of statistical information. The main types are structural metadata and reference metadata [IRES, Ch. IX, para. 9.34, PDF p. 138, 2018].

• Structural metadata are identifiers and descriptors of the data that are essential for discovering, organizing, retrieving and processing statistical datasets — the “labels” associated with each data item for it to have meaning, such as the names of table columns, unit of measurement, time period, commodity code, etc. Structural metadata items are an integral part of the statistics database and should be extractable together with any given data item; if not associated with the data, it would be impossible to identify, retrieve and browse the data [IRES, Ch. IX, para. 9.35, PDF p. 138, 2018].
• Reference metadata describe the content and quality of the statistical data — for example, conceptual metadata describing concepts used and their practical implementation, methodological metadata describing methods used for generating the data, and quality metadata describing the different quality dimensions of the resulting statistics (timeliness, accuracy, etc.). Reference metadata are often linked (“referenced”) to the data but, unlike structural metadata, can be presented separately from the data via the Internet or in publications [IRES, Ch. IX, para. 9.36, PDF p. 138, 2018].

Metadata items and the SIMS example

When disseminating comprehensive energy statistics, the compiling agency is responsible for making the corresponding metadata available and easily accessible to users. There are numerous metadata items that describe a statistical series, and many countries and organizations have developed metadata templates, lists or inventories presenting the concepts, definitions and descriptions of methods used in the collection, compilation, transformation, revision, dissemination and evaluation of their statistics. One such comprehensive inventory is the Single Integrated Metadata Structure (SIMS) for metadata and quality reporting in the European Statistical System, whose methodological and quality metadata items are presented in full at Box 9.3. In practice, the amount of metadata detail different countries disseminate along with their energy data varies, as does the way metadata are presented; the basic purpose is always the same — to help users understand the data and their strengths and limitations [IRES, Ch. IX, para. 9.37, PDF p. 139, 2018].

Users and levels of metadata detail

There are many types of users for any given dataset. The wide range of possible users, with their different needs and statistical expertise, means a broad spectrum of metadata requirements must be addressed. Responsible agencies, as data suppliers, must make sufficient metadata available to enable both the least and the most sophisticated users to interpret and readily assess the data and their quality. It is recommended that different levels of metadata detail be made available to users to meet the requirements of the various user groups [IRES, Ch. IX, para. 9.38, PDF p. 139, 2018] — recommendations tracker row IX/9.38.

The metadata pyramid

One approach for presenting metadata is to organize it in layers within a pyramid, where methodological information describing statistics becomes more detailed moving down from the narrower apex — where summary metadata sit — to the wider base level of the “metadata pyramid,” where the most detailed metadata sit. This way, users can dig as deeply as they want or need to reach a more thorough understanding of the concepts and practices [IRES, Ch. IX, para. 9.39, PDF p. 139, 2018].

Promoting international comparability

Metadata provide a mechanism for comparing national practices in the compilation of statistics. This can help and encourage countries to implement international standards and adopt best practices in data collection in particular areas. The use of standard terminology and definitions, and better harmonization of approaches adopted by different countries, improves the general quality and coverage of key statistical indicators [IRES, Ch. IX, para. 9.40, PDF p. 139, 2018].

SDMX

Statistical Data and Metadata Exchange (SDMX) technical standards and content-oriented guidelines provide common formats and nomenclatures for exchanging and sharing statistical data and metadata using modern technology. Developing country capacity to disseminate national data and metadata using web technology and SDMX standards, such as cross-domain concepts, is recommended as a means to standardize and reduce the international reporting burden [IRES, Ch. IX, para. 9.41, PDF p. 139, 2018] — recommendations tracker row IX/9.41. See Statistical Data and Metadata Exchange for the dedicated Chapter X treatment of SDMX and international reporting.

Metadata as a high priority

It is recommended that countries accord high priority to the development of metadata, to keeping metadata up-to-date, and to treating the dissemination of metadata as an integral part of the dissemination of energy statistics. Additional country-specific metadata for purposes related to energy statistics will be presented in the forthcoming Energy Statistics Compilers Manual. Given the integrated approach to compiling economic statistics, it is also recommended that a coherent system and a structured approach to metadata across all areas of statistics be developed and adopted, focusing on improving their quantity and coverage [IRES, Ch. IX, para. 9.42, PDF p. 139, 2018] — recommendations tracker row IX/9.42.

Related

• Single Integrated Metadata Structure — the worked example of a comprehensive metadata inventory (Box 9.3 reproduced in full there)
• Data Quality — quality is one of the reference metadata types (quality metadata), and clarity (one of the nine dimensions) is defined in terms of available metadata
• Quality Measurement and Reporting — quality reports are frequently issued as part of an agency’s broader metadata
• Data Compilation Methods and Statistical Data Sources — the compilation processes that methodological (reference) metadata document
• Energy Statistics Compilers Manual — forthcoming companion guidance on country-specific metadata for energy statistics
• IRES Chapter IX — Data Quality Assurance and Metadata — the Chapter IX digest this page supports