=== ires-2018-page-128.pdf === 115 Chapter IX Data quality assurance and metadata A. Introduction 9.1. Ensuring data quality is a core challenge of all statistical offices and other dataproducing­agencies. The management of data quality is an integral part of every statistical domain or programme and has to be addressed in each of them. Energy data made available to users, like other subject-matter area statistics, are the end product of a complex process comprising many stages. These include the definition of concepts and variables (such as energy products and flows), the collection of data from various sources, the processing, analysis and formatting of the data to meet user needs, and data dissemination, which should be followed by an evaluation of the process and outputs to confirm that the objectives have been met and to suggest possible improvement actions. Achieving overall data quality is dependent upon ensuring quality in all stages of this process. 9.2. This chapter discusses quality concepts and quality frameworks, defines and describes the different dimensions of statistical quality and trade-offs among them, and discusses quality measures and indicators for measuring quality. Quality reports are described next, followed by a summary of the types of quality reviews that can be undertaken to evaluate statistical programmes. The chapter ends with a discussion of metadata. B. Data quality, quality assurance and quality assurance frameworks 1. Data quality 9.3. For sound decision-making and policy formulation in the energy domain, it is essential that high-quality statistical information about the supply and use of energy be available. While the word “quality” can have different meanings depending on the context in which it is used, data quality is most commonly defined in terms of its “fitness for use”, or how well the statistical outputs meet user needs. The definition is thus a relative one that allows for various perspectives on what constitutes quality, depending on the purposes for which the outputs are intended. 2. Quality assurance 9.4. Quality assurance comprises all planned and systematic activities that can be demonstrated to provide confidence that the statistical products or services are adequate or fit for their intended uses by clients and stakeholders. It entails anticipating and avoiding problems, with the goal of preventing, reducing or limiting the occurrence of errors (e.g. in a survey). It is worth noting here that quality assessment is a part of quality assurance that focuses on assessing or determining the extent to which quality requirements have been fulfilled. 9.5. Activities or measures for ensuring that attention is paid to the quality of the data cover not only the final outputs, but also the organization producing the outputs and the  === ires-2018-page-129.pdf === 116 International Recommendations for Energy Statistics (IRES) underlying processes that lead to the outputs. The outputs or products are typically described in terms of quality dimensions such as relevance, accuracy, reliability, timeliness, punctuality, accessibility, clarity, coherence and comparability. The organization or agency exhibits high quality when it maintains a professionally independent, impartial and objective institutional environment, a commitment to quality, the guarantee of confidentiality and transparency, and provides adequate resources for producing the outputs. For those processes that the organization considers to be high-priority, the use of sound statistical methodologies and cost-effective procedures that minimize the reporting burden must be paramount. 9.6. To achieve this, quality is approached along three lines: statistical product (or output) quality, process quality and the quality or characteristics of the environment in which the office/agency operates. The focus in this chapter will be on ensuring statistical product (or output) quality. 3. Data quality assurance frameworks 9.7. In the context of a statistical office, systematic data quality management typically takes the form of a quality assurance framework. A national quality assurance framework can be viewed as an overarching framework that can provide context for a country’s quality concerns, activities and initiatives, and explain the relationships between the various quality procedures and tools. Quality assurance frameworks have been developed and adopted to date by countries and international organizations to varying degrees. While all countries’ statistical offices have in place some type of quality assurance approach and a number of quality assurance procedures, and most have similar outlines of the various dimensions of quality (also referred to in the quality assurance literature as criteria, components or aspects), not all countries yet have a formalized quality assurance framework in place. 69 See http://unstats.un.org/unsd /dnss/QualityNQAF/nqaf.aspx. 70 The NQAF Template is available from http://unstats.un.org/unsd /dnss/QualityNQAF/nqaf.aspx, the ES Code of Practice from http://ec.europa.eu/eurostat /documents/3859598/5921861 /KS-32-11-955-EN.PDF/ 5fa1ebc6-90bb-43fa-888fdde032471e15 and http://ec .europa.eu/eurostat/documents /3859598/5923349/QAF_2012-EN .PDF/fcdf3c44-8ab8-41b8-9fd091bd1299e3ef?version=1.0; IMF’s DQAF from http://dsbb .imf.org/images/pdfs/dqrs_ Genframework.pdf; the LAC’s Code of Good Practice in Statistics at www.dane.gov.co /files/noticias/BuenasPracticas_ en.pdf; and Statistics Canada’s Quality Assurance Framework from www.statcan.gc.ca/pub/12586-x/12-586-x2002001-eng.pdf. 9.8. In 2012, the United Nations Statistical Commission endorsed the generic National Quality Assurance Framework (NQAF) Template developed by the Expert Group on National Quality Assurance Frameworks to assist countries in formulating and operationalizing their national quality assurance frameworks or to further enhance existing ones. The work of the Expert Group built upon and helped raise greater awareness of the various data quality management references and tools developed by international, regional, national and other organizations. They are posted on the United Nations Statistics Division (UNSD) NQAF website.69 9.9. The NQAF Template drew heavily upon, and was designed to be in close alignment with, other main frameworks, i.e., the European Statistics Code of Practice, the International Monetary Fund (IMF) Data Quality Assessment Framework (DQAF), Statistics Canada Quality Assurance Framework, and the Code of Good Statistical Practice for Latin America and the Caribbean,70 which have been successfully adopted by many countries and continue to be in use in them. Although these quality frameworks may differ slightly from each other, they share common aspects and provide comprehensive and flexible structures for the qualitative assessment of a broad range of statistics, including energy statistics. They also facilitate taking stock of quality concerns, activities, requirements and initiatives and the fostering of standardization and systematization of quality practices and measurement within statistical offices and across countries. A mapping of each of them to the NQAF Template is available on the UNSD NQAF website. 9.10. The NQAF Template is presented in box 9.1 below. Its five sections outline the elements that a national quality assurance framework should include. The paragraphs on quality assurance that follow focus mainly on the NQAF Template sections 3 and 4, and provide an overview of quality assurance objectives, considerations and practices, including measurement, reporting and evaluation. Additional information on other frameworks can be found in the Energy Statistics Compilers Manual.  === ires-2018-page-130.pdf === Data quality assurance and metadata Box 9.1 Template for a Generic National Quality Assurance Framework (NQAF) 1. Quality context 1a. Circumstances and key issues driving the need for quality management 1b. Benefits and challenges 1c. Relationship to other statistical agency policies, strategies and frameworks and evolution over time 2. Quality concepts and frameworks 2a. Concepts and terminology 2b. Mapping to existing frameworks 3. Quality assurance guidelines 3a. Managing the statistical system [NQAF 1] Coordinating the national statistical system [NQAF 2] Managing relationships with data users and data providers [NQAF 3] Managing statistical standards 3b. Managing the institutional environment [NQAF 4] Assuring professional independence [NQAF 5] Assuring impartiality and objectivity [NQAF 6] Assuring transparency [NQAF 7] Assuring statistical confidentiality and security [NQAF 8] Assuring the quality commitment [NQAF 9] Assuring adequacy of resources 3c. Managing statistical processes [NQAF 10] Assuring methodological soundness [NQAF 11] Assuring cost-effectiveness [NQAF 12] Assuring soundness of implementation [NQAF 13] Managing the respondent burden 3d. Managing statistical outputs [NQAF14] Assuring relevance [NQAF15] Assuring accuracy and reliability [NQAF16] Assuring timeliness and punctuality [NQAF17] Assuring accessibility and clarity [NQAF18] Assuring coherence and comparability [NQAF19] Managing metadata 4. Quality assessment and reporting 4a. Measuring product and process quality—use of quality indicators, quality targets and process variables and descriptions 4b. Communicating about quality—quality reports 4c. Obtaining feedback from users 4d. Conducting assessments; labelling and certification 4e. Assuring continuous quality improvement 5. Quality and other management frameworks 5a. Performance management 5b. Resource management 5c. Ethical standards 5d. Continuous improvement 5e. Governance 117  === ires-2018-page-131.pdf === 118 International Recommendations for Energy Statistics (IRES) 4. Objectives, uses and benefits of quality assurance frameworks 9.11. The overall objective of quality assurance frameworks is to standardize and systematize quality practices and measurement within statistical offices and across countries. They are useful as organizing frameworks that provide a single place to record and reference the full range of current quality concepts, policies and practices, and for being forward-looking, since they take into account future actions and activities. For energy statistics programmes, the framework can allow the assessment of national practices in energy statistics in terms of internationally (or regionally) accepted approaches for data-quality management and measurement and facilitate reviews of a country’s energy statistics programme as performed by international organizations and other groups of data users. 9.12. The main benefits of having a quality-assurance framework in place are that it: (a) makes the processes by which quality is assured more transparent and reinforces the image of the office as a credible provider of good-quality statistics; (b) creates a quality culture within the organization; (c) guides countries in strengthening their statistical systems by promoting self-assessments to identify quality problems; and (d) facilitates the exchange of ideas on quality management with other producers of statistics at the national, regional and international levels. 9.13. For those energy statistics programmes without a quality-assurance framework yet in place, national statistical offices, ministries and/or agencies responsible for energy statistics can avoid “reinventing the wheel” by reviewing the above-mentioned frameworks and considering whether to directly follow one or structure their own in line with one or some of them in a way that best fits their country’s practices and circumstances. Countries are encouraged to develop their own national quality assurance frameworks based on the above-mentioned approaches or other internationally recognized approaches, taking into consideration their specific national circumstances. 5. 71 Some frameworks also include other dimensions, for example, interpretability (which is similar to clarity), credibility, integrity, serviceability, etc. Dimensions of quality 9.14. It is widely recognized that the concept of quality in relation to statistical information is multidimensional; there is no one single measure of data quality, and no longer is accuracy thought to be the one absolute measure or indicator of high-quality data. Data outputs are typically described in the various quality assurance frameworks in terms of several dimensions or components of quality. The dimensions are assessed, measured, reported on and monitored over time to provide an indication of output quality to both the data users and data producers. The following dimensions of quality reflect a broad perspective and have been incorporated in most of the existing frameworks: relevance, accuracy, reliability, timeliness, punctuality, accessibility, clarity, coherence and comparability.71 As the dimensions of quality are overlapping and interrelated, the adequacy of the management of each of them is essential if the information produced is to be fit for use. They should be taken into account when describing, measuring and reporting the quality of statistics in general and energy statistics in particular. (a) Relevance. The relevance of statistical information reflects the degree to which the information meets or satisfies the current and/or emerging needs of key users. Relevance therefore refers to whether the required statistics are produced and whether those produced are in fact needed and shed light on the issues of most importance to users. To know this requires the identification of user groups and knowledge about their various data needs and expectations. Relevance also covers methodological soundness, particularly the extent to which the concepts, definitions and classifications correspond to those that users require. Relevance can be seen as having the following three components: completeness, user needs and user satisfaction.  === ires-2018-page-132.pdf === Data quality assurance and metadata An energy statistics programme’s challenge would be to weigh and balance the conflicting needs of its current and potential users in order to produce energy statistics that satisfy the most important needs of key users in terms of the data’s content, coverage, timeliness, etc., within given resource constraints. To ensure or manage relevance, producers must engage with their users and data providers before and during the production process, as well as after the outputs have been released. Some strategies for measuring the relevance of an energy programme’s outputs include consulting directly with key users about their needs, priorities and views concerning any deficiencies in the programme, tracking requests from users and evaluating the ability of the programme to respond, and analysing the results of user-satisfaction surveys. Also, since needs evolve over time, ongoing statistical programmes should be regularly reviewed to ensure their continued relevance. (b) Accuracy and reliability. The accuracy of statistical information reflects the degree to which the information correctly estimates or describes the phenomena it was designed to measure, i.e. the degree of closeness of estimates to true values. It has many facets, and there is no single overall measure of accuracy. It is usually characterized in terms of errors in statistical estimates and is traditionally decomposed into bias (systematic error) and variance (random error) components. In the case of energy estimates based on data from sample surveys, the accuracy can be measured using the following indicators: coverage rates, sampling errors, non-response errors, response errors, processing errors, and measurement and model assumption errors. Reliability is an aspect of accuracy. It concerns whether the statistics consistently measure over time the reality they are designed to represent. The regular monitoring of the nature and extent of revisions to energy statistics is considered a gauge of reliability. (c) Timeliness and punctuality. Timeliness of information refers to the length of time between the end of the reference period to which the information relates and its availability to users. Timeliness targets are derived from considerations of relevance, in particular the period for which the information remains useful for its main purposes. This varies with the rate of change of the phenomena being measured, the frequency of measurement, and the immediacy of user response to the latest data. Planned timeliness is a design decision, often based on a trade-off between accuracy and cost. Thus, improved timeliness is not an unconditional objective. However, timeliness is an important characteristic that should be monitored over time to provide a warning of deterioration, especially as the timeliness expectations of users are likely to heighten as they continue to experience faster and faster service delivery, thanks to the impact of technology. Punctuality refers to whether data are delivered on the dates promised, advertised or announced (for example, in an official release calendar). Mechanisms for managing timeliness and punctuality include announcing release dates well in advance, implementing follow-up procedures with data providers if they have not responded by the specified deadlines, releasing preliminary data followed by revised and/or final figures, making the best use of modern technology and adhering to the pre-announced release schedules (and if necessary, informing users of any divergences from the advance release calendar and the reasons for the delays). Paying attention to timeliness and punctuality and announcing schedules and release dates in advance help users plan, provide internal discipline and guarantee equal access to all by undermining any potential effort by interested parties to influence or delay any particular release for their own benefit. 119  === ires-2018-page-133.pdf === 120 International Recommendations for Energy Statistics (IRES) (d) Coherence and comparability. The coherence of energy statistics reflects the degree to which the data are logically connected and mutually consistent, that is to say, the degree to which they can be successfully brought together with other statistical information within a broad analytic framework over time. Comparability is a measurement of the impact of differences in applied statistical concepts, measurement tools and procedures, when statistics are compared between geographical areas or over time. The use of standard concepts, definitions, classifications and target populations promotes coherence and comparability, as does the use of a common methodology across surveys. The concepts of coherence and comparability can be broken down into coherence within a dataset (internal coherence, e.g. checking across products in an energy balance), coherence across datasets (e.g. checking that concepts such as production and trade agree with economic and customs statistics, respectively), and comparability over time and across countries. Mechanisms for managing the coherence and comparability of energy statistics include adherence to the methodological basis of the recommendations presented in IRES when data items are compiled and the promotion of cooperation and exchange of knowledge between individual statistical programmes. Automated processes and methods, such as coding tools, can be used to identify issues and promote coherence and consistency within a dataset. The use of common concepts, definitions, classifications and methodology will result in coherence across datasets (e.g. between energy and other statistics such as economic and environmental), and comparability over time and across countries. Divergences from the recommendations and common concepts, definitions, classifications and methodology, as well as breaks in series resulting from changes in the concepts, definitions, etc. should be explained. (e) Accessibility and clarity. Accessibility of information refers to the ease with which users can learn of its existence and locate and import it into their own working environment. It includes the suitability of the form or medium through which the information can be accessed and its cost. An advance release calendar or timetable to inform users about when and where the data will be available and how to access them promotes accessibility and also enables equal access to information for all groups of users. A provision for allowing access to microdata for research purposes, in accordance with an established policy that ensures statistical confidentiality, also promotes accessibility. Clarity refers to the extent to which easily comprehensible metadata are available in cases where the metadata are necessary to give a full understanding of the statistics. It is sometimes referred to as interpretability. The clarity dimension is fulfilled by the existence of user-support services and the provision of metadata, which should cover the underlying concepts and definitions, origins of the data, the variables and classifications used, the methodology of data collection and processing, and indications of the quality of the statistical information. User feedback is the best way to assess the clarity of data from the user’s perspective, e.g. through questions regarding their understanding and interpretation in user satisfaction surveys. 6. Interconnectedness and trade-offs 9.15. The dimensions of quality described above are interconnected and as such are involved in a complex relationship. Action taken to address or modify one dimension of quality may affect other dimensions. The accuracy-timeliness trade-off is probably the most frequently occurring and most important of the trade-offs. For example, striving for improvements in timeliness by reducing collection and processing time may reduce accuracy. A similar  === ires-2018-page-134.pdf === Data quality assurance and metadata situation requiring consideration by energy statistics programmes, for example, would be the trade-off between aiming for the most accurate estimation of the total annual energy production or consumption by all potential producers and consumers, and providing this information in a timely manner when it is still of interest to users. It is recommended that if, while compiling a particular energy statistics dataset, countries are not in a position to meet the accuracy and timeliness requirements simultaneously, they should produce provisional estimates, which would be available soon after the end of the reference period but would be based on less comprehensive data content. These estimates would be supplemented at a later date with information based on more comprehensive data content but would be less timely than their provisional version. In such cases, the tracking of the size and direction of revisions can serve to assess the appropriateness of the chosen timeliness-accuracy trade-off. Additional trade-offs, such as those between relevance and comparability over time, may need to be dealt with when changes made in classifications used in ongoing surveys to improve relevance lead to reductions in comparability over time due to breaks in series. 9.16. Other trade-offs. The trade-offs described above relate to those between two dimensions of output quality. Other conflicting situations may emerge requiring difficult tradeoffs, such as those between one of the dimensions and such quality considerations as the respondent burden, confidentiality, transparency, security or costs. For example, ensuring the efficiency or cost-effectiveness of the statistical programme may create challenges for ensuring relevance by limiting the flexibility of the programme to address important gaps and deficiencies. A careful examination of all relevant factors and priorities will be required to make the necessary decisions relating to these types of difficult trade-offs, and those decisions already made should be communicated to users, along with the reasons for making those decisions. C. Measuring and reporting on the quality of statistical outputs 1. Quality measures and indicators 9.17. There are essentially two ways to measure quality—using quality measures and quality indicators. The quantitative and qualitative quality measures and indicators developed around dimensions such as those described above enable data producers to describe, measure, assess and report output quality to assist users in determining whether the outputs are fit for their intended purposes. The measures and indicators can also be used by data producers to monitor data quality for the purpose of continuous improvement. 9.18. Quality measures are defined as those items that directly measure a particular aspect of quality. For example, the time lag from the reference date to the day of publication of particular energy statistics, measured in the number of days, weeks or months, is a direct quality measure of timeliness. In practice, many other quality measures can be difficult and costly to calculate. In these cases, quality indicators can be used to supplement or act as substitutes for the desired quality measures. 9.19. Quality indicators usually consist of information that is a by-product of the statistical process. They do not measure quality directly but can provide enough information to give an insight into quality. For example, in the case of accuracy, measuring non-response bias is challenging, since the characteristics of the non-responders can be difficult and costly to ascertain. In this instance, response rates are often utilized as a proxy to provide a quality indicator of the possible extent of non-response bias. Other data sources can also serve as a quality indicator to validate or confront the data. For example, commodity balances can be used to compare energy consumption data with energy supply figures (in the statistical difference flow) to flag potential problem areas. 121  === ires-2018-page-135.pdf === 122 International Recommendations for Energy Statistics (IRES) 2. Examples and selection of quality measures and indicators 9.20. There are numerous examples of quality indicators and measures that have already been defined around specific dimensions and are in use by statistical organizations. Some are presented in the form of descriptive statements or assertions (e.g. the majority of the indicators of good practice relating to the principles of the European Statistics Code of Practice; the “elements to be assured” in the NQAF Guidelines and NQAF Checklist, and those relating to the IMF “elements of good practice” in the DQAF and the “compliance criteria” in the Code of Good Statistical Practice for Latin America and the Caribbean). Others may be quantitative statements or quantified measures calculated according to specific formulas (e.g. the ESS Standard Quality and Performance Indicators). The various quality indicators and measures are intended to make the description of a product by quality dimensions more informative and increase transparency. Countries are encouraged to develop or identify quality measures and indicators for describing, measuring, assessing, documenting and monitoring over time the quality of their energy statistics outputs and make them available to users. The Energy Statistics Compilers Manual presents several sets of indicators for consideration and selection for describing the quality of statistical outputs in general. 9.21. The objective of quality measurement is to have a practical set (limited number) of quality measures and indicators to describe and monitor over time the quality of the data produced by the responsible agencies and to ensure that users are provided with a useful summary of overall quality, while not overburdening respondents with demands for unrealistic amounts of metadata. As such, it is not intended that all quality measures and indicators be addressed for all data. Instead, countries are encouraged to select practical sets of quality measures and indicators that are most relevant to their specific outputs and can be used to describe and monitor the quality of the data over time. They should also ensure that the selected measures and indicators cover each of the quality dimensions that describe their outputs, have well-established methodologies for their compilation, and are easy to interpret by both internal and external users. Box 9.2 presents a sample of some indicators and measures that countries’ energy statistics programmes can consider using to indicate the quality of their energy statistics. 9.22. Data compilers should decide how frequently the measures or indicators for different key outputs are to be produced. Certain types of quality measures and indicators can be produced for each data item in line with the frequency of production or publication of the data. For example, response rates for total energy production can be calculated and disseminated with each new estimate. However, other measures could be produced once for longer periods and only produced again for newly released data if there were major changes. 3. Quality reports 9.23. In order for the users of energy statistics to be able to make informed use of the statistical information provided, they need to know whether the data are of sufficient quality. For some dimensions of quality, such as timeliness, users are able to easily assess the quality for themselves, while others, such as coherence and even relevance, may not be as obvious. The dimension of accuracy in particular is one that users may often have no way of assessing and must rely on the statistical agency for guidance to assess. A quality report or similar documentation is meant to provide this guidance. 9.24. National practices for reporting on the quality of outputs vary. The quality of documentation provided by data producers can range from short and concise to very detailed, depending on the users for whom the information is meant. General users will most likely only be interested in a level of detail necessary for knowing whether the data is reliable, while producers will want more detailed information to be able to evaluate whether the output meets the quality requirements and to identify strengths and areas that might need further improvement.  === ires-2018-page-136.pdf === 123 Data quality assurance and metadata Box 9.2 Selected Indicators for Measuring the Quality of Energy Statistics72 Quality dimension Quality measure/indicator Relevance •• Procedures are in place to identify the users of energy data and consult with them about their needs; •• Unmet user needs—Gaps between key user needs and compiled energy statistics in terms of concepts, coverage and detail are identified and addressed; •• Requests for energy information are monitored and the capacity to respond is evaluated; •• User satisfaction surveys on the agency’s energy statistics outputs are regularly conducted and the results are analysed and acted upon; Accuracy and reliability •• Energy source data are systematically assessed and validated; •• Sampling errors of estimates, e.g. standard errors, are measured, evaluated and systematically documented; •• Non-sampling errors, e.g. item non-response rates and unit non-response rates, are measured, evaluated and systematically documented; •• Coverage—The proportion of the population covered by the energy data collected is assessed; •• Imputation rates are reported; •• Information on the size and direction of revisions to energy data is provided and made known publicly; Timeliness and punctuality •• A published release calendar announces in advance the dates that (key) energy statistics are to be released; •• The time lag between the end of the reference period and the date of the first release (or the release of final results) of energy data is monitored and reported; •• The possibility and usefulness of releasing preliminary data is regularly considered, while at the same time taking into account the data’s accuracy; •• The time lag between the date of the release or publication of the data and the date on which they were announced or promised to be released is monitored and reported; •• Any divergences from pre-announced release times for the energy data are published in advance; a new release time is then announced with explanations on the reasons for the delays; Coherence and comparability •• Comparison and joint use of related energy data from different sources are made; •• Energy statistics are comparable over a reasonable period of time; •• Divergences from the relevant international statistical standards in concepts and measurement procedures used in the collection/compilation of energy statistics are monitored and explained; •• Energy statistics are internally coherent and consistent; Accessibility and clarity •• Energy statistics and the corresponding metadata are presented in a form that facilitates proper interpretation and meaningful comparisons, and are archived; •• Modern information and communication technology (ICT) is mainly used for dissem­ inating energy statistics; traditional hard copy and other services are provided, when appropriate, to ensure that users have appropriate access to the statistics they need; •• An information or user-support service, call centre or hotline is available for handling requests for energy data and for providing answers to questions about statistical results, metadata, etc.; •• Access to energy microdata is allowed for research purposes, subject to specific rules and protocols on statistical confidentiality; •• The regular production of up-to-date quality reports and methodological documents (on energy concepts, definitions, scope, classifications, basis of recording, data sources (including the use of administrative data), compilation methods, statistical techniques, etc.) is part of the work programme, and the reports and documents are made known publicly; 72 The indicators listed represent only a sample of possible indicators that can be used for measuring quality. Refer to the Energy Statistics Compilers Manual for more information.  === ires-2018-page-137.pdf === 124 International Recommendations for Energy Statistics (IRES) 9.25. Quality information is often structured in a template format to promote comparability and consistency across statistical domains. Sometimes it is issued in a quality report that is separate from the other metadata—not as a replacement for, but as a complement to them. Other times, it may be included as part of other metadata (for example along with explanatory and technical notes and other more detailed documentation) provided by the compiling agency. Some compilers refer to it as a quality statement or quality declaration. Typically though, the quality reports or quality documentation examine and describe quality according to the dimensions used by the agency to define its products’ fitness for purpose in terms of relevance, accuracy, reliability, timeliness, punctuality, coherence, comparability, accessibility and clarity, as focused on in this chapter. 9.26. Two types of quality reports can be distinguished—the shorter “user-oriented” report and the more detailed “producer-oriented” report. The focus of the user-oriented reports is on output quality, so they are often limited to brief descriptions of the output dimensions and generally include just a few of the indicators for measuring quality listed in the previous section. On the other hand, the longer “producer-oriented” quality reports, such as the comprehensive type European Statistical System (ESS) members are recommended to produce periodically (every five years or so or after major changes), go into greater detail on the dimensions, especially on errors and other aspects affecting accuracy, and provide additional information on the processes and other issues, such as confidentiality, costs and the response burden. For the users, such details may be confusing and unnecessary for their purposes, but for the producers, the comprehensive reports serve as an internal self-assessment. Quality reporting therefore underpins quality assessment, which in turn is the starting point for quality improvements in statistical programmes. See the Energy Statistics Compilers Manual for more information on quality reports and descriptions of quality reporting practices. 9.27. The preparation and updating of quality reports depend on the survey frequency and the stability of the quality characteristics. Balance should be sought between the need for recent information and the report compiling burden. If necessary, the quality report should be updated as frequently as the survey is carried out. However, if the characteristics are stable, the inclusion of quality indicators in the newest survey results could be enough to update the report. Another option is to provide a detailed quality report less frequently, and a shorter one after each survey, covering only the updated characteristics, such as some of the accuracy-related indicators. Countries are encouraged to regularly issue quality reports as part of their metadata. 4. 73 For more information, see for example, the European Statistical System’s Development of a Self-Assessment Programme (DESAP) and Data Quality Assessment and Tools, see http://ec.europa.eu/eurostat /documents/64157/4373903/07Checklist-for-Survey-Managers_ DESAP-EN.pdf/ec76e3a3-46b5409e-a7c3-52305d05bd42. Quality reviews 9.28. Quality reviews can be done in the form of self-assessments, audits or peer reviews. They can be undertaken by internal or external experts, and the timeframe can vary from days to months, depending on the scope of the review. However, the results are more or less identical—the identification of improvement actions/opportunities in processes and products. It is recommended that some form of quality review of energy statistics programmes be undertaken periodically, for example every four to five years or more frequently, if significant methodological or other changes in the data sources occur. 9.29. Self-assessments are comprehensive, systematic and regular reviews of an organization’s activities and results are referenced against a model/framework. They are “do it yourself” evaluations. Typically, self-assessment checklists or questionnaires are developed to be used for the systematic assessment of the quality of statistical production processes.73 9.30. A quality audit is a systematic, independent and documented process for obtaining quality evidence concerning the quality of a statistical process and evaluating it objectively to determine the extent to which policies, procedures and requirements on quality are fulfilled.  === ires-2018-page-138.pdf === Data quality assurance and metadata In contrast to the self-assessments, audits are always carried out by a third party (internal or external to the organization). Internal audits are conducted for the purpose of reviewing the quality system in place (policies, standards, procedures and methods) and the internal objectives. They are led by a team of internal quality auditors who are not in charge of the process or product under review. External audits are conducted either by stakeholders or other parties with an interest in the organization, by an external and independent auditing organization, or by a suitably qualified expert. 9.31. Peer reviews are a type of external audit which aim to assess a statistical process at a higher level, not to check conformity with requirements item by item from a detailed checklist. They are therefore often more informal and less structured than formal external audits. Normally, peer reviews do not address specific aspects of data quality but focus rather on broader organizational and strategic questions. They are typically systematic examinations and assessments of the performance of one organization by another, with the ultimate goal of helping the organization under review to comply with established standards and principles, improve its policy making and adopt best practices. The assessments are conducted on a non-adversarial basis, and rely heavily on mutual trust among the organization and assessors involved, as well as their shared confidence in the process. D. Metadata on energy statistics 9.32. Types of statistical data include microdata, macrodata and metadata. Microdata are non-aggregated observations or measurements of characteristics of individual units, macrodata are data derived from microdata by grouping or aggregating them, and metadata are data that describe the microdata, macrodata or other metadata. This section of the chapter will focus on metadata. 9.33. Greater emphasis has been given over the years to the importance of ensuring that statistics published by national statistical offices, international organizations and other dataproducing agencies are accompanied by adequate metadata. Metadata, or the “data about data” (and statistical metadata, the “data about statistical data”), are a specific form of documentation that defines and describes data so that users can locate and understand them, make an informed assessment of their strengths, limitations, usefulness and relevance, and use and share them. Without metadata, statistical data are just numbers. 9.34. Metadata therefore are important tools that support the production and final use of statistical information. The main types of metadata are structural metadata and reference metadata. 9.35. Structural metadata are identifiers and descriptors of the data that are essential for discovering, organizing, retrieving and processing statistical datasets. They can be thought of as the “labels” associated with each data item for it to have meaning, such as the names of the table columns, unit of measurement, time period, commodity code, etc. Structural metadata items are an integral part of the statistics database and should be extractable together with any given data item. If they are not associated with the data, it would be impossible to identify, retrieve and browse the data. 9.36. Reference metadata describe the content and quality of the statistical data. They are, for example, conceptual metadata describing concepts used and their practical implementation; methodological metadata describing methods used for generating the data; and quality metadata describing the different quality dimensions of the resulting statistics, i.e. timeliness, accuracy, etc. These reference metadata are often linked (“referenced”) to the data but, unlike structural data, can be presented separately from the data via the Internet or in publications. 125  === ires-2018-page-139.pdf === 126 International Recommendations for Energy Statistics (IRES) 9.37. Metadata items. When disseminating comprehensive energy statistics, the compiling agency has the responsibility of making the corresponding metadata available and easily accessible to users. There are numerous metadata items that describe a statistical series, and many countries and organizations have developed metadata templates, lists or inventories for the presentation of the concepts, definitions and descriptions of the methods used in the collection, compilation, transformation, revision, dissemination and evaluation of their statistics. One such comprehensive inventory is the Single Integrated Metadata Structure (SIMS) for metadata and quality reporting in the ESS whose methodological and quality metadata items are presented in box 9.3. In practice, the amount of metadata detail that different countries disseminate along with their energy data varies, as does the way in which the metadata are presented. The basic purpose though is always the same—to help users understand the data and their strengths and limitations. 9.38. Users and levels of metadata detail. There are many types of users for any given set of data. The wide range of possible users, with their different needs and statistical expertise, means that a broad spectrum of metadata requirements has to be addressed. The responsible agencies, as data suppliers, must make sufficient metadata available to enable both the least and the most sophisticated users to interpret and readily assess the data and their quality. It is recommended that different levels of metadata detail be made available to users to meet the requirements of the various user groups. 9.39. An approach for presenting metadata is to organize them as if they were in layers within a pyramid, where the methodological information describing statistics becomes more detailed as one moves down from the narrower apex (where the summary metadata are) to the wider base level of the “metadata pyramid” (for the more detailed metadata). In this way, users will be able to dig as deeply as they want or need to get a more thorough understanding of the concepts and practices. 9.40. Use of metadata to promote international comparability. Metadata provide a mechanism for comparing national practices in the compilation of statistics. This may help and encourage countries to implement international standards and to adopt best practices in the collection of data in particular areas. The use of standard terminology and definitions, and better harmonization of approaches adopted by different countries will improve the general quality and coverage of key statistical indicators. 74 For more information on SDMX, see http://sdmx.org. 9.41. Statistical Data and Metadata Exchange (SDMX). SDMX technical standards and content-oriented guidelines provide common formats and nomenclatures for exchanging and sharing statistical data and metadata using modern technology.74 The development of capacity in countries to disseminate national data and metadata using web technology and SDMX standards such as cross-domain concepts is recommended as a means to standardize and reduce the international reporting burden. 9.42. Metadata must be a high priority. It is recommended that countries accord high priority to the development of metadata, to keeping them up-to-date, and to consider the dissemination of metadata to be an integral part of the dissemination of energy statistics. Additional country-specific metadata for purposes related to energy statistics will be presented in the forthcoming Energy Statistics Compilers Manual. In consideration of the integrated approach to the compilation of economic statistics, it is also recommended that a coherent system and a structured approach to metadata across all areas of statistics be developed and adopted, focusing on improving their quantity and coverage.  === ires-2018-page-140.pdf === 127 Data quality assurance and metadata 75 Box 9.3 Metadata items for statistical releases75 SIMS code Survey/product name S.1 Contact (organization, contact person, address, email, phone, fax) S.2 Introduction S.3 Metadata update (last certified, last posted and last update) S.4 Statistical presentation S.4.1 Data description S.4.2 Classification system S.4.3 Sector coverage S.4.4 Statistical concepts and definitions S.4.5 Statistical unit S.4.6 Statistical population S.4.7 Reference area S.4.8 Time coverage S.4.9 Base period S.5 Unit of measure S.6 Reference period S.7 Institutional mandate (legal acts and other agreements, data sharing) S.8 Confidentiality (policy, data treatment) S.9 Release policy (release calendar, calendar access, user access) S.10 Frequency of dissemination S.11 Dissemination format, accessibility and clarity (News release, publications, online database, micro-data access, other), S.12 Accessibility of documentation (documentation on methodology, quality documentation) S.13 Quality management (quality assurance, quality assessment) S.14 Relevance (user needs, user satisfaction, completeness) S.15 Accuracy and reliability (overall accuracy, sampling error, non-sampling error (coverage errors, measurement errors, non-response errors, processing errors, model assumption errors)) S.16 Timeliness (time lag to final results) and punctuality (delivery and publication) S.17 Comparability (geographical, over time) S.18 Coherence (cross-domain, internal) S.19 Cost and burden S.20 Data revision (policy, practice) S.21 Statistical processing S.21.1 Source data S.21.2 Frequency of data collection S.21.3 Data collection S.21.4 Data validation S.21.5 Data compilation S.21.6 Adjustments S.21.61 Seasonal adjustment From the Technical Manual of the Single Integrated Metadata Structure (SIMS), available from http://ec.europa.eu/eurostat /documents/64157/4373903 /03-Single-Integrated-MetadataStructure-and-its-TechnicalManual.pdf/6013a162-e8e24a8a-8219-83e3318cbb39.  === ires-2018-page-141.pdf ===