Quality Measurement and Reporting

Once the nine quality dimensions are defined, a statistical agency needs concrete ways to measure them, report on them to users, and periodically review its own performance. IRES Chapter IX Section C addresses these three activities in turn: quality measures and indicators, quality reports, and quality reviews.

Quality measures vs. quality indicators

There are essentially two ways to measure quality — using quality measures and quality indicators. Quantitative and qualitative measures and indicators built around the quality dimensions enable data producers to describe, measure, assess and report output quality so users can judge whether outputs are fit for their intended purposes; producers also use them to monitor data quality for continuous improvement [IRES, Ch. IX, para. 9.17, PDF p. 134, 2018].

Quality measures are items that directly measure a particular aspect of quality. For example, the time lag from the reference date to the day of publication of particular energy statistics, measured in days, weeks or months, is a direct quality measure of timeliness. In practice, many other quality measures can be difficult and costly to calculate; in these cases, quality indicators can be used to supplement or substitute for the desired quality measures [IRES, Ch. IX, para. 9.18, PDF p. 134, 2018].

Quality indicators usually consist of information that is a by-product of the statistical process. They do not measure quality directly but provide enough information to give insight into quality. For example, in the case of accuracy, measuring non-response bias directly is challenging because the characteristics of non-responders can be difficult and costly to ascertain — so response rates are often used as a proxy indicator of the possible extent of non-response bias. Other data sources can also serve as a quality indicator to validate or confront the data: commodity balances can be used to compare energy consumption data with energy supply figures (in the statistical difference flow) to flag potential problem areas [IRES, Ch. IX, para. 9.19, PDF p. 134, 2018]. This is the same statistical difference row defined in the energy balance framework — IRES names it here explicitly as a quality indicator that validates the supply side of a balance against the consumption side.

Selecting a practical subset of measures and indicators

Numerous quality indicators and measures have already been defined around specific dimensions and are in use by statistical organizations. Some take the form of descriptive statements or assertions — e.g., most of the “indicators of good practice” in the European Statistics Code of Practice, the “elements to be assured” in the NQAF Guidelines and NQAF Checklist, the IMF’s “elements of good practice” in the DQAF, and the “compliance criteria” in the Code of Good Statistical Practice for Latin America and the Caribbean. Others are quantitative statements or quantified measures calculated to specific formulas — e.g., the ESS Standard Quality and Performance Indicators. Countries are encouraged to develop or identify quality measures and indicators for describing, measuring, assessing, documenting and monitoring over time the quality of their energy statistics outputs, and to make them available to users. The Energy Statistics Compilers Manual presents several sets of indicators for consideration and selection [IRES, Ch. IX, para. 9.20, PDF p. 135, 2018] — recommendations tracker row IX/9.20.

The objective of quality measurement is to have a practical set (limited number) of quality measures and indicators to describe and monitor the quality of the data produced over time, giving users a useful summary of overall quality without overburdening respondents with demands for unrealistic amounts of metadata. It is not intended that all quality measures and indicators be addressed for all data. Instead, countries are encouraged to select practical sets of quality measures and indicators that are most relevant to their specific outputs and can be used to describe and monitor quality over time. The selected measures and indicators should cover each of the quality dimensions describing the outputs, have well-established compilation methodologies, and be easy to interpret by both internal and external users. Box 9.2 presents a sample of indicators and measures that energy statistics programmes can consider using [IRES, Ch. IX, para. 9.21, PDF p. 135, 2018] — recommendations tracker row IX/9.21.

Data compilers should also decide how frequently measures or indicators for different key outputs are produced. Some types — such as response rates — can be calculated and disseminated with each new estimate, in line with the frequency of production or publication of the data. Others could be produced once for longer periods and only refreshed for newly released data if major changes occurred [IRES, Ch. IX, para. 9.22, PDF p. 135, 2018].

Box 9.2 — Selected Indicators for Measuring the Quality of Energy Statistics

Reproduced in full [IRES, Ch. IX, Box 9.2, PDF p. 136, 2018]. The indicators listed represent only a sample of possible indicators that can be used for measuring quality; refer to the Energy Statistics Compilers Manual for more information [IRES, Ch. IX, Box 9.2 n.72, PDF p. 136, 2018].

Relevance

• Procedures are in place to identify the users of energy data and consult with them about their needs.
• Unmet user needs — gaps between key user needs and compiled energy statistics in terms of concepts, coverage and detail are identified and addressed.
• Requests for energy information are monitored and the capacity to respond is evaluated.
• User satisfaction surveys on the agency’s energy statistics outputs are regularly conducted and the results are analysed and acted upon.

Accuracy and reliability

• Energy source data are systematically assessed and validated.
• Sampling errors of estimates, e.g. standard errors, are measured, evaluated and systematically documented.
• Non-sampling errors, e.g. item non-response rates and unit non-response rates, are measured, evaluated and systematically documented.
• Coverage — the proportion of the population covered by the energy data collected is assessed.
• Imputation rates are reported.
• Information on the size and direction of revisions to energy data is provided and made known publicly.

Timeliness and punctuality

• A published release calendar announces in advance the dates that (key) energy statistics are to be released.
• The time lag between the end of the reference period and the date of the first release (or the release of final results) of energy data is monitored and reported.
• The possibility and usefulness of releasing preliminary data is regularly considered, while at the same time taking into account the data’s accuracy.
• The time lag between the date of the release or publication of the data and the date on which they were announced or promised to be released is monitored and reported.
• Any divergences from pre-announced release times for the energy data are published in advance; a new release time is then announced with explanations on the reasons for the delays.

Coherence and comparability

• Comparison and joint use of related energy data from different sources are made.
• Energy statistics are comparable over a reasonable period of time.
• Divergences from the relevant international statistical standards in concepts and measurement procedures used in the collection/compilation of energy statistics are monitored and explained.
• Energy statistics are internally coherent and consistent.

Accessibility and clarity

• Energy statistics and the corresponding metadata are presented in a form that facilitates proper interpretation and meaningful comparisons, and are archived.
• Modern information and communication technology (ICT) is mainly used for disseminating energy statistics; traditional hard copy and other services are provided, when appropriate, to ensure that users have appropriate access to the statistics they need.
• An information or user-support service, call centre or hotline is available for handling requests for energy data and for providing answers to questions about statistical results, metadata, etc.
• Access to energy microdata is allowed for research purposes, subject to specific rules and protocols on statistical confidentiality.
• The regular production of up-to-date quality reports and methodological documents (on energy concepts, definitions, scope, classifications, basis of recording, data sources (including the use of administrative data), compilation methods, statistical techniques, etc.) is part of the work programme, and the reports and documents are made known publicly.

[IRES, Ch. IX, Box 9.2, PDF p. 136, 2018]

Quality reports

For users of energy statistics to make informed use of the statistical information provided, they need to know whether the data are of sufficient quality. For some dimensions, such as timeliness, users can assess quality for themselves; for others, such as coherence and even relevance, this is less obvious. Accuracy in particular is a dimension users often cannot assess and must rely on the statistical agency for guidance. A quality report or similar documentation provides this guidance [IRES, Ch. IX, para. 9.23, PDF p. 135, 2018].

National practices for reporting on the quality of outputs vary, from short and concise to very detailed, depending on the intended users. General users are typically interested only in enough detail to know whether the data are reliable; producers want more detailed information to evaluate whether the output meets quality requirements and to identify strengths and areas needing improvement [IRES, Ch. IX, para. 9.24, PDF p. 135, 2018].

Quality information is often structured in a template format to promote comparability and consistency across statistical domains. Sometimes it is issued as a quality report separate from other metadata — a complement to them, not a replacement. Other times it is included within other metadata (alongside explanatory and technical notes and other detailed documentation). Some compilers call it a quality statement or quality declaration. Quality reports or documentation typically examine and describe quality according to the dimensions used by the agency to define fitness for purpose — relevance, accuracy, reliability, timeliness, punctuality, coherence, comparability, accessibility and clarity [IRES, Ch. IX, para. 9.25, PDF p. 137, 2018].

Two types of quality reports can be distinguished:

• The shorter “user-oriented” report, focused on output quality — often limited to brief descriptions of the output dimensions and just a few of the indicators listed in Box 9.2.
• The longer “producer-oriented” report — such as the comprehensive type ESS members are recommended to produce periodically (every five years or so, or after major changes) — which goes into greater detail on the dimensions, especially errors and other aspects affecting accuracy, and adds information on processes and other issues such as confidentiality, costs and response burden. For users such detail may be confusing and unnecessary, but for producers the comprehensive report serves as an internal self-assessment. Quality reporting therefore underpins quality assessment, which is in turn the starting point for quality improvements in statistical programmes. See the Energy Statistics Compilers Manual for more on quality reports and reporting practices [IRES, Ch. IX, para. 9.26, PDF p. 137, 2018].

The preparation and updating of quality reports depends on the survey frequency and the stability of the quality characteristics — balancing the need for recent information against the reporting burden. If necessary, the quality report should be updated as frequently as the survey is carried out; but if characteristics are stable, including quality indicators in the newest survey results may be enough to update the report. Another option is to provide a detailed quality report less frequently, with a shorter one after each survey covering only the updated characteristics (e.g., some accuracy-related indicators). Countries are encouraged to regularly issue quality reports as part of their metadata [IRES, Ch. IX, para. 9.27, PDF p. 137, 2018] — recommendations tracker row IX/9.27.

Quality reviews

Quality reviews can take the form of self-assessments, audits or peer reviews. They can be undertaken by internal or external experts, over timeframes ranging from days to months depending on the review’s scope, but the results are broadly the same — identification of improvement actions/opportunities in processes and products. It is recommended that some form of quality review of energy statistics programmes be undertaken periodically, for example every four to five years, or more frequently if significant methodological or other changes in data sources occur [IRES, Ch. IX, para. 9.28, PDF p. 137, 2018] — recommendations tracker row IX/9.28.

• Self-assessments are comprehensive, systematic, regular “do it yourself” reviews of an organization’s activities and results, referenced against a model/framework. Self-assessment checklists or questionnaires are typically developed for the systematic assessment of the quality of statistical production processes [IRES, Ch. IX, para. 9.29, PDF p. 137, 2018].
• A quality audit is a systematic, independent and documented process for obtaining quality evidence about a statistical process and evaluating it objectively to determine the extent to which quality policies, procedures and requirements are fulfilled. Unlike self-assessments, audits are always carried out by a third party (internal or external to the organization). Internal audits review the quality system in place (policies, standards, procedures and methods) and internal objectives, led by a team of internal quality auditors not in charge of the process or product under review. External audits are conducted by stakeholders or other interested parties, by an external and independent auditing organization, or by a suitably qualified expert [IRES, Ch. IX, para. 9.30, PDF pp. 137–138, 2018].
• Peer reviews are a type of external audit that assess a statistical process at a higher level rather than checking conformity with requirements item by item from a detailed checklist — they are often more informal and less structured than formal external audits. Peer reviews normally do not address specific aspects of data quality but focus on broader organizational and strategic questions. They are systematic examinations and assessments of the performance of one organization by another, aiming to help the reviewed organization comply with established standards and principles, improve its policy making and adopt best practices. Assessments are conducted on a non-adversarial basis, relying heavily on mutual trust between the organization and assessors and their shared confidence in the process [IRES, Ch. IX, para. 9.31, PDF p. 138, 2018].

Related

• Data Quality — the nine quality dimensions Box 9.2 is organized by, and that quality measures/indicators describe
• Quality Assurance Frameworks — NQAF Template section 4 (“Quality assessment and reporting”) is the framework-level counterpart to this page’s measurement/reporting/review material
• Energy Balance and Statistical Difference — the statistical difference is named at para 9.19 as a quality indicator that validates supply against consumption
• Metadata — quality reports are frequently issued as part of an agency’s broader metadata
• Energy Statistics Compilers Manual — forthcoming companion guidance on selecting quality indicators and quality reporting practices
• IRES Chapter IX — Data Quality Assurance and Metadata — the Chapter IX digest this page supports