Quality Assurance Frameworks

In a statistical office, systematic data quality management typically takes the form of a quality assurance framework. A national quality assurance framework is an overarching framework that can provide context for a country’s quality concerns, activities and initiatives, and explain the relationships between the various quality procedures and tools. Such frameworks have been developed and adopted, to varying degrees, by countries and international organizations. While all national statistical offices have in place some quality assurance approach and a number of quality assurance procedures — and most outline similar dimensions of quality (also called criteria, components or aspects in the literature) — not all countries yet have a formalized quality assurance framework [IRES, Ch. IX, para. 9.7, PDF p. 129, 2018].

The NQAF Template

In 2012, the United Nations Statistical Commission endorsed the generic National Quality Assurance Framework (NQAF) Template, developed by the Expert Group on National Quality Assurance Frameworks, to assist countries in formulating and operationalizing their national quality assurance frameworks or enhancing existing ones. The Expert Group’s work built upon, and helped raise greater awareness of, the various data quality management references and tools developed by international, regional, national and other organizations; these are posted on the UNSD NQAF website [IRES, Ch. IX, para. 9.8, PDF p. 129, 2018].

The NQAF Template drew heavily upon, and was designed to be in close alignment with, other main frameworks in use internationally: the European Statistics Code of Practice, the International Monetary Fund (IMF) Data Quality Assessment Framework (DQAF), the Statistics Canada Quality Assurance Framework, and the Code of Good Statistical Practice for Latin America and the Caribbean. Although these frameworks differ slightly from one another, they share common aspects and provide comprehensive, flexible structures for the qualitative assessment of a broad range of statistics, including energy statistics. They also help countries take stock of quality concerns, activities, requirements and initiatives, and foster standardization and systematization of quality practices and measurement within statistical offices and across countries. A mapping of each framework to the NQAF Template is available on the UNSD NQAF website [IRES, Ch. IX, para. 9.9, PDF p. 129, 2018].

The NQAF Template is presented in Box 9.1 below. Its five sections outline the elements a national quality assurance framework should include; IRES’s own discussion of quality assurance focuses mainly on Template sections 3 and 4, covering quality assurance objectives, considerations and practices, including measurement, reporting and evaluation. Additional information on other frameworks is available in the Energy Statistics Compilers Manual [IRES, Ch. IX, para. 9.10, PDF p. 129, 2018].

Box 9.1 — Template for a Generic National Quality Assurance Framework (NQAF)

Reproduced in full [IRES, Ch. IX, Box 9.1, PDF p. 130, 2018]:

1. Quality context
   • 1a. Circumstances and key issues driving the need for quality management
   • 1b. Benefits and challenges
   • 1c. Relationship to other statistical agency policies, strategies and frameworks and evolution over time
2. Quality concepts and frameworks
   • 2a. Concepts and terminology
   • 2b. Mapping to existing frameworks
3. Quality assurance guidelines
   • 3a. Managing the statistical system
     • [NQAF 1] Coordinating the national statistical system
     • [NQAF 2] Managing relationships with data users and data providers
     • [NQAF 3] Managing statistical standards
   • 3b. Managing the institutional environment
     • [NQAF 4] Assuring professional independence
     • [NQAF 5] Assuring impartiality and objectivity
     • [NQAF 6] Assuring transparency
     • [NQAF 7] Assuring statistical confidentiality and security
     • [NQAF 8] Assuring the quality commitment
     • [NQAF 9] Assuring adequacy of resources
   • 3c. Managing statistical processes
     • [NQAF 10] Assuring methodological soundness
     • [NQAF 11] Assuring cost-effectiveness
     • [NQAF 12] Assuring soundness of implementation
     • [NQAF 13] Managing the respondent burden
   • 3d. Managing statistical outputs
     • [NQAF 14] Assuring relevance
     • [NQAF 15] Assuring accuracy and reliability
     • [NQAF 16] Assuring timeliness and punctuality
     • [NQAF 17] Assuring accessibility and clarity
     • [NQAF 18] Assuring coherence and comparability
     • [NQAF 19] Managing metadata
4. Quality assessment and reporting
   • 4a. Measuring product and process quality — use of quality indicators, quality targets and process variables and descriptions
   • 4b. Communicating about quality — quality reports
   • 4c. Obtaining feedback from users
   • 4d. Conducting assessments; labelling and certification
   • 4e. Assuring continuous quality improvement
5. Quality and other management frameworks
   • 5a. Performance management
   • 5b. Resource management
   • 5c. Ethical standards
   • 5d. Continuous improvement
   • 5e. Governance

[IRES, Ch. IX, Box 9.1, PDF p. 130, 2018]

Section 3d’s five output-quality elements (NQAF 14–18) correspond directly to the nine quality dimensions defined later in the chapter (relevance; accuracy and reliability; timeliness and punctuality; accessibility and clarity; coherence and comparability); NQAF 19 (managing metadata) is treated separately in IRES Chapter IX Section D — see Metadata. Template section 4 (“Quality assessment and reporting” — measuring product/process quality via quality indicators, communicating about quality via quality reports, and conducting assessments) is the framework-level counterpart to the detailed Section C material on quality measurement and reporting.

Objectives, uses and benefits

The overall objective of quality assurance frameworks is to standardize and systematize quality practices and measurement within statistical offices and across countries. They serve as organizing frameworks that provide a single place to record and reference the full range of current quality concepts, policies and practices, and are forward-looking, taking into account future actions and activities. For energy statistics programmes specifically, a framework can allow the assessment of national practices against internationally (or regionally) accepted approaches for data quality management and measurement, and facilitate reviews of a country’s energy statistics programme by international organizations and other groups of data users [IRES, Ch. IX, para. 9.11, PDF p. 131, 2018].

The main benefits of having a quality assurance framework in place are that it: (a) makes the processes by which quality is assured more transparent and reinforces the image of the office as a credible provider of good-quality statistics; (b) creates a quality culture within the organization; (c) guides countries in strengthening their statistical systems by promoting self-assessments to identify quality problems; and (d) facilitates the exchange of ideas on quality management with other producers of statistics at the national, regional and international levels [IRES, Ch. IX, para. 9.12, PDF p. 131, 2018].

Recommendation (tracker row IX/9.13): energy statistics programmes without a quality assurance framework in place can avoid “reinventing the wheel” by reviewing the frameworks above and considering whether to directly follow one, or to structure their own in line with one or several of them, in a way that best fits their country’s practices and circumstances. Countries are encouraged to develop their own national quality assurance frameworks based on the above-mentioned approaches or other internationally recognized approaches, taking their specific national circumstances into account [IRES, Ch. IX, para. 9.13, PDF p. 131, 2018] — recommendations tracker row IX/9.13.

Related

• Data Quality — the underlying quality concepts (assurance vs. assessment, the three lines of quality, the nine dimensions) that frameworks like the NQAF Template exist to systematize
• Quality Measurement and Reporting — Section C’s detailed treatment of NQAF Template element 4a/4b (quality measures and indicators, quality reports, quality reviews)
• Metadata — Section D’s detailed treatment of NQAF 19 (managing metadata)
• Data Quality Assessment Framework — the IMF DQAF, one of the frameworks the NQAF Template was built in alignment with
• United Nations Statistical Commission — endorsed the NQAF Template in 2012
• IRES Chapter IX — Data Quality Assurance and Metadata — the Chapter IX digest this page supports