Approach to derive indicators for chains of statistical ...

Approach to derive indicators for chains of
statistical production processes

Arnout van Delden, Peter-Paul de Wolf , Li-Chun Zhang and Johan Fosen

When processes at national statistical institutes (NSI’s) become more complex and interconnected, it may be
useful to use process monitoring and improvement as an aid to control quality. We present a stepwise
approach to derive indicators for chains of statistical production processes at NSI’s

1. Introduction
As national statistical institutes (NSI’s) increasingly produce statistics based on administrative
data and multiple sources, processes become more complex. It becomes more difficult for
production staff to be sure that processes are “in control”, i.e. that certain quality demands are
either met or can be (positively) influenced. With increased complexity, it may be useful to use
process monitoring and improvement as an aid to control quality.
Applications of continuous quality improvement for end-to-end processes at NSIs are still limited
[3], although there are some first examples [11]. Most work so far concerns measuring or
controlling quality of single processing steps. We give practical guidelines to derive indicators for
management of chains of end-to-end processes applied to use of administrative data in business
statistics. In economic statistics interconnected (chains of) end-to-end processes and re-use of data
occurs: output of one statistics may be input to the next one.
This paper is organised as follows. Section 2 gives an overview of the guidelines. Section 3
describes the guidelines. Section 4 addresses objectives for chain management. Acase studies is
given in section 5. Section 6, gives some concluding remarks. The work is part of the EU-project
BLUE (print for) Enterprise and trade statistics (BLUE-ETS). For a full version of this paper,
including more references and explanation of terms we refer to [4].

1

2. Overview of the guidelines

In order to develop guidelines, we reviewed a number of well-known quality improvement
methods in manufacturing, such as Business Process Management, Total Quality management, Six
Sigma and Lean, see [8]. In addition, we made use of approaches developed for NSI’s [1],[5]and
[10]. We identified five key elements that are common to the quality frameworks that we reviewed
(rectangles in Fig. 2.1). These elements form the foundation for our practical guidelines.

3

Environment/policy

2,3 2 *
set threshold
specify Objectives
5
1,2,3 6,7
Strategy
Focus Area Indicators
*
4 4 visualise

localise Knowledge

Figure 2-1 General framework of [8]. The numbers correspond to the steps taken to derive indicators, see next section.
* refer to steps that are beyond scope.

The key element ‘objective’ defines what quality demands users want to achieve with respect to
the process under consideration. Each objective is expressed in terms of one or more ‘focus areas’
(see [10]), which denotes a combination of a quality aspect (property) of a selected object.
‘Indicators’ are used to monitor how well objectives are achieved. If indicator values are beyond
thresholds, the subsequent steps are needed to maintain or improve quality: the ‘strategy’. Often a
person’s ‘knowledge’ is needed to judge what action should be taken when the objectives are not
met and for structural improvements. The key elements are influenced by the environment (the
organization, policy, human interactions, etc.) (see dotted line in Fig 2.1). The actions at the
crossings of different key elements – ‘specify’ etc. are given attention in section 3.

Table 2-1 Steps for deriving indicators
Step Description
1 Design process flow map and identify transitional stages and major states
2 Relate the objective to focus areas
3 Account for the environment / policy
4 Localise which production parts affect the relevant focus areas
5 define strategies
6 compile a gross list of indicators
7 select relevant indicators

2

We developed seven steps to translate quality demands of users into indicators for end-to-end
processes building forth on Fig 2.1, see Table 2.1; the relation with framework is given in Fig 2.1.

3. Deriving indicators for process and chain management

3.1 Step 1: Design process flow map(s) and identify transitional stages and major states

Process and chain management requires an overview of the concerning processing steps. Process-
flow-maps can be used for that purpose. For each step, input, output, processing step and auxiliary
information is laid down. Examples can be found in section 5.

We have experienced that it is useful to divide end-to-end processes into a number of smaller
phases. After each phase, a well-defined (intermediate) data set is produced. Natural points of
those phases are points of data transfer from one organisational unit to the next one. In the
remainder of this paper we will refer to end points of processing phases by ‘transitional stages’
and to the data sets within them by ‘major states’. At the transitional stages you can assess
performance of the process, value added by processing steps and quality of the major states.
Examples of institutes using transitional stages with major states for quality management are the
Australian Bureau of Statistics [11], the Food and Agricultural Organisation [9] and Statistics
Netherlands [2].

3.2 Step 2: relate the objective to focus areas

In the second step, we first identify the objectives of the production process by which we mean the
quality demands of external and internal users. The objective operates as a ‘benchmark’ for
production staff to decide whether actions are needed for the process to become or remain in
control. Usually the demands as given by users are expressed in rather general terms, such as
‘timely and relevant data’. We then make the objective more explicit and specific by relating it to
to objects within the statistical process and to quality dimensions; the focus area.

We limit ourselves to all end products and data sets suitable for re-use. Obviously, objectives
could also be related to processing steps, especially in case of process management. In agreement
with [[10]], we think a production process is in control when it meets preset quality demands
while risks are acceptable. We consider quality demands more generally in terms of risks. In the
present paper, we will interchangeably use the terms ‘risk demand’ and ‘quality demand’ when
dealing with quality aspects concerning processing steps. Since different objectives may influence
each other, a ‘risk analysis’ is a good way to find relations between objectives.

3

3.3 Step 3: Account for the environment / policy

Statistical agencies may have strategic business goals (policy) that have an impact on the way
statistical processes are run and the kind of issues that are monitored. Well-known examples of
strategic business goals are a reduction of response burden or a reduction in size of production
staff. Strategic goals may be a driving factor to initiate monitoring systems.

These strategic goals of NSI’s may be translated into additional conditions with respect to the
quality demands given in step 2. Moreover, they could give rise to additional quality demands. For
example, at Statistics Netherlands, a monitoring system within the SBS production process was
designed that aimed to collect editing information in order to improve the SBS questionnaire. One
driving factor behind the monitoring system was to reduce time spent on manual editing [7].

3.4 Step 4: Localise which production parts affect the relevant focus areas

We need to derive for each focus area that is identified by steps 2 and 3, by which processing
steps, products and business rules it is affected. This is what we call localisation which production
parts affect the relevant focus areas. For instance, accuracy of a statistic based on a survey may be
affected by the selected weighting model and by the extent final data are free of measurement
errors. The number of measurement errors in the final data may in turn depend on the
effectiveness of a score function that identifies potentially influential errors. Whether the
weighting model results in unbiased estimates depends on the validity of the model assumptions.

In practice, several approaches may support this step. For instance drawing a network of (causal)
relations (see [1]); create a network of focus areas (see [10]) and pass through all processing steps
and major states and indicate whether they affect the quality demand under consideration.

3.5 Step 5: define strategies

In order to keep end-to-end process ‘in control’, we need to think about the kind of strategies that
may be taken. For each processing step, strategies at both operational and tactical level are
possible. The tactical level concerns the design stage of the methodology. For instance,
effectiveness of editing may be improved at tactical level by designing business rules to select just
the relevant records for clerical review. At operational level a stop-criterion is needed that shows
at which point editing another record is not needed, given the pre-set quality demands.

In addition to the tactical and operational level, we discern four types of strategies:
- a descriptive strategy aims just to report the “quality level”;

4

- a preventive strategy aims to prevent an error, failure or problem;
- a curative strategy aims to correct or counteract an error, failure or problem;
- a mitigatory strategy aims to moderate the effect of an error, failure or problem.

3.6 Step 6: compile a gross list of indicators

For each combination of focus area × processing step × strategy we try to define indicators.
Transitional stages are natural points to produce a report with indicators of the relevant processing
steps. But also at other places within processing steps, indicators may be computed (and judged).
In order to structure the indicators that might be used, we distinguish three types of indicators:

- Indicators describing data quality. These indicators measure the actual quality of ‘at that
time’. For instance the fraction of records in input data with item non-response.

- Indicators tracking processing changes. These indicators quantify to which extent a data set
has been changed due to a processing step.

- Indicators measuring effectiveness. These indicators quantify the effect of a processing
step on a statistical estimate (output), or on auxiliary variables.

A more detailed elaboration of indicator types is given in Chapter 4 of [4]. The data where upon
the indicators are computed on may be input sources, half-products, end products or auxiliary data.

3.7 Step 7: select relevant indicators

In step 7 a selection is made to arrive at the ‘final’ list of quality indicators to be implemented. As
a first step in reducing the gross list of indicators resulting from step 6, we think it is useful to use
the following quality demands of indicators for process and chain monitoring:

- indicators should be easy to interpret (‘clarity’);
- indicators should be based on available data in the production process (‘measurability’);
- indicators should relate to focus area(s) of interest (‘relevance’);
- indicators should support strategies to keep the process in control with respect to the focus

area(s) of interest (‘effectiveness’).

These properties are similar to the properties for responsive data collection designs mentioned in
[12]. When the remaining list might still (too) long two additional criteria may be used:

- make a choice among indicators measuring nearly the same (‘parsimonious’);
- keep only those indicators handling the largest risks (‘efficiency’)

5

Some criteria require experience in practice. This implies indicators used in practise are evaluated
regularly and adjustments are made when needed.

4. Deriving indicators for chain management

The steps of section 3 can be applied to end-to-end processes as well as to chains of processes.
One could start at the level of individual end-to-end processes and then apply a next cycle at chain
level. One could also start at chain level and then move onto the level of individual end-to-end
processes. Usually, one would go both ways in an iterative way.

We limit ourselves here by addressing what kind of quality demands is crucial to external and
internal users to assess whether the chain as a whole is ‘in control’. These quality demands are in
addition to those of the underlying end-to-end processes. For example, we think the following
topics are of importance for chain-related quality demands:

- Re-use of data: data may involve input data, half-products, end products or auxiliary data;
- Co-ordination, which refers to a uniform treatment across end-to-end processes of units,

concepts, variables and /or business rules.
- Dependencies in the chain: a problem in one part of the chain may affect other parts of the

chain. These effects may refer to accuracy or timeliness.

The above three topics are not exhaustive, others may be just as relevant when dealing with chain-
related quality demands. Moreover, the policy of an NSI may influence the chain as a whole as
well, e.g., effectiveness (use as little resources as possible) influences the whole chain.

5. Implementing the approach in practice

Below, we present a case study to explain how the method works in practice. Due to space
limitations we selected the main points. For more details and for an additional case study we refer
to [4].

5.1 A case study: Norwegian register-based employment

The register-based employment statistics is disseminated annually by Statistics Norway. In the
production process, several registers are linked and data from the different sources need to be
reconciled with each other. Each person in the target population is classified as employee, self-
employed or not employed. Employment status is derived from register information. We
distinguish three types of cases. If employment status can be deduced from register data (RD)
alone, without changes in or reconciliation of values coming from the RD, we classify them as a

6

consistent case. If some changes were needed, we classify them as a resolved case. The rest,
unresolved cases, refer to those persons whose employment status cannot be derived from RD
alone. Additional information on yearly employment total from Labor Force Survey (LFS) will be
used which leads to adjusted cases. For more explanation see [4]. Below we will try the stepwise
method for finding quality indicators.

Step 1. We identified three major states and two transitional stages in between. The first major
state refers to the input data from different administrative sources (RD), the second one contains
the linked micro data with persons as the unit, and the third state refers to the final statistical data
for tabulation. In the transitional stage from input to linked micro data, information that is
organized by the various “objects” [13] of the input data is transformed to a common unit, i.e.
person. At the transitional stage from linked micro data to statistical data, processing is
concentrated on the measurements [13], where the various input data measures are harmonized, re-
classified and adjusted. This yields the classification of persons according to the employment
status. See Figure below.

LFS

RD Linkage
change from
RD object to unit Harmonization / Adjustment
Re-classification

adjusted cases
RD consistent cases

resolved cases

Linked data Statistical register

Figure 5-1 Process flow map. Dotted lines illustrate end points of Transitional Stages (TS), i.e., natural stages in the
process-chain. Large cylinders illustrate Major States (MS), i.e., semi-final products. RD: register data.

Step 2. The objective is to produce timely and adequate employment figures. We relate this
objective to the accuracy of the objects input and output datasets and to the timeliness of the
delivered data sets in the chain. More specifically, we choose the following focus areas (FA):

FA1: Under-coverage of objects in input datasets, which may result in misclassification of persons
in employment at the second transitional stage, should be minimal

FA2: The input data sets should be received without consequential delay, and should have content
and format as agreed and anticipated.

7

FA3: After processing, as many as possible of the cases should be consistent, and as few as
possible should be adjusted.

Step 3. We apply the strategic objective of efficiency to the object output, and obtain:
FA4. Data processing should aim to create re-usable data for related statistical production.
As a result we identify the linked micro data as a major state. In the actual production process
today, it is not possible to locate this single dataset as it is simply not designed to allow for reuse
of processed data, except the final output. Due to space limitations we omitted step 4-7 for FA4. It
can be found in [4].

Step 4. The focus areas are influenced by the following parts of the process:

FA1: not directly influenced by other parts. In the long run it can and should be influenced by
chain management, i.e. by contacts with register owners to improve data quality.

FA2: influenced by operational chain management

FA3: influenced by all previous steps in the process. Adjustment of unresolved cases is especially
important, which depends on LFS yearly estimate and on method of adjustment .
Step 5. We derived strategies for each of the process parts as found in step 4. They strategies are first of all
directed at operational level. Off course it concerns design and chain level when larger changes to
methodology etc are needed.
FA1: (a) to uncover systematic errors, make comparisons to the previous input data sets and look
for large deviations, (b) to intervene in case of major changes in sources, receive early notification
from register owners in case of unusual events.

FA2: Receive early notification from input register owners if there are reasons for possible delay
in data delivery.

FA3: To monitor (a) the stability of output data over time, and the composition of consistent,
resolved and unresolved cases.

Step 6 and 7. We derived a number of quality indicators to support the strategies. Using the same
numbering as in step 5, we found:

FA1: (a) number of objects in each of the input datasets, compared to those from previous years;
(b) cross table of persons by input sources. For input sources with person as the object: number of
persons that can be found in both data sets.

FA2: how often is each register delayed, and how many of these were announced? In the cases of
delay: how many days?. How many extra resources is needed to cope with the delays?

8

FA3:number of persons in employment (employees, self-employed, not employed) against type fo
cases ('consistent measure' compared to 'resolved measure' and 'adjusted measure').

5.2 Experiences from the case study

We experienced that much of the work consisted of defining an appropriate process flow. The
choice of how to represent the process in a process flow and also how detailed the flow should be
depends on the use of the flow. Hence, we found it useful to go through the sequence 1-7 several
times, where the process flow representation gradually found its final form.

In addition, from applications we discovered that strategies and number of indicators can expand
substantially when passing through all steps. To avoid over-completeness, it is important to be
parsimonious. Furthermore, for smaller processes the approach can be onerous. In all cases select
only those steps relevant to your situation.

6. Concluding remarks

We presented guidelines to derive indicators for process and chain management at NSI’s, building
forth on a key elements derived from existing quality frameworks [8], focussing on chains of
processes within statistical institutes. The purpose is to give practical support in the design of
quality indicators. [3], [5] and [11] also worked on monitoring and improving quality at NSI’s.
[10] provides a generic framework that is also applicable to processes within NSI’s.

The main lines in [1], [3] and [10] match rather well with the current paper, although the order of
the steps varies and not all papers treat the same steps. The current approach addresses three key
points. The first key point is that process monitoring is not just ‘done for itself’, but needs to have
an objective. This objective is derived from quality demands of external and internal users. This
key point can also be found in [1], [3] and [10]. Secondly, it is very useful to define transitional
stages with major states as the natural places for quality indicators to be located. This offers a
structured way to split up large chains of end-to-end processes and yields natural locations for
quality reports. The third key point we make is that indicators are meant to support decisions on
strategies for quality management. This means that strategies need to be considered before quality
indicators are sought. General indicators meant for ‘potential’ (but yet unknown) users could be
considered as well.

In the current paper we focus on tailor made solutions, although first suggestions for
standardisation of indicator types for process monitoring are given in [4]. The guidelines we

9

present are only a start to provide practical tools to derive sets of indicators. Applying them to real
life processes at NSI’s should lead to improvements. Moreover, we hope that it is possible to find
some standard indicator types for a number of major states within (chains of) statistical processes.

References
[1] Aitken, A., Hörngren, J., Jones, N.(ed.), Lewis, D (ed.)., Zilhão, M.J. (2004). Handbook on

improving quality by analysis of process variables. European Commssion, 7 July 2004.
[2] Braaksma, B. Redesign of the chain of economic statistics in the Netherlands. Seminar on

registers in Statistics – methodology and quality, 21-23 May 2007, Helsinki.
[3] Bushery, J.M., and McGovern, P.D. (2010). Measuring process quality and performance at

Statistical Organisations. Paper presented at the European Conference on Quality in
Statistics 2010.
[4] Delden, A. van, de Wolf, P.P. en L.C. Zhang (2012). Guideliness for deriving indicators for
process and chain management. Deliverable 7.3 for BLUE ETS.
[5] Eurostat (2009) ESS Handbook for Quality Reports, 2009 edition. Methodologies and
Working papers, Eurostat, Luxembourg, ISSN 1977-0375.
[6] Eurostat (2005) Quality in statistics, Standard quality indicators. Seventh meeting, 23-24
May, Luxembourg.
[7] Giesen, I.D.N., Kruiskamp, P.H. (2011). Use of editing information to improve
questionnaires: first exploration in the Structural Business Statistics (in Dutch). Internal
report Statistics Netherlands.
[8] Griffoen, A.R., Delden, A. van, de Wolf, P.P. (2011). Quality frameworks applied to data,
processes and chains. Deliverable for WP7 of BLUE ETS.
[9] Mayo, R. (2005). Integrating data quality indicators into the FAO statistical system. Seventh
meeting, 23-24 May, Luxembourg, Luxembourg
[10] Nederpelt, P.W.M. (2012). Object-oriented quality and risk management (OQRM). A
practical, scalable and generic method to manage quality and risks. Alphen aan den
Rijn/New York: MicroData/Lulu Press. http://www.oqrm.org/English/
[11] Pink, B. (2010). Quality management of statistical processes using quality gates. Australian
Bureau of Statistics. www.abs.gov.au

10

[12] Schouten, B., Shlomo, N and Skinner, C. (2011). Indicators for monitoring and improving
representativeness of response. Journal of Official statistics 27(2): 231–253.

[13] (Zhang, 2012, “Topics for statistical theory for register-based statistics and data integration”,
Statistica Neerlandica, vol. 66, pp. 41-63)

11