Open access peer-reviewed chapter - ONLINE FIRST

Evaluation of the Long-Term Stability of Metrology Instruments

By Romain Coulon

Submitted: June 25th 2021Reviewed: September 17th 2021Published: November 2nd 2021

DOI: 10.5772/intechopen.100511

Downloaded: 53

Abstract

This chapter aims to emphasize the issue of the long-term stability of instruments used in metrology. This issue is a concern mentioned in the IEC/ISO17025:2017 standard and the JCGM100:2008 guide. Control charts are mentioned in these key documents as tools to assess whether a measurement process is under statistical control or not. Control charts (Shewhart charts, CUSUM chart, EWMA chart) are introduced and tested with simulated and real datasets from metrology instruments that operate at the ionizing department of the BIPM. The interest and the limits of such statistical analysis are discussed. They take their basis in a measurement model composed of Gaussian white noise. Although a measurement monitored over a relatively short period may be consistent with this model, it has been observed that the autocorrelation of the measurement data acquired over a long period limits the relevance of control charts. In this case, time series analysis seems more appropriate than conventional control charts. As an illustration, an optimal Bayesian smoother is introduced to demonstrate how to deconvolve the low-frequency random noise and refine the evaluation of uncertainty according to the measurement model for long-term measurement.

Keywords

  • metrology
  • control charts
  • time series
  • statistical control
  • long-term stability

1. Introduction

Long-term reproducibility in experimental sciences is a critical metrological topic recently discussed in [1]. In laboratories, the quality system is the frame in which such an issue is controlled. It is notably required in the ISO/IEC 17025 [2] that laboratories must ensure the validity of their results. For this, monitoring of the state of operating of a measurement system must be set up by periodic checks and statistical analysis of the obtained process control value.

Techniques to assess the reproducibility of a process is a matter of concern since the industrial era. An important step has been taken in 1924 with the invention of control charts by Walter Andrew Shewhart (Bell Telephone). Control charts are efficient tools to control if a process is under statistical control and is now wildly used in many domains and recognized by the Project Management Institute as one of the seven basic tools of quality [3]. Moreover, control charts are explicitly proposed in ISO/IEC 17025 [2] and the JCGM100:2008 Guide [4] as tools to implement to analyze periodic checks using control standards.

The main advantages of control charts are their simplicity of implementation and their generic use, independent of the domains. A control chart displays a set of points and control limits. This representation makes it easy to check the number of points exceeding the control limits. Therefore, control charts can be analyzed by anybody without any specific skills in statistics.

Four typical control charts are introduced in this document: the original Shewhart charts (x-chart and R-chart), the CUSUM chart, and the EWMA chart. The manners that they are introduced below should be considered as examples that can slightly vary from one reference to another [5, 6]. These techniques are applied below to simulated data and real data from metrology instruments used in ionizing radiation metrology. Then, the interest of time series analysis is highlighted for the monitoring of measures over a long period using the same datasets.

Advertisement

2. Introduction to control charts

2.1 The x-chart

Periodical control using a check standard consists of periodically reproduce the measurement xiat the time tiand forms the measurement set x1x2xixNwhere xNis the last recorded point. In the x-chart, a central line is drawn by the moving empirical average x¯Nover a period of 31 samples such as

x¯N=i=1NxiN,forN31x¯N=i=N31Nxi31,forN>31.E1

The control limits (LCL and UCL) are given by the multiple of the standard deviations:

sxN=i=1Nxix¯N2N1,forN31,sxN=i=N31Nxix¯N230,forN>31,E2

such as

LCL=x¯NksxNUCL=x¯N+ksxN,E3

where the multiplication factor kis set equal to 2 for the first control level and 3 for the second control level.

Failure criteria are defined according to a white Gaussian model that will be introduced later. It is important to note that the relevance of the analysis is low when the number of measures N<10, relevant for 10N31,and good for N31. Some criteria can be founded in the ISO 7870 standard [5]. As an example, the following rules are proposed where a measurement process is not considered under statistical control when:

  • more than 0.35% of the points are above or below the control limits with the multiplication factor k=3,

  • more than 20% of the points are above or below the control limits with the multiplication factor k=2,

  • more than 9 points are in a row one side of x¯N,

  • more than 6 points are in a row increasing or decreasing,

  • more than 14 points are in a row alternating between increase and decrease.

2.2 The R-chart

In the R-chart, the plotted points Riare the derivatives of the measurements:

Ri=xixi1foralli2N.E4

The central line and two levels of control limits are obtained in the same manner as for the x-chart by calculating the moving empirical average and standards deviations of the values Ri. Also, the same failure criteria can be applied.

2.3 The CUSUM chart

The cumulative of deviations Ciis obtained by the following recursive formula:

Ci=Ci1+xix¯NE5

In this chart, no central line is displayed but two control limits Viare defined by the so-called V-mask:

Vi=CN±ksxid+NiE6

where k=0.1and d=147.22.

Any points out of the V-mask mean that a significant drift of the process control value is detected.

2.4 The EWMA chart

The EWMA chart is obtained by plotting the exponentially smoothed data points ziby using the following recursive formula:

zi=λxi+1λzi1E7

And where λis a smoothing parameter.

The control limits CLare defined by

CLi=x¯i±3sxiλ2λ11λ2iE8

With the smoothing parameter λ=2/N.

Any point out of these control limits means that a significant drift of the process control value is detected.

It must be noted that these two families of control charts, the Shewhart charts on the one hand and the CUSUM and EWMA charts on the other hand, are complementary. Indeed, the Shewhart charts aim to detect only large shifts above 1.5sxi(detection of outliers), whereas the CUSUM and EWMA charts aim to detect only small shifts below 1.5sxiand more generally drifts in the process. In other words, Shewhart charts perform an analysis in the high-frequency domain, whereas the CUSUM and EWMA charts are more suitable to analyze in the low-frequency domain.

Advertisement

3. Test of control charts with simulated data

3.1 The Gaussian white noise model

N=100normally distributed data points xiare simulated using the random generator from the numpy.py library.

The model of the simulated process is the following:

xi=μi+εiεiN0σi2E9

with μi: the expectation parameter at the time ti, N: the normal distribution, and σi2: the variance parameter of the measurement noise εiat the time ti. It is referred below as a Gaussian white noise model.

The reference dataset is defined by setting, for all i1N, the parameters μi=0and σi2=1. The control charts are presented below, and the tests conclude—as expected—that the simulated process is under statistical control (Figures 14).

Figure 1.

x-chart of the process simulated form the Gaussian white noise model.

Figure 2.

R-chart of the process simulated form the Gaussian white noise model.

Figure 3.

CUSUM chart of the process simulated form the Gaussian white noise model.

Figure 4.

EWMA chart of the process simulated form the Gaussian white noise model.

3.2 The Gaussian white noise model with a deterministic drift

Now, a slight positive drift is considered in the following model referred to as a deterministic drift model in a Gaussian white noise such as the expectation parameter takes values such as:

μi=0,i1,N2μi=μi1+0.3,iN2+1,N.E10

This leads to a detection of this trend in the x-chart because more than nine points are in a row on one side of the central line. Also, the CUSUM chart and the EWMA chart detect the trend with some points out of their control limits (Figures 58).

Figure 5.

x-chart of the process simulated form the Gaussian white noise model with a slight drift.

Figure 6.

R-chart of the process simulated form the Gaussian white noise model with a slight drift.

Figure 7.

CUSUM chart of the process simulated form the Gaussian white noise model with a slight drift.

Figure 8.

EWMA chart of the process simulated form the Gaussian white noise model with a slight drift.

3.3 The autocorrelated model

The third model studied here is the autocorrelated model (or the colored noise model) where a low-frequency Brownian noise is introduced in the following autocorrelated model:

xi=xi1+ẋi+εiẋi=ẋi1+δiE11

where the measurement noise is εiN0σi2(idem as in Eq. (9)) and a process noise δiN0τi2is added with a variance parameter τi2. In the example below, τi=0.3while σiis still equal to 1.

Again, this type of deviation from the Gaussian white noise model is detected by the value chart with more than nine points in a row on one side of the central line and by CUSUM and EWMA charts (Figures 912).

Figure 9.

x-chart of the process simulated form the autocorrelated model.

Figure 10.

Range chart of the process simulated form the autocorrelated model.

Figure 11.

CUSUM chart of the process simulated form the autocorrelated model.

Figure 12.

EWMA chart of the process simulated form the autocorrelated model.

These three examples underline the fact that control charts are—by definition—build to verify whether a dataset fits with a Gaussian white noise model or not. Any deviation from this model is efficiently revealed by control charts. An operator could therefore conclude by the fact that the process is not under statistical control. This conclusion is good in the case of deterministic trends that are that a metrology system seeks to avoid maintaining the references that it produces. However, it has been demonstrated that the autocorrelated noise could also lead to an alert by control charts. On the contrary to the deterministic drift, the latter is a natural random process that appears in every system, which is observed on enough long-term periods. This particular limit of control charts was already pointed out by many authors [7, 8, 9, 10]. This issue is an emphasis in the real cases presented below.

Advertisement

4. Test of control charts with real data

Two data are here considered. The first one comes from the periodic control of a recent metrology system that operates at the BIPM for radionuclide metrology and is called the ESIR (extended SIR) [11, 12]. The second example comes from the periodic control of a metrology service, the SIR, which operates at the BIPM for radionuclide metrology since 1976 [13].

4.1 Test with 1-year period data

The ESIR is under development, and the periodical control using check standards has been set up for 1 year. Figures 1316 show the analysis of the measured data by control charts. The monitored value is a comparison indicator obtained by using a check source of 14C. The analysis from the four-control charts leads to the conclusion that the process is under statistical control. In other words, the measurement model complies with a Gaussian white noise model. At least, it is true for this short period of observation. The activity estimation is of the 14C check source is stable and equal to 24.3373±0.0184A.U. There is no unit associated with the measurand because the aim of the ESIR is only to deliver a measure proportional to the activity but is very reproducible during the time. The relative standard uncertainty associated with this measurement is about 0.08%.

Figure 13.

x-chart of the periodic control of the ESIR metrology system.

Figure 14.

R-chart of the periodic control of the ESIR metrology system.

Figure 15.

CUSUM chart of the periodic control of the ESIR metrology system.

Figure 16.

EWMA chart of the periodic control of the ESIR metrology system.

4.2 Test with 20-year period data

The SIR produces international references for decades. The 226Ra source is used as a control standard to monitor the reproducibility of the measured current. In this case, the control charts are applied to a 20-year period of data acquisition. They have detected that the process is not under statistical control. However, these low-frequency fluctuations could certainly be a random colored noise and not a deterministic drift. This phenomenon appears when measurement values are observed over a very long period such as here. The only conclusion that can be drawn from control charts is that the data do not fit with the Gaussian white noise model and that a type A evaluation of the uncertainty associated with the measurement values (from t1 to tN) cannot be applied. Nevertheless, it is no need to refine the measurement model for the SIR measurement because the quantity delivered by the system is a ratio of a punctual current measurement (at a time ti) from which this low-frequency component revealed here is suppressed by the method (Figure 1720).

Figure 17.

x-chart of the periodic control of the SIR metrology system.

Figure 18.

R-chart of the periodic control of the SIR metrology system.

Figure 19.

CUSUM chart of the periodic control of the SIR metrology system.

Figure 20.

EWMA chart of the periodic control of the SIR metrology system.

We have underlined, through these simulated and real datasets, the limitation of control charts to deal with the evaluation of the long-term stability of metrology instruments. In this case, time series analysis could be used as more appropriate tool to address this objective.

An example of time series analysis is introduced below.

Advertisement

5. Introduction to time series analysis applied to a long-term measurement model

It has been seen that the autocorrelated model introduced in Eq. (10) is appropriate to represent measurement data when observed over a long-term period. Eq. (10) is here rewritten in a state-space model where the measurement is decomposed by the following:

  • the fluctuation of the expectation value x1xixNwith the low-frequency random noise at measurement times t1titN,

  • and the measurements gathered in the vector y1yiyNwhere the Gaussian white noise is finally added to x1xixN.

The model becomes,

xi=xi1+ẋi1ẋi=ẋi1+δiyi=xi+εiE12

The drift velocity ẋi=xixi1is introduced such that the dynamic Gaussian noise δiN0τ2describes a random walk parametrized by a variance τ2. And the measurement noise εiN0σ2is Gaussian white noise with a variance σ2.

The idea behind time series analysis is to deconvolve the low-frequency fluctuation of the expectation values x1xixNfrom the observations y1yiyN.

At first, the measurement variance σ2is evaluated by using the median of all the punctual type A evaluation of the standard deviation syiwhich are (by definition) isolated from the dynamic component.

σ2=medians2yiE13

with,

s2yi=j=1MyijM1l=1Myil2M1.E14

yN+11yN+1jyN+1Mis the set of measurements obtained at the time tiduring a period that allowing type A evaluation according to the Gaussian white noise model. Here, control charts can be used to verify this hypothesis.

Eq. (12) is transformed in the following matrix form:

Xi=FXi1+Wiyi=HXi+εiE15

with the “state vector” Xi=xiẋi, the “transition matrix” F=1101, the” dynamic noise matrix” Wk=0δi,and the “observation matrix” H=10.

5.1 The Kalman filter

Given the Gaussianity of the noises (both dynamic noise δiand measurement noise εi) of this linear model (in the sense of time series), the Kalman filter (KF) provides an optimal estimation of the states Xigiven the current and past measurements y1:i. The optimality of KF to solve Eq. (15) is established through both least-square interpretation and the probabilistic Bayesian analysis [14, 15]. Hence, KF is a recursive Bayesian filter where the states Xiand the measurements yiare Markov processes implying

pXiX1:i1=pXiXi1pyiX1:i=pyiXi.E16

KF evaluates, at each time step i, the parameters of the Gaussian posterior distribution pXiy1:iin two steps:

  • The first step consists of predicting the state Xigiven past measurement data y1:i1. The posterior distribution pXi1y1:i1at ti1is used to build a prior distribution at the time tiaccording to the linear model (see Eq. (15)) from ti1to tisuch as

pXiyi1=pXiXi1pXi1yi1dXi1.E17

  • The second step aims to update the posterior distribution at tigiven the predicted prior pXiy1:i1, and the likelihood pyiXiof the measurement yigiven the estimate Xi.

pXiyipXiyi1pyiXi.E18

The posterior state estimate X̂iiand the posterior of the covariance Piiof the posterior distribution pXiy1:iare evaluated through the Kalman filtering process described below.

From i=1to i=N, KF predicts the state Xiand the measurement yiat the time step igiven the previous state at i1:

X̂ii1=FX̂i1i1ŷii1=HX̂ii1.E19

With the state and measurement covariances, Piand Sisuch that:

Pii1=FPi1i1FT+τ2Si=HPii1HT+σ2.E20

Then, the pre-fit residual (innovation) eiof the estimation and the Kalman gain Kiare calculated by

ei=yiŷii1Ki=Pii1HTSi1.E21

Finally, the updated state estimation is

X̂ii=X̂ii1+KieiPii=IKiHPii1,E22

with I: the identity matrix.

5.2 The Rauch-Tung-Striebel smoother

The KF estimates the states X̂i1:igiven present and past observations from t1to ti. To refine the estimation, a KF backward pass (smoothing pass) processes the filtered estimates data from tNto t1. This procedure is called the Rauch-Tung-Striebel smoother (RST) [16, 17]. Symmetrically to the forward KF optimal filter, the optimal smoothing also consists in:

  • a prediction step where the state Xi+1is estimated given measurement data y1:iand the forecasted state backward transition according to the linear model from ti+1to ti.

pXi+1y1:i=pXi+1XipXiy1:idXiE23

  • The second step aims to update the posterior distribution at tigiven all the data pXiy1:N.

pXiy1:N=pXiy1:ipXi+1XipXi+1y1:NpXi+1y1:idXi+1.E24

The RTS smoother calculates the Maximum A Posteriori (MAP) estimate X̂i1:Nof the posterior probability pXiy1:Ngiven all the observations (past, present, and future) from t1to tN.

From i=Nto i=1, the residual eibetween the filtered state x̂i+1iand the prediction x̂i+11:Nis used to calculate a new Kalman gain Gisuch as

ei=X̂i+11:NX̂i+1iGi=Pii+1FTPi+1i1.E25

The optimal state estimation is finally given by

X̂i1:N=X̂iiGieiPi1:N=Pii+GiPi+11:NPi+1iGiT.E26

5.3 The auto-tuning of the RTS smoother

As the measurement noise variance σ2is evaluated, by definition, through measurement repeated during a period free from long-term fluctuation (Eq. (17)), the variance of the dynamic noise τ2is the remaining unknown parameter of the model. The auto-tuning procedure proposed here to evaluate τstarts with a loss function Ldefined by

Lτ=i=2Nei2+x̂i1:Nx̂i11:N2.E27

The value τ̂minimizing Lτaims to minimize the residual between the measurements yiand the estimations of the measurements ŷii1as well as the distance between two consecutive filtered estimations x̂i1:Nand x̂i11:N. On the one hand, if this last smoothing constraint is not considered, τwill diverge toward high values describing a totally unstable system. On the other hand, if the prediction error of the model is not considered, τwill be canceled by describing a stable system with constant x̂iivalues. So, the loss function of Eq. (27) permits to produce a model of measurement of a controlled process containing a low-frequency dynamic noise. The variance of the low-frequency dynamic noise is determined through an iterative procedure minimizing Lτsuch that,

τ̂=argminτLτ.E28

The auto-tune RTS smoother provides filtered values x̂i1:Nwhere the measurement noise, εi,is removed given the knowledge of the variances of the measurement noise σ2(see Eqs. (13) and (14)) and the dynamic noise τ̂2. So, the global hidden uncertainty component from the long-term process random noise can be evaluated from the deconvolved data x̂i1:Nsuch that,

sx̂i1:N=N11i=1Nx̂i1:NN1j=1Nx̂i1:N2.E29

Thanks to the intrinsic unfolding of the measurement noise (Gaussian white) and the process noise (Gaussian red) defined in the dynamic model (Eq. 12), the standard uncertainty uxN+1of a new measurement at tN+1can be obtained by the quadratic summation of the type A uncertainty evaluation sxiN+1obtained under repeated conditions yN+11yN+1jyN+1Mand the type B uncertainty evaluation evaluated using all historical data y1yiyNthrough the above-described auto-tunned RTS smoother.

Therefore, the standard uncertainty u2xN+1of the next measurement at tN+1should be inflated considering the impact of the dynamic noise such that,

u2xT+1=s2xiT+1+PN+11:N1E30

With,

PN+11:N=FPN1:NFT+τ2E31

This auto-tuning of a RTS smoother applied to measurement data recorded over a long-term period fully address the recommendation of the JCGM100:2008 guide [4]:

Because the mathematical model may be incomplete, all relevant quantities should be varied to the fullest practicable extent so that the evaluation of uncertainty can be based as much as possible on observed data. Whenever feasible, the use of empirical models of the measurement founded on long-term quantitative data, and the use of check standards and control charts that can indicate if a measurement is under statistical control, should be part of the effort to obtain reliable evaluations of uncertainty. The mathematical model should always be revised when the observed data, including the result of independent determinations of the same measurand, demonstrate that the model is incomplete. A well-designed experiment can greatly facilitate reliable evaluations of uncertainty and is an important part of the art of measurement.

Advertisement

6. Application of time series analysis on simulated and real measurement data

The time series analysis through the auto-tuned RST smother is presented in Figure 21 for the simulated dataset from Section 3.3 obtained with the autocorrelated model. Black points are the measurement points yiwith error bars representing the standard deviation syi. Green points are the estimations of the low-frequency fluctuations (red noise) of the measurement x̂iwith error bars representing the standard deviation sx̂i. The red line is the global expectation value where the thickness of the line represents +/− 1 standard deviation.

Figure 21.

Time series analysis of the measurement data simulated form the autocorrelated model.

It can be seen the good deconvolution of the low-frequency random noise shown in green points. This leads to a standard uncertainty from this autocorrelated noise, sx̂i1:N=1.31.This value is significant when compared to type A evaluation of individual measurement points σ2=1.74.

Also, the time series analysis through the auto-tuned RST smother is presented in Figure 22 when applied to the periodic control of the SIR metrology system. Again, black points are the measurement points yiwith error bars representing the standard deviation syi. Green points are the estimations of the low-frequency fluctuations (red noise) of the measurement x̂iwith error bars representing the standard deviation sx̂i. The red line is the global expectation value where the thickness of the line represents +/− 1 standard deviation.

Figure 22.

Time series analysis of the periodic control of the SIR metrology system.

The analysis using control charts (see Section 4.2) has led to conclude that the system is not under statistical control although autocorrelated noise is expected for such a long-term period of monitoring. The optimal Bayesian smoother permits to properly deconvolve this component (see green points) and to evaluate the standard uncertainty associated with it: sx̂i1:N=0.0183. This value is significant when compared to type A evaluation of individual measurement points σ2=0.025.

Advertisement

7. Conclusion

It has been emphasized through simulated and experimental data that control charts (Shewhart, CUSUM, EWMA charts) are convenient tools to assess whether a measurement system is under statistical control or not. They test the compliance of the measured dataset with a Gaussian white noise model. It can notably be used to determine the possibility to apply a type A evaluation of uncertainty on the measurement data.

However, care should be taken when monitoring over long periods of time. A low-frequency random noise will inevitably appear. Despite its stochastic nature, the latter will induce detection by control charts. In this case, it is not appropriate to conclude by a loss of control of the system but rather to re-evaluate the measurement model to include this long-term component. To that end, time series analysis could be implemented to better deal with this particular case of measurement data.

An auto-tuned Rauch-Tung-Striebel smoother, based on the optimal Bayesian filter called Kalman filter, is introduced to illustrate how time series analysis can help to deconvolve the components of the long-term measurement model. The low-frequency random fluctuation (aka red noise) can be estimated and used to evaluate the standard uncertainty component it induces on the measurement.

DOWNLOAD FOR FREE

chapter PDF

© 2021 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution 3.0 License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite and reference

Link to this chapter Copy to clipboard

Cite this chapter Copy to clipboard

Romain Coulon (November 2nd 2021). Evaluation of the Long-Term Stability of Metrology Instruments [Online First], IntechOpen, DOI: 10.5772/intechopen.100511. Available from:

chapter statistics

53total chapter downloads

More statistics for editors and authors

Login to your personal dashboard for more detailed statistics on your publications.

Access personal reporting

We are IntechOpen, the world's leading publisher of Open Access books. Built by scientists, for scientists. Our readership spans scientists, professors, researchers, librarians, and students, as well as business professionals. We share our knowledge and peer-reveiwed research papers with libraries, scientific and engineering societies, and also work with corporate R&D departments and government entities.

More About Us