Due to the recent rapid growth of advanced sensing and production technologies, the monitoring and diagnosis of multivariate process operating performance have drawn increasing interest in process industries. The multivariate statistical process control (MSPC) chart is one of the most commonly used tools for detecting process faults. However, an out-of-control MSPC signal only indicates that process faults have intruded the underlying process. Identifying which of the monitored quality variables is responsible for the MSPC signal is fairly difficult. Pinpointing the responsible variable is vital for process improvement because it effectively determines the root causes of the process faults. Accordingly, this identification has become an important research issue concerning recent multivariate process applications. In contrast with the traditional single classifier approach, the present study proposes hybrid modeling schemes to address problems that involve a large number of quality variables in a multivariate normal process. The proposed scheme includes multivariate adaptive regression splines (MARS), logistic regression (LR), and artificial neural network (ANN). By applying MARS and LR techniques, we may obtain fewer but more significant quality variables, which can serve as inputs to the ANN classifier. The performance of our proposed approaches was evaluated by conducting a series of experiments.
A multivariate process monitors two or more quality variables. When a signal is triggered by the multivariate statistical process control (MSPC) chart, process personnel are typically only aware that the underlying process is in an unstable state. Identifying which of the monitored quality characteristics (or variables) is responsible for this MSPC signal is challenging. Accordingly, effective determination of the source of process faults becomes an important and challenging issue in MSPC applications, because these sources are associated with specific assignable causes that adversely affect the process.
Typically, a literature review has shown that there are different kinds of approaches to investigate on source identification of faults in a multivariate process. The first type of approach uses various graphical techniques, such as polygonal charts [
The second type of approach uses the statistical decomposition techniques to interpret the contributors to an MSPC signal. Mason et al. [
The third type of approach employs the machine learning (ML) mechanisms, such as artificial neural networks (ANN) and support vector machine (SVM), to identify the quality variables which are responsible for the MSPC signal. A comparative study has been conducted by the studies [
The literature review has shown that most of the existing studies are concerned with the determination of which variable or group of variables has caused the signal through single step modeling. However, there is a difficulty that may not have been addressed yet. When the number of quality characteristics is large, the existing decomposition methods and/or machine learning methods may lack the capability to handle such a situation. In addition, because process faults are typically attributed to mean shifts and the multivariate normal process is one of the most widely used applications, the present study is motivated by addressing mean shift faults for a multivariate normal process with a large number of quality variables. A review of relevant literature also indicates that the application of ANN for process fault determination is promising; however, it suffers from the requirement of a large number of controlling parameters and the risk of model overfitting [
The rest of this paper is organized as follows. Section
The structure of the process model is addressed. The proposed hybrid schemes are also described in this section.
This study considers the situation of process mean shifts and assumes that the multivariate process is initially in a normal state and the sample observations are derived from a
The purpose of performing logistic regression modeling in stage I was to identify important influencing variables and refine the entire set of input variables. The structure of the logistic regression model can be briefly described as follows. Let
Before screening significant independent variables, we performed the collinearity diagnosis procedure to exclude variables that exhibited high collinearity. After this diagnosis, the remaining variables served as independent variables for logistic regression modeling and testing. The Wald forward method was applied to identify independent variables with significant influence on an abnormal state probability. These significant independent variables and the dependent variable were then substituted into the ANN to construct a two-stage model.
The superior performance of the MARS has been reported in many applications [
The ANN has been widely used in many SPC applications [
Figure
Structure of ANN model.
To evaluate the performance of the proposed approach, a series of simulations were conducted. Without loss of generality, this study assumed that each quality characteristic was initially sampled from a normal distribution with zero mean and one standard deviation. In addition, we assumed that twenty quality characteristics were monitored simultaneously (i.e.,
Because we considered 20 quality characteristics for the multivariate normal process, there are
According to the suggestions of the study [
For the hybrid LR-ANN model, this study calculated the variance inflation factor (VIF) to examine the presence of collinearity, used a 0.05 significance level, and employed logistic regression analysis to select important influencing variables in the initial stage. Values of VIFs greater than 10 were considered large enough to suspect serious multicollinearity [
Collinearity diagnosis for LR models.
Variables |
|
|
|
|||
---|---|---|---|---|---|---|
|
|
|
|
|
|
|
|
1.11 | 1.23 | 1.65 | 1.56 | 3.66 | 2.26 |
|
1.09 | 1.09 | 1.95 | 1.95 | 9.85 | 9.85 |
|
1.21 | 1.54 | 1.80 | 1.97 | 3.98 | 2.89 |
|
1.07 | 1.07 | 1.91 | 1.91 | 9.54 | 9.54 |
|
1.21 | 1.55 | 1.81 | 2.00 | 4.03 | 2.91 |
|
1.07 | 1.07 | 1.88 | 1.88 | 9.31 | 9.31 |
|
1.13 | 1.28 | 1.69 | 1.63 | 3.78 | 2.33 |
|
1.11 | 1.11 | 1.98 | 1.98 | 9.82 | 9.82 |
|
1.07 | 1.07 | 1.89 | 1.88 | 9.21 | 9.21 |
|
1.05 | 1.05 | 1.81 | 1.80 | 9.21 | 9.19 |
|
1.07 | 1.07 | 1.90 | 1.90 | 9.43 | 9.41 |
|
1.07 | 1.07 | 1.89 | 1.89 | 9.60 | 9.59 |
|
1.07 | 1.07 | 1.91 | 1.90 | 9.84 | 9.83 |
|
1.06 | 1.06 | 1.83 | 1.82 | 9.13 | 9.11 |
|
1.08 | 1.08 | 1.92 | 1.92 | 9.62 | 9.60 |
|
1.07 | 1.06 | 1.93 | 1.92 | 9.95 | 9.93 |
|
1.07 | 1.07 | 1.84 | 1.83 | 8.93 | 8.92 |
|
1.09 | 1.09 | 1.96 | 1.96 | 9.78 | 9.78 |
|
1.09 | 1.09 | 1.94 | 1.93 | 9.69 | 9.68 |
|
1.09 | 1.09 | 1.97 | 1.96 | 9.64 | 9.63 |
Significant variables selected by LR analysis.
Correlation |
Shift value |
Significant explanatory variables |
---|---|---|
0.1 | 0.5 |
|
1.0 |
|
|
| ||
0.5 | 0.5 |
|
1.0 |
|
|
| ||
0.9 | 0.5 |
|
1.0 |
|
For the hybrid MARS-ANN model, we obtained the selection results of the variables after performing the MARS procedure. Tables
Basis functions and important explanatory variables for the MARS model with
Function | Std. dev. | Cost of omission | Number of BF | Variable | Relative importance (%) |
---|---|---|---|---|---|
1 | 0.501 | 1.070 | 1 |
|
100.000 |
2 | 0.500 | 1.065 | 3 |
|
98.993 |
3 | 0.339 | 0.963 | 1 |
|
70.614 |
4 | 0.332 | 0.957 | 1 |
|
68.366 |
5 | 0.131 | 0.873 | 1 |
|
26.827 |
6 | 0.104 | 0.867 | 1 |
|
20.879 |
7 | 0.099 | 0.866 | 1 |
|
19.621 |
8 | 0.093 | 0.865 | 1 |
|
18.071 |
Basis functions and important explanatory variables for the MARS model with
Function | Std. dev. | Cost of omission | Number of BF | Variable | Relative importance (%) |
---|---|---|---|---|---|
1 | 0.517 | 0.372 | 3 |
|
100.000 |
2 | 0.519 | 0.370 | 2 |
|
99.356 |
3 | 0.395 | 0.328 | 2 |
|
85.740 |
4 | 0.369 | 0.310 | 1 |
|
79.157 |
5 | 0.066 | 0.211 | 1 |
|
15.440 |
Basis functions and important explanatory variables for the MARS model with
Function | Std. dev. | Cost of omission | Number of BF | Variable | Relative importance (%) |
---|---|---|---|---|---|
1 | 0.596 | 0.871 | 2 |
|
100.000 |
2 | 0.582 | 0.864 | 2 |
|
98.281 |
3 | 0.380 | 0.764 | 2 |
|
68.774 |
4 | 0.369 | 0.763 | 1 |
|
68.220 |
5 | 0.245 | 0.702 | 1 |
|
40.739 |
6 | 0.234 | 0.700 | 1 |
|
39.271 |
7 | 0.203 | 0.692 | 2 |
|
33.836 |
8 | 0.195 | 0.691 | 1 |
|
32.728 |
9 | 0.188 | 0.688 | 1 |
|
30.632 |
10 | 0.186 | 0.688 | 1 |
|
30.530 |
Basis functions and important explanatory variables for the MARS model with
Function | Std. dev. | Cost of omission | Number of BF | Variable | Relative importance (%) |
---|---|---|---|---|---|
1 | 0.540 | 0.352 | 1 |
|
100.000 |
2 | 0.535 | 0.344 | 2 |
|
97.416 |
3 | 0.389 | 0.306 | 1 |
|
83.637 |
4 | 0.380 | 0.294 | 1 |
|
78.935 |
5 | 0.174 | 0.217 | 2 |
|
35.183 |
6 | 0.167 | 0.216 | 2 |
|
33.487 |
7 | 0.149 | 0.212 | 1 |
|
29.230 |
8 | 0.143 | 0.211 | 1 |
|
29.042 |
Basis functions and important explanatory variables for the MARS model with
Function | Std. dev. | Cost of omission | Number of BF | Variable | Relative importance (%) |
---|---|---|---|---|---|
1 | 0.789 | 0.363 | 2 |
|
100.000 |
2 | 0.761 | 0.356 | 2 |
|
98.013 |
3 | 0.570 | 0.304 | 2 |
|
79.033 |
4 | 0.531 | 0.289 | 2 |
|
72.994 |
5 | 0.412 | 0.225 | 1 |
|
35.419 |
6 | 0.402 | 0.225 | 1 |
|
34.692 |
7 | 0.349 | 0.221 | 1 |
|
30.760 |
8 | 0.315 | 0.218 | 1 |
|
27.641 |
9 | 0.317 | 0.218 | 1 |
|
27.426 |
10 | 0.306 | 0.217 | 1 |
|
26.965 |
Basis functions and important explanatory variables for the MARS model with
Function | Std. dev. | Cost of omission | Number of BF | Variable | Relative importance (%) |
---|---|---|---|---|---|
1 | 0.579 | 0.179 | 2 |
|
100.000 |
2 | 0.565 | 0.176 | 2 |
|
98.746 |
3 | 0.450 | 0.160 | 2 |
|
91.803 |
4 | 0.434 | 0.151 | 2 |
|
87.535 |
5 | 0.295 | 0.072 | 2 |
|
31.978 |
6 | 0.293 | 0.072 | 2 |
|
31.287 |
7 | 0.249 | 0.069 | 1 |
|
26.538 |
8 | 0.222 | 0.067 | 1 |
|
23.337 |
When the first stage of hybrid modeling was completed, the ANN topology settings were established. Table
ANN topology settings for different hybrid LR-ANN and MARS-ANN models.
Correlation |
Shift value |
Type of mean shifts | ANN topology for LR-ANN | ANN topology for MARS-ANN |
---|---|---|---|---|
0.1 | 0.5 | (1, 0, …, 0) | {8-8-1} | {8-9-1} |
(1, 0, 1, 0, …, 0) | {8-8-1} | {8-7-1} | ||
(1, 0, 1, 0, 1, 0, …, 0) | {8-8-1} | {8-10-1} | ||
1.0 | (1, 0, …, 0) | {11-13-1} | {5-5-1} | |
(1, 0, 1, 0, …, 0) | {11-12-1} | {5-6-1} | ||
(1, 0, 1, 0, 1, 0, …, 0) | {11-12-1} | {5-7-1} | ||
| ||||
0.5 | 0.5 | (1, 0, …, 0) | {10-10-1} | {10-10-1} |
(1, 0, 1, 0, …, 0) | {10-10-1} | {10-10-1} | ||
(1, 0, 1, 0, 1, 0, …, 0) | {10-10-1} | {10-10-1} | ||
1.0 | (1, 0, …, 0) | {9-9-1} | {8-9-1} | |
(1, 0, 1, 0, …, 0) | {9-7-1} | {8-8-1} | ||
(1, 0, 1, 0, 1, 0, …, 0) | {9-11-1} | {8-9-1} | ||
| ||||
0.9 | 0.5 | (1, 0, …, 0) | {8-9-1} | {10-9-1} |
(1, 0, 1, 0, …, 0) | {8-10-1} | {10-10-1} | ||
(1, 0, 1, 0, 1, 0, …, 0) | {8-9-1} | {10-11-1} | ||
1.0 | (1, 0, …, 0) | {2-2-1} | {8-6-1} | |
(1, 0, 1, 0, …, 0) | {2-2-1} | {8-8-1} | ||
(1, 0, 1, 0, 1, 0, …, 0) | {2-2-1} | {8-7-1} |
This study used the classical single stage of an ANN model and the proposed two-stage of MARS-ANN and LR-ANN models to determine the source of mean shift faults in a multivariate process. The experimental results are displayed in Table
Comparison of classification accuracy among the ANN, LR-ANN, and MARS-ANN models.
Correlation |
Shift value |
Type of mean shifts | ANN | LR-ANN | MARS-ANN |
---|---|---|---|---|---|
0.1 | 0.5 | (1, 0, …, 0) | 60.80% | 61.20% | 58.40% |
(1, 0, 1, 0, …, 0) | 60.80% | 59.20% | 62.40% | ||
(1, 0, 1, 0, 1, 0, …, 0) | 56.00% | 70.40% | 58.80% | ||
1.0 | (1, 0, …, 0) | 58.00% | 80.40% | 85.60% | |
(1, 0, 1, 0, …, 0) | 86.40% | 84.80% | 81.20% | ||
(1, 0, 1, 0, 1, 0, …, 0) | 56.00% | 92.80% | 86.00% | ||
| |||||
0.5 | 0.5 | (1, 0, …, 0) | 38.80% | 66.00% | 59.60% |
(1, 0, 1, 0, …, 0) | 46.00% | 55.60% | 62.00% | ||
(1, 0, 1, 0, 1, 0, …, 0) | 46.40% | 70.80% | 64.80% | ||
1.0 | (1, 0, …, 0) | 90.40% | 91.60% | 94.00% | |
(1, 0, 1, 0, …, 0) | 94.40% | 63.60% | 94.40% | ||
(1, 0, 1, 0, 1, 0, …, 0) | 85.20% | 96.40% | 94.00% | ||
| |||||
0.9 | 0.5 | (1, 0, …, 0) | 82.40% | 93.20% | 81.30% |
(1, 0, 1, 0, …, 0) | 94.40% | 82.80% | 94.40% | ||
(1, 0, 1, 0, 1, 0, …, 0) | 56.00% | 96.80% | 92.00% | ||
1.0 | (1, 0, …, 0) | 98.80% | 44.00% | 100.00% | |
(1, 0, 1, 0, …, 0) | 100.00% | 100.00% | 100.00% | ||
(1, 0, 1, 0, 1, 0, …, 0) | 98.40% | 100.00% | 99.20% | ||
| |||||
Average of accurate identification rates | 72.73% | 78.31% | 81.57% | ||
Standard error of accurate identification rates | 0.05007 | 0.04092 | 0.03767 |
Table
Table
AIR comparison of the classical-single stage and the proposed models.
ANN | Proposed | Proposed | |
---|---|---|---|
LR-ANN | MARS-ANN | ||
|
63.00% | 74.80% | 72.07% |
(0.1166) | (0.1344) | (0.1354) | |
|
66.87% | 74.00% | 78.13% |
(0.2565) | (0.1632) | (0.1760) | |
|
88.33% | 86.13% | 94.48% |
(0.1711) | (0.2161) | (0.0725) |
Table
AIR improvement of the proposed models in comparison with classical ANN model.
Proposed | Proposed | |
---|---|---|
LR-ANN | MARS-ANN | |
|
18.73% | 14.39% |
|
10.67% | 16.85% |
|
−2.50% | 6.96% |
| ||
Average improvement | 8.97% | 12.73% |
One important result is that our proposed approach is useful in dealing with difficulties of the smaller shifts for a multivariate process. The case of the smaller shift value (i.e.,
The ANN has been criticized for its long training process; however, the combination of LR/MARS and ANN is a good alternative for performing classification tasks. Accordingly, the proposed combination of the LR-ANN and MARS-ANN schemes was proven to be useful for determining the mean shift faults in a multivariate process.
The rationale behind the proposed schemes was initially to obtain fewer important explanatory variables by performing LR or MARS modeling. The resulting significant variables served as inputs to the designed ANN models. The proposed LR-ANN and MARS-ANN models not only have fewer input variables but also possess better classification capabilities.
The proposed hybrid two-stage models in this study are not the only combination techniques; other artificial intelligence techniques, such as decision tree or genetic algorithms, can be integrated with neural networks or a support vector machine to further refine the structure of the classifiers and improve classification accuracy. The applications of other process faults, such as variance shift faults, for a multivariate process should be further investigated.
The data-driven methods of multivariate statistical process control have been the subject of considerable interest from both the academic community and industry as an important implement in the process monitoring area. Since the practical systems become more and more complicated and the physical models become extremely hard to obtain, considering the related topics within data-driven framework seems more meaningful in the current and future work to achieve more industrial oriented results [
This work is partially supported by the National Science Council of the Republic of China, Grant no. NSC 102-2221-E-030-019 and Grant no. NSC 102-2118-M-030-001.