A Partial Hierarchical Model for Online Low-Resolution Wear Particle Images Classification

,


Introduction
In the mechanical system, most failures are caused by severe friction or wear. Wear particle monitoring and analysis has been deemed as a powerful technology for machine wear assessment [1,2]. It is considered that wear particles imply important clues for wear rate and wear mechanism [3]. Many methods [4][5][6] are used to recognize the features of wear particles. Among these techniques, image analysis is regarded as a very effective method for its easy acquiring and rapid processing speed [7][8][9]. e processing methods are undergoing through two generation approaches. e first generation is denoted by feature engineering [10][11][12][13]. Yuan et al. [11] analyzed the boundary of wear particles, extracted a variety of different boundary parameters, and used four different machine learning models to explore the performance of the model. Xu et al. [13] constructed a three-level classification model including K-means and support vector machine (SVM) by extracting the color and geometric features of wear particles. e model can classify different wear particles on different classifiers. Myshkin et al. [14] analyzed the color characteristics of metallic wear debris to recognize different wear debris. Stachowiak et al. [15] statistically analyzed the surface texture parameters of wear particles and designed an automatic classification system based on the surface texture of wear particles. Wang and Wang [16] combined principal component analysis (PCA) and grey relational analysis (CPGA) to identify different wear particles. CPGA is adopted to solve the complicated interrelationships between the specified characteristics of similar wear particles with features of color, shape factor, texture, and so on. However, there are two problems in the methods based on manual feature extraction. Firstly, the selection of wear particle features requires deep insight into both domain-expert knowledge and the learning algorithms. Secondly, feature extraction is not adaptive. When the dataset is transformed, the original features may reduce the recognition rate of the model.
With the development of machine learning, the second generation is denoted by deep learning networks [17][18][19][20][21][22][23]. Peng and Wang [21] proposed a small sample wear particle recognition model based on CNN. Compared with the general CNN model, this model can identify overlapping particles in the case of a small number of samples and uses one-dimensional convolution to reduce the computational complexity of the model. Peng et al. [22] proposed a hybrid CNN network to automatically classify four different wear particles. e hybrid model uses the idea of transfer learning to initialize the model parameters by using the learning parameters and weights of ImageNet. Aiming at the lack of enough information in two-dimensional wear particle images, Wang et al. [24] proposed a nonparametric recognition model of three-dimensional similar particles by taking fatigue and sliding wear particles as examples. is model can transform 3D image with rich information into 2D image and uses the CNN model to optimize. Peng and Wang [25] used the Inception-v3 network structure to automatically extract the characteristics of wear particles and then used the artificial neural network to classify the types of wear particles, which can realize the classification of overlapping particles. Liu et al. [26] constructed a deep convolutional neural network with Encoder-ASPP-Decoder architecture, which can classify and segment concurrently five different wear particles in ferrography images. However, in these works, the images are high definition ferrography images which have rich texture information. It is not economical and practical by using these kinds of acquisition equipment for online wear particles monitoring and analysis.
Online wear particle images acquiring and analysis make wear condition monitoring real time and straightforward by avoiding frequent oil sampling. Optical techniques using LED or laser beam and a charge-coupled device (CCD) sensor to acquire wear particle images are not new [1,12,27]. When particles pass through the view of light beam, their morphology can be captured by using the camera. However, the analysis of the online images is remained on the firstgeneration approach-feature engineering level, that is, because the online wear particle images are of low resolution and no texture information contained. Figure 1 shows the difference of high definition ferrography images (extracted from Ref. [18]) and online wear particle images. e classification of online low-resolution wear particle images by using the deep learning method is still not adequately addressed.
is paper introduces a partial hierarchical model of deep learning convolution neural network assembling with the feature engineering method, which can process and classify online low-resolution wear particle images in onestop manner. e proposed network structure is called SR-PHnet. It includes three parts, super-resolution layer, the first-stage classifier module, and the second-stage classifier module. Firstly, the low-resolution images are transformed into super-resolution images by super-resolution image reconstruction in super-resolution layer. en, the superresolution images are input to the classification module. e partial hierarchical mechanism is put forward with different classification modules to precision recognize the easy recognition particles (e.g., cutting, sphere, and nonmetallic particles) and difficult recognition particles (e.g., fatigue and sliding particles). e easily confused fatigue and sliding wear particles are identified as one class on the convolution neural network classification module and then recognized separately on the SVM classifier module of the SR-PHnet model. Firstly, convolutional neural network is adopted for classifying four categories (cutting, sphere, nonmetallic, and fatigue or sliding particles); then, fatigue and sliding particle images are inputted into the second classifier for further recognition. A new feature, called radial edge factor (REF), is proposed, which is combined with area, aspect ratio, and roundness to identify fatigue and sliding particles better. e main contributions of this work are as follows: (1) e super-resolution layer, the first-stage classifier module and the second-stage classifier module are integrated together to realize unclear online wear particles images in one step. (2) e partial hierarchical modeling method is effective for easy recognition particles and difficult recognition particles (e.g., fatigue and sliding particles). e former is recognized by convolution neural network, and the latter is classified by the feature engineering method. (3) Radial edge factor (REF) feature, which can be decoupled from various features, is put forward specially for fatigue and sliding particles. (4) e mechanism of partial hierarchical modeling determines that the modeling scale and the number of features are smaller but more effective than recent modeling methods for online wear particle classification. It provides higher recognition rate and less computing time.
e detailed procedure of modeling is described in Section 2. e experiments and discussion are explained in Section 3 and Section 4.

The Proposed SR-PHnet for Online Low-Resolution Wear Particle
Images Classification e overall structure of the proposed SR-PHnet is shown in Figure 2. Computational Intelligence and Neuroscience

Super-Resolution Layers.
Since the online collected wear particle image is with low resolution (LR), many details are lost compared with the ferrography wear particle image. When the low-resolution image is directly used as the input of CNN, the prediction accuracy is relatively low. erefore, we hope to recover some image details from low-resolution wear particle images, and image super-resolution reconstruction [28] is used to achieve this effect. Image superresolution reconstruction is a technique to reconstruct a high-resolution (HR) image by using one or more lowresolution images based on certain assumptions or prior information [29]. In this paper, the single image superresolution reconstruction (SISR) [30] technology is used to achieve the goal of image reconstruction. SISR techniques are generally classified into three categories: interpolationbased, reconstruction-based, and sample-based learning. Bicubic interpolation [31] is one of the most common methods, but it cannot reconstruct better image details. When the magnification factor is large, the image obtained by bicubic interpolation will be blurred at the edge. e reconstruction-based method [32] focuses on reconstructing lost high-frequency signals.
is method only uses some prior knowledge to regularize the reconstruction process and does not use training ideas to obtain prior information.
e learning-based method [33,34] uses machine learning or deep learning algorithm to learn the mapping relationship between the LR image and the HR image to predict the missing high-frequency information of the image to reconstruct the HR image. e learning-based SISR method is used in this paper to reconstruct the low-resolution wear particle image.
As shown in Figure 2, three convolutional layers are used to achieve super-resolution reconstruction. Each convolution layer does not change the size of the image but change the number of channels of the image. After passing three convolutional layers, the original image size and channel number are restored. e convolution kernel sizes of the three convolutional layers are 7 × 7, 5 × 5, and 3 × 3, respectively, and the zero-padding required is 3, 2, and 1, respectively. e kernel stride is 1, and ReLU activation function is used for each layer. e output size of each layer is as follows:  Computational Intelligence and Neuroscience Here, i is the size of the input image, k is the convolution kernel size of the layer, p is the zero-padding number, and s is the kernel stride. e input of the super-resolution layer is the original low-resolution (LR) image, and the output is the changes required to convert the low-resolution (LR) image into the super-resolution (SR) image, that is, the pixel difference between the SR image and the LR image. Since the SR image needs to match the input size of the convolutional classification layer, we apply the bicubic interpolation method for each LR image to enlarge it into the matched size. en, through the super-resolution layer, we obtain the difference image and add it to the corresponding LR image to obtain the transformed SR image. e SR image I SR is solved by the following formula: where I LR represents the amplified low-resolution (LR) image through bicubic interpolation and s(I LR ) represents the output of the super-resolution layer. e super-resolution layer learns the mapping function from the low-resolution image to the difference between the super-resolution image and the low-resolution image [35]. In order to learn the mapping function, we propose the cost function J SR for the super-resolution layer as follows: where I HR is the high-resolution form of image I and I SR is the super-resolution reconstruction image obtained after image I passes through the super-resolution layer. e optimization objective of the super-resolution layer is to minimize the cost function value J SR . We use ImageNet datasets to train the super-resolution layer. e high-resolution images I HR in the ImageNet dataset are first down-sampled to generate the low-resolution images I LR and then amplified to the original image size to obtain the difference Δ(I HR − I LR ) between it and the original high-resolution image.

e CNN Classifier Module of SR-PHnet Model.
In view of the advantage that no need to extract features manually for images classification by the deep learning method [36][37][38][39], the proposed SR-PHnet model in this paper uses convolution neural network as the first classifier to recognize wear particles automatically. ere contains convolutional layer, pooling layer, and fully connected layer [36] in firststage classifier. e images after super-resolution layer are inputted to the convolutional layer as shown in Figure 2. e output is the classification of wear particles. Here, we use four categories, cutting, sphere, nonmetallic, and FoS particles (for convenience of expression, FoS is used to as the abbreviation of fatigue or sliding particles). e new portable convolutional neural network ShuffleNetV2 [40,41] structure is used to classify the wear particles. is structure uses block convolution instead of traditional convolution layer, which greatly accelerates the training and testing process of the network.

Convolutional Layer.
Convolution layer is convoluted by multiple convolution kernels. Convolution kernel is generally composed of a three-dimensional weight tensor and a bias. It is assumed that the input tensor is T in ∈ R W 1 * H 1 * L 1 , where R W 1 * H 1 * L 1 represents a three-dimension real number tensor of size W 1 * H 1 * L 1 . W 1 ,H 1 , and L 1 are width, height, and depth (also called channel number), respectively. e output of convolution layer is also a three-dimensional tensor, which is recorded as T out ∈ R W 2 * H 2 * L 2 , where R W 2 * H 2 * L 2 represents a three-dimension real number tensor of size W 2 * H 2 * L 2 . en, the convolutional operation is calculated as follows: where y (i,j,k) is the value of T out at position (i, j, k), W k is the weight tensor of the kth convolution kernel of the convolution layer, b k is the bias vector of the kth convolution kernel of the convolution layer, and Xis the input tensor, for the first convolution layer, X � T in . W C , H C , and L C are the width, height, and depth of the kth convolution kernel, respectively. e total parameter of the kth convolution kernel isW C * H C * L C + l, W k (i′,j′,l) is the value of W k at position (i ′ , j ′ , l), x ((i−1) * s+i′ * s+j′,l) is the value of X at position ((i − 1) * s + i ′ , (j − 1) * s + j ′ , l), b k l is the value of b k at position l, s is the stride of moving the convolution kernel, i.e., it is calculated by adding across s positions, and f C is a function. ReLU [36] operation is adopted, and its calculation is expressed as follows: where x is the output of the upper layer network.

Pooling Layer.
e pooling layer is a layer without parameters. e output of general pool layer is also a threedimensional tensor. Generally, there are maximum pooling layer and average pooling layer. e maximum pooling operation is generally used in the pooling layer. e maximum pooling operation can be described by a mathematical formula as follows: p (p,q,k) � max y s p * (p− 1)+1: s p * (p− 1)+h p , s q * (q− 1)+1: s q * (q− 1)+w p ,k , where s p is span and h p and w p are width and height of neighborhood, respectively. 4 Computational Intelligence and Neuroscience

Full Connect Layer.
e full connection layer is generally composed of weight matrix and offset for vector input. Suppose the input is x � [x 1 , x 2 , . . . , x n ]. e weight matrix of all connected layer is W ∈ R m * m , and the offset is b ∈ R m . e output is A � [a 1 , a 2 , . . . , a m ], and they are described as follows:

e SVM Classifier Module of SR-PHnet Model.
After the first classification, cutting, sphere, nonmetallic, and FoS (fatigue or sliding) particles are recognized automatically without the guidance of feature manual extraction. e objective of the SVM classifier module of the SR-PHnet model is to recognize fatigue or sliding particles. e partial hierarchical mechanism can reduce the model scale, save computing time, and enhance the recognition accuracy.
To improve the recognition accuracy of fatigue and sliding wear particle images, feature extraction is used for guidance of classification. e normal features for recognizing different wear particles are area, aspect ratio, roundness, shape factor, and so on. e recognition rate for all wear particles by only using the feature engineering method is not high. One of the reasons is that the multiple features been coupled together make it hard to distinguish all kinds of wear particles and make the model complicated. Characteristic feature is beneficial for improving recognition rate and reducing model scale. For just aiming at fatigue or sliding particles, it is possible to make features decoupling from the others and reduce the number of features to complete the identifying task. For fatigue or sliding particles, a new feature, radial edge factor (denoted as REF), is put forward in this paper. e schematic diagram is shown in Figure 3. e brown dotted circle in Figure 3 is an analytically equivalent circle (AEC) [11] with equal perimeter. It is the ratio of the characteristic radius to the minimum radius of the inscribed circle. It is denoted as follows: Here, CR is the characteristic radius of wear particles and r is the minimum radius of the inscribed circle as shown in Figure 3. e characteristic radius CR is calculated as follows: Here, P is perimeter of wear particle. Eight neighborhood distances are used as the perimeter of wear particles by accumulating edge pixel points. It is slightly different from the actual perimeter drawn as the line, but it has short computing time and little influence on the value.  Table 1. A is the area from the number of pixels of wear particle image. AR is the ratio of length to width. It is calculated as follows: Here, L is the length of the smallest circumscribed rectangle surrounding the wear particle and W is the width of the smallest circumscribed rectangle surrounding the wear particle.
Roundness of wear particle is the similarity between the shape of wear particles and the circle. It is denoted as follows: Here, A is particle area and P is perimeter of wear particles. e SVM algorithm [42,43] is adopted as the feature engineering modeling, and the outputs are two classifications, fatigue and sliding wear particles. For the two-classifier, the kernel function used is the Gaussian kernel function RBF, and the 3-fold cross validation is used to verify the generalization ability of the model.

Dataset.
Online images acquisition is carried out with the CCD camera. e typical labeled images are shown in Figure 4. e collected image dataset is divided into two
After super-resolution reconstruction, the images are transformed to the resolution of 224 × 224 × 3 and are input into the CNN classifier module of the SR-PHnet model for training. e structure settings of the superresolution layer and the convolution neural network classifier are listed in Table 2. For the convolution neural network classifier, in the training, the batch size is 64, the learning rate is 0.0004, and the iteration number is 2000. e output of the CNN classifier includes four categories, i.e., cutting, sphere, nonmetallic, and fatigue or sliding (FoS) particles. e classification result is listed in Table 3. It can be seen that the recognition accuracy of cutting, sphere, and nonmetallic particles is high. Even for FoS particles, it is high to 92.5%, that is, because the automatic feature extraction in the SR-PHnet model is suitable for distinctive wear particle images.
After CNN classifier module of the SR-PHnet model, fatigue or sliding wear particles are transferred to the SVM classifier module of the SR-PHnet model. e SVM classifier distinguishes FoS particles with four features.
ere are 941 samples in use, including 483 sliding grains and 458 fatigue grains. Using 3-folded cross validation, 142 sliding wear particles and 141 fatigue wear particles were randomly selected as test set and 341 sliding wear particles and 317 fatigue wear particles as training set. e recognition accuracy is listed in Table 4. It shows that just four features, including the new feature REF proposed in this paper, have a good performance for distinguishing fatigue or sliding wear particles. e recognition accuracy of fatigue particles and sliding particles is up to 92.9% and 91.5%, respectively.
Finally, the recognition accuracy of the SR-PHnet model for all the five types of wear particles is listed in Table 5. From the recognition result, the average recognition accuracy is up to 92.7%, and the accuracy for fatigue and sliding particles of online recognition is 85.9% and 84.6%, respectively. It has great progress in online wear particles recognition. e recognition accuracy is just 85.7% and 80% for fatigue and sliding particles in the offline model of Wang et al.'s [23] with high definition ferrography images. Our model is preferable for its lightweight computing and high recognition accuracy on low-resolution images.

Discussion
For distinguishable wear particles with distinctive features (e.g., cutting, sphere, and nonmetallic wear particles), the convolutional neural network method is preferred. Because it recognizes particles directly without feature engineering, it has high recognition accuracy and short computing time. For specific particles that are difficult to identify (e.g., fatigue and sliding wear particles), feature engineering is a good choice for recognition. So, the partial hierarchic method is the optimal combination by using convolutional neural network as the first classifier module and feature engineering method SVM as the second classifier module.
To compare the recognition performance, the method of the baseline standard CNN model (AlexNet, ResNet50, and ShuffleNetV2), SVM without REF feature, SVM with REF feature, and our SR-PHnet model proposed in this paper is executed, respectively. e comparison results with AlexNet, ResNet50, and ShuffleNetV2 are listed in Table 6. e comparison of recognition precision with AlexNet, SVM without REF feature, SVM with REF feature, and our SR-PHnet is shown in Figure 5. Not only the total sample recognition precision but the fatigue and sliding particles recognition precision, the method of our SR-PHnet model has the best performance as shown in Figure 5. e SVM algorithm has better performance than the AlexNet model for sliding and fatigue particles, that is, because feature extraction manually can customize the feature parameters and it is better than feature extraction automatically for confusing recognition objects. SVM with the REF feature  e SR-PHnet model proposed in this paper combines the respective advantages and avoids the disadvantages of different modeling methods for feature extraction automatically and manually. Regardless of sliding and fatigue particles recognition or the average recognition accuracy, SR-PHnet has the best performance. e comparison of recognition accuracy with iteration times of our SR-PHnet model and AlexNet is shown in Figure 6. It shows that the SR-PHnet model needs fewer iteration times to achieve high recognition accuracy and convergence to stability.
To compare our proposed model with the other method, the same dataset of this paper is executed by the model of Ref. [23] in the comparison experiment. e BP-CNN model in Ref. [23] is constructed. Due to the lack of texture information, the feature parameters adopted in the BP-CNN model are set as the same as the SR-PHnet model. e comparison result is listed in Table 7.
Comparing between the BP-CNN in Ref. [23] and SR-PHnet model, the proposed SR-PHnet model has a much higher recognition accuracy than BP-CNN in the three wear particles: cutting, spherical, and nonmetal. e recognition rate of fatigue and sliding particles is also improved in the SR-PHnet model, which shows the effectiveness of the SR-PHnet model. It can be seen that the SR-PHnet model is more adaptable. Because CNN is the first classifier to automatically extract image features, it is more adaptive than manual feature extraction. In the method of BP-CNN, BP neural network is as the first classifier which needs manual feature extraction. It limits the recognition accuracy due to    Computational Intelligence and Neuroscience   Computational Intelligence and Neuroscience the error of manual feature extraction in the first step. e SR-PHnet model has higher accuracy for not only cutting, spherical, and nonmetal but also for fatigue and sliding particles.

Conclusions
is work aims to develop a super-resolution reconstruction partial hierarchical SR-PHnet model for online wear particle recognition and classification. e low-resolution online wear particle images lacking of texture information are firstly reconstructed by super-resolution technique, and then the super-resolution images are input to the classifier of the SR-PHnet model. e types of wear particles are merged into four categories (cutting, sphere, nonmetallic, and fatigue or sliding (FoS) particles) on the first state classifier, and then fatigue and sliding (FoS) wear particle images are transferred to the second-stage classifier. On the first stage, there are characteristic patterns existing in the four categories, so convolution neural network is the best classifier for its high recognition precision and short computing time. e recognition accuracy is up to 94%. On the second stage, the feature engineering method is the best choice for recognizing the easily confused fatigue and sliding wear particles. A new feature REF is put forward to combine with the other three features to obtain a high recognition rate. e final recognition accuracy of fatigue and sliding particles after the SR-PHnet model is up to 85.9% and 84.6%, respectively. e comparison experiment results confirm the effectiveness of the SR-PHnet model in online wear particles recognition and classification for low-resolution wear particle images lacking of texture information.
Data Availability e original data used in the study can be made available from the corresponding author upon reasonable request.

Disclosure
Xuxu Guo and Rui Tan are co-first authors. Computational Intelligence and Neuroscience 9