Multi-Layer Perceptron Classifier with the Proposed Combined Feature Vector of 3D CNN Features and Lung Radiomics Features for COPD Stage Classification

Computed tomography (CT) has been regarded as the most effective modality for characterizing and quantifying chronic obstructive pulmonary disease (COPD). Therefore, chest CT images should provide more information for COPD diagnosis, such as COPD stage classification. This paper proposes a features combination strategy by concatenating three-dimension (3D) CNN features and lung radiomics features for COPD stage classification based on the multi-layer perceptron (MLP) classifier. First, 465 sets of chest HRCT images are automatically segmented by a trained ResU-Net, obtaining the lung images with the Hounsfield unit. Second, the 3D CNN features are extracted from the lung region images based on a truncated transfer learning strategy. Then, the lung radiomics features are extracted from the lung region images by PyRadiomics. Third, the MLP classifier with the best classification performance is determined by the 3D CNN features and the lung radiomics features. Finally, the proposed combined feature vector is used to improve the MLP classifier's performance. The results show that compared with CNN models and other ML classifiers, the MLP classifier with the best classification performance is determined. The MLP classifier with the proposed combined feature vector has achieved accuracy, mean precision, mean recall, mean F1-score, and AUC of 0.879, 0.879, 0.879, 0.875, and 0.971, respectively. Compared to the MLP classifier with the 3D CNN features selected by Lasso, our method based on the MLP classifier has improved the classification performance by 5.8% (accuracy), 5.3% (mean precision), 5.8% (mean recall), 5.4% (mean F1-score), and 2.5% (AUC). Compared to the MLP classifier with lung radiomics features selected by Lasso, our method based on the MLP classifier has improved the classification performance by 5.0% (accuracy), 5.1% (mean precision), 5.0% (mean recall), 5.1% (mean F1-score), and 2.1% (AUC). Therefore, it is concluded that our method is effective in improving the classification performance for COPD stage classification.


Introduction
Chronic obstructive pulmonary disease (COPD) is a common and non-infectious lung disease characterized by persistent airfow limitation [1][2][3].Because of this characterization, the COPD stage is diagnosed from stage 0 to IV according to Global Initiative for Chronic Obstructive Lung Disease (GOLD) criteria accepted by the American Toracic Society and the European Respiratory Society [4].GOLD is examined by the pulmonary function test (PFT) and diagnosed by the forced expiratory volume in 1 second/forced vital capacity (FEV1/FVC) and FEV1% predicted [1,2].PFT can explain the impact on symptoms and life quality of COPD patients [5,6], but it cannot refect the change of the lung tissue in COPD patients with the COPD stage evolution.PFT changes from normal to abnormal occur when lung tissue is destroyed to a certain extent.Terefore, the PFT makes it challenging to identify the etiology of COPD.
Compared with the GOLD criteria and other imaging equipment, computed tomography (CT) has been regarded as the most efective modality for characterizing and quantifying COPD [7].Compared with PFT, chest CT images can indicate that the patients have sufered from mild lobular central emphysema and decreased exercise tolerance in smokers without airfow limitation [8].In addition, the chest CT images are also used to quantitatively analyze the bronchial, airway disease, emphysema, and vascular for COPD patients [7].However, automatic multi-classifcation based on convolutional neural networks (CNNs) using chest CT images remains a challenging task for the COPD stage.One main reason is that the number of medical images is limited compared to natural images.In particular, few people seek medical treatment in the early stage of COPD and undergo CT scans simultaneously.Transfer learning [9] may solve the above problems.Since radiomics was proposed to mine more information from medical images using advanced feature analysis in 2007 [10], it has been widely used to analyze lung disease imaging [11][12][13][14][15].However, radiomics features are extracted from medical images by specifc calculation equations, preset types of images, and preset classes, limiting the forms of radiomics features.Some deep features from CNN (CNN features) are also needed to improve the classifer's performance in multi-classifcation.CNN features extracted from medical images will make up for the limitations of radiomics features.
Radiomics features in COPD develop slower than those in other lung diseases, such as lung cancer and pulmonary nodules.Until 2020, reference [16] points out that radiomics features in COPD have not been extensively investigated yet.Nevertheless, there are potential applications of radiomics features in COPD for the diagnosis, treatment, and followup of COPD and future directions [16].A critical reason limiting the development of radiomics features in COPD is its difuse distribution in the lung.At the same time, radiomics features need to be extracted from the region of interest (ROI) of the chest CT images.However, the difuse distribution of COPD makes it difcult to determine ROI.COPD results from the joint action of the peripheral airway, pulmonary parenchyma, and pulmonary vessels [17][18][19].Tus, the peripheral airway, pulmonary parenchyma, and pulmonary vessels as ROI to extracting lung radiomics features are reasonable for COPD stage classifcation.
Currently, radiomics features also have been used in COPD for survival prediction [20,21], COPD presence prediction [22], COPD exacerbations [23], COPD early decision [4], and analysis of COPD and resting heart rate [3].However, as mentioned above, lung radiomics features have not been applied in the COPD stage classifcation.On the other hand, radiomics based on machine learning (ML) and chest CT images based on CNN have been widely and respectively used in COPD and its evaluation.However, the advantages of radiomics based on machine learning and medical images based on CNN need to be further integrated to improve the performance of COPD stage classifcation.Terefore, this paper proposes a feature combination strategy by concatenating three-dimension (3D) CNN features and lung radiomics features for COPD stage classifcation based on the multilayer perceptron (MLP) classifer.Our contributions in this paper are briefy described as follows.(1) MLP classifer with the best classifcation performances is determined in the ML classifer for 3D CNN features or lung radiomics features.(2) Truncated transfer learning is proposed from the excellent segmentation model for generating nonlinear 3D CNN features.(3) Te proposed feature combination strategy by concatenating 3D CNN features and lung radiomics features efectively improves the MLP classifer's performance.

Materials.
Te participants are enrolled by the national clinical research center of respiratory diseases, China, from May 25, 2009, to January 11, 2011. Finally, 465 Chinese subjects participated in the study after being strictly selected by the inclusion and exclusion criteria [24].Te 465 subjects underwent chest HRCT scans at the full inspiration state.In addition, the 465 subjects also underwent the PFT, and the COPD stage of each subject is diagnosed by PFT in Global Initiative for Chronic Obstructive Lung Disease (GOLD) criteria 2008 accepted by the American Toracic Society and the European Respiratory Society.
Figure 1 shows the COPD stage distribution of the subjects in this study.Tere are 129, 108, 121, and 107 subjects in each COPD stage (GOLD 0, GOLD I, GOLD II, GOLD III, and GOLD IV).Tis study was approved by the ethics committee of the national clinical research center for respiratory diseases in China.In addition, all 465 subjects have been provided written informed consent to the frst afliated hospital of Guangzhou medical university before chest HRCT scans and PFT.Refer to our previous study [4] for a more detailed description of the materials.

Methods.
Figure 2 shows the proposed method in this study.Te main idea of the proposed method proposed in this paper is to combine 3D CNN features and lung radiomics features for COPD stage classifcation.When generating the 3D CNN features, we adopt a truncated transfer learning strategy that only intercepts the encoder backbone of the pretrained Med3d [25].

Lung Radiomics Features Extraction.
First, 465 sets of chest HRCT images are automatically segmented by a trained ResU-Net [26], obtaining 465 sets of lung images with the Hounsfeld unit (Hu) [27].Te lung images include the peripheral airway, pulmonary parenchyma, and pulmonary vessels.Te architecture of the ResU-Net has been described in detail in our previous study [28].Ten, lung radiomics features of 465 subjects are extracted from the

Encoder backbone Decoders
Med3d [25] ...  Journal of Healthcare Engineering lung images by PyRadiomics [29].Refer to our previous study [4] for a more detailed description of the lung radiomics feature extraction.

3D CNN Feature Extraction.
A truncated transfer learning strategy is proposed to extract the 3D CNN features based on the pretrained Med3d [25].Med3d, a heterogeneous 3D network, is used to extract general medical 3D features by building a 3DSeg-8 dataset with diverse modalities, target organs, and pathologies.Tus, we only transfer the encoder backbone of the pretrained Med3d (3D ResNet10) for generating the 3D CNN features, as shown in Figure 2(a).
Figure 2(b) shows that the 465 sets of lung images with Hu are input to the transferring encoder backbone, generating CNN feature vectors in detail.First, the lung images (512 × 512 × N) are cropped into the size 280 × 400 × N′, retaining the lung region.Te non-lung images are also deleted, so N changes into N′ (N′ < N).Second, the cropped lung images are preprocessed by the method in reference [25], normalizing the lung region and generating random values outside the lung region in accordance with Gaussian distribution.Equation (1) shows the mathematical form of normalization: where x is the value of the lung region, x is the mean value of the lung region, and σ is the mean square deviation of the lung region.Tird, the CNN feature maps (512 × 35 × 50 × 75) are generated by the cropped and preprocessed lung images (1 × 280 × 400 × N′) and the pretrained Med3d.Last, higherorder CNN feature maps (512 × 3 × 3 × 3) need to be extracted from the CNN feature maps (512 × 35 × 50 × 75) by 3D average pooling.Ten, the higher-order CNN feature maps (512 × 3 × 3 × 3) are fattened into the CNN feature vector.Finally, each CNN feature vector (per subject) includes 13824 3D CNN features (512 × 3 × 3 × 3 � 13824).

Combined Feature Vector for COPD Classifcation.
Figure 2(c) shows that the combined feature vector is generated by concatenating the CNN feature vector and the radiomics feature vector.First, the CNN feature vector (13824) and the radiomics feature vector (1316) are selected by the least absolute shrinkage and selection operator (Lasso) [30], respectively.After Lasso, the number of the selected CNN feature vector and the selected radiomics feature vector is 60 and 106, respectively.A standard python package LassoCV, with tenfold cross-validation, is performed in this paper.Equation (2) shows the mathematical form of Lasso [4]: where matrix A denotes the selected lung radiomics feature.
x ij denotes the lung radiomics features (the independent variable).y i denotes the COPD stage (the independent variable).λ denotes the penalty parameter (λ ≥ 0).β j denotes the regression coefcient, i ∈ [1, n], and j ∈ [0, p].Second, the combined feature vector is generated by concatenating the selected CNN feature vector and the selected radiomics feature vector.Finally, the combined feature vector is the size 1 × 166 per subject.Figure 2(d) shows that MLP [31,32] with the combined feature vector is used to classify the COPD stage in this paper.

Experiments and Evaluation Metrics.
Our proposed method uses the combined feature vector of 3D CNN features and lung radiomics features for COPD stage classifcation based on the MLP classifer.Our experiment includes fve experiments in this section to verify the efectiveness of our proposed method.
Figure 3 shows the experimental design in this paper.End-to-end CNN models based on parenchyma images are used for COPD stage classifcation in experiments 1-2.Specifcally, two classic CNN models, DenseNet and Goo-gLeNet, based on parenchyma images, are adopted to compare the classifcation performance of the six diferent ML classifers.Te classifcation performance of DenseNet and GoogleNet has been evaluated by our previous study [33], which achieved the best classifcation performance for image classifcation.Furthermore, compared with experiment 1, multiple-instance learning (MIL) [34], a form of weakly supervised learning, is applied in experiment 2. Specifcally, experiments 3-5 are designed to compare the classifcation performance of the six diferent classifers based on the CNN feature vector (13824), radiomics feature vector (1316), their selected feature vector by Lasso, and the proposed combined feature vector (166), respectively.First, based on 3D ResNet10, we use six classic classifers (SVM [35], MLP, RF [36], LR [37], GB [38], and LDA [39]) to determine the best COPD classifcation classifer by diferent feature vectors.Table 1 reports the six diferent classifers with their defnitions in this paper.Te diferent feature vectors include the CNN feature vector (13824), CNN feature vector selected by Lasso (60), radiomics feature vector (1316), and radiomics feature vector selected by Lasso (106).Te MLP classifer with the best classifcation performance is determined.Second, we further verify the proposed combined feature vector (166) to improve the MLP classifer's performance.Tird, 3D ResNet18 and 3D ResNet34 are also transferred to generate the CNN feature vector, and the 3D ResNet10 is determined as the encoder backbone with the best performance on the MLP classifer.Te 465 subjects are divided into the train set (70%) and the test set (30%). Figure 4 shows the detailed dataset division for training and test set in each COPD stage.
Standard evaluation metrics of the CNN and ML models include the accuracy, precision, recall, F1-score, and area under the curve (AUC).Te above standard evaluation metrics are defned as in equations ( 3)-( 6).Te evaluation metric AUC for multi-classifcation is calculated by the receiver operating characteristic curve (ROC) [40].
where the true positive (TP) and false positive (FP), respectively, represent the positive and negative samples classifed to be positive by the CNN and ML models and the true negative (TN) and false negative (FN), respectively, represent the positive and negative samples classifed to be negative by the CNN and ML models.

Results
Tis section reports the experimental results, including (1) the classifcation performance of the parenchyma images based on the DenseNet and GoogleNet; (2) the classifcation performance of the CNN feature vector and lung radiomics vector based on diferent classifers; (3) the MLP classifer's performance with the combined feature vector; and (4) the MLP classifer's performance with combined feature vector based on diferent 3D ResNet.

Te Classifcation Performance of CNN Feature Vector and Lung Radiomics Vector Based on Diferent Classifers.
Tis section shows the classifcation performance of the CNN feature vector (13824), the CNN feature vector selected by Lasso (60), the lung radiomics vector (1316), and the lung radiomics vector selected by Lasso (106) based on diferent classifers, respectively.

6
Journal of Healthcare Engineering Table 5 also reports that Lasso only plays a role in improving the classifcation performance of the MLP classifer with the CNN feature vector.It does not improve the classifcation performance of other classifers with the CNN feature vector.However, Table 7 reports that Lasso does play a role in improving the classifcation performance of all classifers with the radiomics feature vector.

Te MLP Classifer's Performance with Combined
Feature Vectors.Te best MLP classifer is determined with the CNN feature vector selected by Lasso (60) or the lung radiomics vector selected by Lasso (106) by Section 3.1.Tis section shows the classifcation performance of the MLP classifer with combined feature vectors.Journal of Healthcare Engineering vectors improve the MLP classifer's performance, achieving 0.879 (accuracy), 0.879 (mean precision), 0.879 (mean recall), 0.875 (mean F1-score), and 0.971 (AUC), respectively.

Te MLP Classifer's Performance with Combined Feature
Vector Based on Diferent 3D ResNet.Te best MLP classifer is determined with the CNN feature vector selected by Lasso (60) or the lung radiomics vector selected by Lasso (106) by Section 3.1.Tis section shows the classifcation performance of the MLP classifer with combined feature vectors.
Figure 8 intuitively shows the confusion matrix and ROC curves of the MLP classifer with combined feature vectors based on diferent 3D ResNet.Te MLP classifer's performance with combined feature vectors based on diferent 3D ResNet reported in Table 7 can be calculated from the confusion matrix.Table 9 reports that the MLP classifer with combined feature vectors based on 3D ResNet10 achieves the best classifcation performance.

Discussion
Tis paper proposes a features combination strategy by concatenating 3D CNN features and lung radiomics features for COPD stage classifcation based on the MLP classifer.Tree sections are discussed in this section, and we also point out the limitations in this study and the future direction.
First, 2D GoogleNet with parenchyma images performs the best in 2D CNN models.Te main reason is that 2D GoogleNet is designed for 2D natural image classifcation (RGB images).Terefore, it achieves the best classifcation performance in 2D parenchyma images.Meanwhile, because of the ability to extract interlayer information, 3D DenseNet with parenchyma images performs the best classifcation in 3D CNN models.However, CNN models with parenchyma images fail to classify the COPD stage.One main reason is that the chest HRCT image cannot fully refect COPD's characteristics for the CNN models.Specifcally, the gold standard of COPD classifcation is characterized by airfow restriction with a slight diference in the chest HRCT image.Te slight diference in COPD is mainly caused by small airway disease with an airway diameter<2 mm [17].Because of the limitation of HRCT resolution, the above diferential features of the small airway will be further blurred in the chest HRCT image.Another reason is that chest HRCT images can refect the COPD anatomical characteristics, but COPD patients are with high heterogeneity and diferent phenotypes [1].Te heterogeneity and diferent phenotypes often result in diferent features of the chest HRCT images in the same stage.Terefore, it is hard for CNN models to learn specifc COPD characteristics, resulting in bad classifcation performance.At the same time, a set of standard medical images is not as easy to obtain as natural images, and the number of chest HRCT images also restricts CNN models for COPD stage classifcation.Terefore, compared with CNN models, the ML classifer can realize the COPD stage classifcation with a small number of samples.Tis paper determines the MLP classifer with 3D CNN features or lung radiomics features, which performs the best for COPD stage  Journal of Healthcare Engineering Journal of Healthcare Engineering classifcation.In addition, compared with the convolution layer in the CNN models, the MLP classifer is composed of three full connection layers, which is more efcient and more suitable for modeling long-range dependencies.Te MLP classifer also can handle complex nonlinear features and discover dependencies between diferent input features compared with other classifers [31,32].Meanwhile, the objective evaluation of the COPD stage is only the degree of airfow limitation tested by GOLD criteria [1,2,4].COPD is a heterogeneous disease [41], resulting in diferences in features (3D CNN features or lung radiomics features extracted from chest HRCT images) with the same degree of airfow limitation.Terefore, a nonlinear relationship exists between 3D CNN features or lung radiomics features and  Second, Lasso can improve the classifcation performance of the MLP classifer with the 3D CNN features and the lung radiomics features.Lasso is often used with survival analysis models to determine variables and eliminate the collinearity problem between variables [30,42].Te results show that Lasso also can improve the MLP classifer's classifcation performance by establishing the relationship between the independent variables (3D CNN features or lung radiomics features extracted from chest HRCT images) and dependent variables (the COPD stages).Furthermore, Lasso selects 3D CNN features or lung radiomics features related to COPD stages to reduce the complexity of the MLP classifers and avoid overftting [43].While reducing the complexity of the MLP classifers, the MLP classifers can focus on the selected lung radiomics features (the radiomics feature vector selected by Lasso) or the selected 3D CNN features (the CNN feature vector selected by Lasso) and improve the classifcation performance.From the results of the Lasso, the number of the CNN feature vector selected by Lasso is 60, and that of the radiomics feature vector selected by Lasso is 106.We are surprised that the number of collinearity features in the CNN feature vector is more than that in the radiomics feature vector.Tis further shows that feature selection of 3D CNN features or the radiomics features is necessary for the COPD stage classifcation, especially in clinical applications.
Tird, the proposed feature combination strategy can further improve the classifcation performance of the MLP classifer.Tis paper does not improve the existing classic classifers and starts with the classifcation features to enhance the classifer's performance.Many nonlinear classifcation features, the 3D CNN features, are obtained by a truncated transfer learning strategy.We concatenate the CNN feature vector and the radiomics feature vector for the COPD stage classifcation, which improves the MLP classifer's performance.Te MLP classifer is good at handling complex nonlinear features by itself [31,32].Terefore, based on the radiomics feature vector, we add the nonlinear CNN feature vector to the radiomics feature vector, generating a combined feature vector.Te combined feature vector with the nonlinear CNN feature vector enhances the MLP classifer's performance.Terefore, this fts the essence of the MLP classifer and is interpretable [44].Te selected encoder backbone of the pretrained Med3D is also directly related to the classifcation performance.Compared with the MLP classifer with 3D ResNet18 or 3D ResNet34, the MLP classifer with 3D ResNet10 performs the best, consistent with the results of multi-class segmentation task (left lung, right lung, and background) in reference [25].
Finally, this study has some limitations, and we point out the future direction.First, from the materials used in this study, there are not enough cases at the COPD stages III and IV.Second, the existing classic classifers are not improved.Tird, the classifcation performance of the ML classifer with the 3D CNN features is also limited by the encoder backbone of the pretrained Med3d.In our future work, the recent networks, an auto-metric graph neural network [45], will be further attempted and modifed for COPD stage classifcation based on the 3D CNN features and/or the lung radiomics features.

Conclusions
Tis paper proposes a feature combination strategy by concatenating 3D CNN features and lung radiomics features for COPD stage classifcation based on the MLP classifer.First, the 3D CNN features are extracted from the lung region images based on a truncated transfer learning strategy.Ten, the lung radiomics features are extracted from the lung region images by PyRadiomics.Compared with CNN models and other ML classifers, the MLP classifer with the best classifcation performance is determined by the 3D CNN features and the lung radiomics features.Lasso plays a role in improving the classifcation performance of the MLP classifer with the CNN feature vector and the radiomics feature vector.Te proposed combined feature vector also improves the MLP classifer's performance.Te MLP classifer with the proposed combined feature vector has accuracy, mean precision, mean recall, mean F1score, and AUC of 0.879, 0.879, 0.879, 0.875, and 0.971, respectively.Tis shows that our method efectively improves the classifcation performance for COPD stage classifcation.

Figure 2 :
Figure 2: Te proposed method in this study.(a) A constructed model of 3D CNN and MLP classifer for COPD stage classifcation.(b) CNN feature vector is generated by transfer learning from Med3D.(c) Te combined feature vector is generated by concatenating the CNN feature vector and the radiomics feature vector.(d) Te combined feature vector is used to classify the COPD stage based on MLP classifer.

Figure 1 :
Figure 1: COPD stage distribution of the subjects in this study.

Figure 3 :
Figure 3: Experimental design in this paper.

Figure 4 :
Figure 4: Dataset division in this paper.(a) Training set.(b) Test set.

Figure 6 :
Figure 6: Te ROC curves of the CNN feature vector and lung radiomics vector are based on diferent classifers.(a) Te ROC curves of the CNN feature vector (13824).(b) Te ROC curves of the CNN feature vector selected by Lasso (60).(c) Te ROC curves of the lung radiomics vector (1316).(d) Te ROC curves of the lung radiomics vector selected by Lasso (106).

Figure 7 :
Figure 7: Te confusion matrix and ROC curves of the MLP classifer with diferent feature vectors based on 3D ResNet10.(a) Te confusion matrix of the MLP classifer with CNN feature vector selected by Lasso (60).(b) Te confusion matrix of the MLP classifer with radiomics feature vector selected by Lasso (106).(c) Te confusion matrix of the MLP classifer with combined feature vector (166).(d) Te ROC curves of the MLP classifer with these feature vectors.

Figure 8 :
Figure 8: Te confusion matrix and ROC curves of the MLP classifer with combined feature vectors based on diferent 3D ResNet.(a) Te confusion matrix of the MLP classifer with combined feature vector based on 3D ResNet10.(b) Te confusion matrix of the MLP classifer with combined feature vector based on 3D ResNet18.(c) Te confusion matrix of the MLP classifer with combined feature vector based on 3D ResNet34.(d) Te ROC curves of the MLP classifer with combined feature vectors based on 3D ResNet.

Table 8
can be calculated from the confusion matrix.Table8reports that the proposed combined feature

Table 5 :
Te diferent classifers' performances based on CNN feature vector selected by Lasso (60) in experiment 3.

Table 7 :
Te diferent classifers' performances based on the radiomics feature vector selected by Lasso (106) in experiment 4.

Table 9 :
3D ResNet's performance based on MLP classifer with the combined feature vector (166).Because of this, the MLP classifer is suitable for classifying the COPD stage and has achieved an excellent result in COPD stage classifcation.