Body constitution classification is the basis and core content of traditional Chinese medicine constitution research. It is to extract the relevant laws from the complex constitution phenomenon and finally build the constitution classification system. Traditional identification methods have the disadvantages of inefficiency and low accuracy, for instance, questionnaires. This paper proposed a body constitution recognition algorithm based on deep convolutional neural network, which can classify individual constitution types according to face images. The proposed model first uses the convolutional neural network to extract the features of face image and then combines the extracted features with the color features. Finally, the fusion features are input to the Softmax classifier to get the classification result. Different comparison experiments show that the algorithm proposed in this paper can achieve the accuracy of 65.29% about the constitution classification. And its performance was accepted by Chinese medicine practitioners.
Traditional Chinese medicine (TCM) constitution theory originated in the “Yellow Emperor” <黄
Constitution classification is the basis and core content of TCM constitution research. It is to extract the relevant laws from the complex constitution phenomenon and finally build the constitution classification system. The commonly used constitution types are determined by the traditional Chinese medicine constitutional questionnaire which is developed by Wang [ It is influenced by subjective factors. At present, the method of constitution classification is mainly through the determination of the constitution measurement table, and it can be said that the basic diagnosis is based on the consultation of Chinese medicine four clinics. There are a large part of the subjective factors, such as the accuracy of the collected data, the respondents’ experience, and the degree of understanding of the respondents. The number of questions is too large and takes a long time, so that many respondents lose their patience in the process of filling out the constitution measurement table; then these elements have an impact on constitution classification. Score calculation formula is more complex, so that many people cannot accurately calculate their constitution type.
It is necessary to develop computer technology to standardize and objectify the constitutional diagnosis method in order to solve the above problems. At present, many scholars have applied biology knowledge and machine learning algorithm to TCM diagnosis process [
Therefore, this paper presents the deep convolutional neural networks for classifying body constitution based on face image. The second section will introduce the collected face dataset, the convolution neural network algorithm, and some commonly used pattern recognition algorithms. The evaluation procedure, the obtained experimental results, and discussion are presented in Section
The algorithm proposed in this paper is divided into four main parts: (
The flow chart of the whole algorithm.
There are 5330 face images used in this article. The face dataset is collected by capturing the patient’s face picture in the three hospitals of the Chinese medicine outpatient department, respectively. The type of body constitution is judged by a medical professor in each TCM outpatient room. The judgment is based on the standard of classification and determination of constitution in TCM which is developed by Professor Wang [
On the other hand, it is proved in practice that the reliability and validity of the diagnosis by these Chinese medicine professors are better than those of the questionnaire survey method for the body constitution identification by Wang’s questionnaire (CCMQ). Now CCMQ systems have been deployed in many hospitals. The survey shows that the actual utilization rate is not high, and the main reason is from the patient’s subjective problem, instead of CCMQ standard itself. The subjective factors of the patients are mainly influenced in three aspects. (
Therefore, all face images are taken by the same type of digital device and the patient’s physical type is specified by the doctor. The indoor environment is no sunshine, and lighting conditions are normal fluorescent lamps. In the face database, there are 8 kinds of constitution types, that is, gentleness, Qi-deficiency, Qi-depression, dampness-heat, phlegm-dampness, blood-stasis, Yang-deficiency, and Yin-deficiency. The number of each constitutional type is shown in Table
The number of samples of different constitution types.
Gentleness | Qi-deficiency | Yang-deficiency | Yin-deficiency | Phlegm-dampness | Dampness-heat | Blood-stasis | Qi-depression | Sum | |
---|---|---|---|---|---|---|---|---|---|
Number |
|
|
|
|
|
|
|
|
|
Traditional Chinese medicine (TCM) is based on more than 2,500 years of Chinese medical practice. The diagnostic principle of traditional Chinese medicine is based on information obtained from four diagnostic procedures, namely, diagnosis through observation, diagnosis through auscultation and olfaction, diagnosis through inquiry, and diagnosis through pulse feeling and palpation [
The color is a very important visual feature of the image. Compared with other features, the color feature is not sensitive to the translation, scale, and rotation of the image, and it is very robust and simple to calculate. In this paper, we use the method of color histogram based on HSV color space to extract the color feature. From the psychological perception of people, the HSV space is more intuitive and easier to accept compared with the RGB space [
The convolution neural network (CNN) [
The convolutional neural network is a feature-based method and applied to physical recognition. It is different from the traditional artificial feature extraction and the high performance classifier design for the feature. Its advantage is that the feature extraction is carried out by layer-by-layer convolution and dimensionality. And then through the multilayer nonlinear mapping, the network can automatically learn to form the identification task for the feature extractor and classifier from the training sample. The method reduces the requirement of the training sample, and the more the network layer is, the more the characteristic of the learning is more global.
This paper is inspired by the literature [
The structure of convolutional neural networks for extracting features.
Pattern classification can be carried out to classify the faces into different types by the features, such as color features, texture features, and features extracted by the CNN model. There are many algorithms in pattern classification, such as Naive Bayes classifier [
In this section, we conducted a series of experiments to measure the effectiveness of the body constitution recognition algorithm. The details of these experiments are described below.
The tools used in this experiment are based on Keras and Scikit-learn [
In this paper, we first extract the color and texture features and the features extracted by the convolution neural network and then compare the classification effect of the feature extraction method under a classification algorithm. Among them, the support vector machine in the kernel function is to select RBF, and the value of
The classification results under different feature extraction methods.
SVM | Random Forest | KNN | Softmax | Decision Tree | Gradient BoostTree | Naive Bayes | |
---|---|---|---|---|---|---|---|
Color feature |
|
|
|
|
|
|
|
Color and texture features |
|
|
|
|
|
|
|
CNN |
|
|
|
|
|
|
|
The confusion matrix of random forest classification based on color texture feature Fusion.
Qi-deficiency | Yin-deficiency | Yang-deficiency | Phlegm-dampness | Dampness-heat | Qi-depression | Blood-stasis | Gentleness | |
---|---|---|---|---|---|---|---|---|
Qi-deficiency |
|
|
|
|
|
|
|
|
Yin-deficiency |
|
|
|
|
|
|
|
|
Yang-deficiency |
|
|
|
|
|
|
|
|
Phlegm-dampness |
|
|
|
|
|
|
|
|
Dampness-heat |
|
|
|
|
|
|
|
|
Qi-depression |
|
|
|
|
|
|
|
|
Blood-stasis |
|
|
|
|
|
|
|
|
Gentleness |
|
|
|
|
|
|
|
|
The confusion matrix of Softmax classification based on convolutional neural network.
Qi-deficiency | Yin-deficiency | Yang-deficiency | Phlegm-dampness | Dampness-heat | Qi-depression | Blood-stasis | Gentleness | |
---|---|---|---|---|---|---|---|---|
Qi-deficiency |
|
|
|
|
|
|
|
|
Yin-deficiency |
|
|
|
|
|
|
|
|
Yang-deficiency |
|
|
|
|
|
|
|
|
Phlegm-dampness |
|
|
|
|
|
|
|
|
Dampness-heat |
|
|
|
|
|
|
|
|
Qi-depression |
|
|
|
|
|
|
|
|
Blood-stasis |
|
|
|
|
|
|
|
|
Gentleness |
|
|
|
|
|
|
|
|
The color is a basis for judgment in the diagnosis through observation. Therefore, this paper proposes a method of combining the features of the convolution neural network and the color features, and the classification results are shown in Table
The classification results based on the convolution neural network feature extraction and color feature fusion.
SVM | Random Forest | KNN | Softmax | Decision Tree | Gradient BoostTree | Naive Bayes | |
---|---|---|---|---|---|---|---|
CNN |
|
|
|
|
|
|
|
CNN + color |
|
|
|
|
|
|
|
The confusion matrix of Softmax classification based on convolutional neural network and color feature fusion.
Qi-deficiency | Yin-deficiency | Yang-deficiency | Phlegm-dampness | Dampness-heat | Qi-depression | Blood-stasis | Gentleness | |
---|---|---|---|---|---|---|---|---|
Qi-deficiency |
|
|
|
|
|
|
|
|
Yin-deficiency |
|
|
|
|
|
|
|
|
Yang-deficiency |
|
|
|
|
|
|
|
|
Phlegm-dampness |
|
|
|
|
|
|
|
|
Dampness-heat |
|
|
|
|
|
|
|
|
Qi-depression |
|
|
|
|
|
|
|
|
Blood-stasis |
|
|
|
|
|
|
|
|
Gentleness |
|
|
|
|
|
|
|
|
The ROC curves are typically used in binary classification to study the output of a classifier. The top left corner of the plot is the “ideal” point—a false positive rate of zero and a true positive rate of one. This is not very realistic, but it does mean that a larger area under the curve (AUC) is usually better. The ROC curve of different classifiers based on the feature of convolution neural network and color feature fusion is in Figure
The ROC curve of different classifiers based on the feature of convolution neural network and color feature fusion. The dotted black line is the baseline in ROC curve. It indicates that the true positive rate (TPR) is equal to the false positive rate (FPR).
The precision-recall curve of different classifiers based on the feature of convolution neural network and color feature fusion.
The micro-average and macro-average ROC curve in the Softmax based on the convolution neural network and the color feature fusion. The dotted black line is the baseline in ROC curve. It indicates that the true positive rate (TPR) is equal to the false positive rate (FPR).
The ROC curve of each label in the Softmax based on the convolution neural network and the color feature fusion. The dotted black line is the baseline in ROC curve. It indicates that the true positive rate (TPR) is equal to the false positive rate (FPR).
We have done the data increment experiment on the existing dataset in this paper. In each dataset, we select 90% of the data as a training set and the remaining 10% as a test set. Under the premise of the feature of convolutional neural network and color feature fusion, the accuracy rate is gradually increasing in the same classifier with the increase of data. The experimental results are shown in Table
The classification results with the increase of data.
SVM | Random Forest | KNN | Naive Bayes | Softmax | Decision Tree | Gradient BoostTree | |
---|---|---|---|---|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Since the experiment of our method is based on the standard dataset whose labels were judged by the experts and the accuracy in the eight categories is 65% which is far greater than the random assignment, it indicates that there is a consistency between the experts. Secondly, the experimental process and accuracy results were reported to the experts. They believe that the proposed method is useful for practical applications. Finally, we organized a small number of volunteers to compare the body constitution recognition result of our method and that of experts, and the consistence almost keeps the same level, showing that the judgment between the different medical experts can be consistent. However, due to the small size of volunteers, the results may be influenced by random so that the large-scale contrast testing between system and experts will be expected in the future.
This paper presented a constitution classification algorithm based on convolutional neural networks. Our approach uses convolutional neural networks to extract the features of face images. We have also presented a set of experiments aiming to validate our algorithm. First of all, the feature extraction method of convolution neural network is better than the color and texture features. Then, under the premise of convolution neural network feature and color feature fusion, the classification of Softmax is the best by comparing different classifiers. At last, the results show that our method obtained the best results with a precision of 65.29%. As the results of the body constitution identification by CCMQ are easily influenced by the subjective factors of patients, our approach can classify body constitution faster and more accurately.
The study has shown that convolutional neural networks are effective in dealing with constitution classification based on face image. In addition, the study will serve as a reference for establishing diagnostic criteria and a diagnostic model for constitution classification and a better guide for clinical practice.
The authors declare that there are no conflicts of interest regarding the publication of this paper.
This study was supported by a China National Science Foundation under Grants 60973083 and 61273363, Science and Technology Planning Projects of Guangdong Province (2014A010103009, 2015A020217002), and Guangzhou Science and Technology Planning Project (201504291154480).