In the process of prenatal ultrasound diagnosis, accurate identification of fetal facial ultrasound standard plane (FFUSP) is essential for accurate facial deformity detection and disease screening, such as cleft lip and palate detection and Down syndrome screening check. However, the traditional method of obtaining standard planes is manual screening by doctors. Due to different levels of doctors, this method often leads to large errors in the results. Therefore, in this study, we propose a texture feature fusion method (LH-SVM) for automatic recognition and classification of FFUSP. First, extract image’s texture features, including Local Binary Pattern (LBP) and Histogram of Oriented Gradient (HOG), then perform feature fusion, and finally adopt Support Vector Machine (SVM) for predictive classification. In our study, we used fetal facial ultrasound images from 20 to 24 weeks of gestation as experimental data for a total of 943 standard plane images (221 ocular axial planes, 298 median sagittal planes, 424 nasolabial coronal planes, and 350 nonstandard planes, OAP, MSP, NCP, N-SP). Based on this data set, we performed five-fold cross-validation. The final test results show that the accuracy rate of the proposed method for FFUSP classification is 94.67%, the average precision rate is 94.27%, the average recall rate is 93.88%, and the average
Ultrasound has been used for prenatal observation, measurement, and diagnosis of fetal diseases for nearly 30 years due to its advantages of low cost, portability, no radiation, and real-time imaging capabilities. Historical experience and the majority of facts show that ultrasound diagnosis is very safe and effective [
The standard planes of fetal ultrasound play a decisive role in understanding fetal anatomy and tissue development [
The FFUSP consists of three elemental planes (Figure
Image of FFUSP (a) OAP, where CL represents the crystalline lens and EB represents the eyeball; (b) MSP, where FB represents the frontal bone, NB represents the nasal bone, AN represents the apex nasi, and LJ represents the lower jawbone; (c) NCP, where AN represents the apex nasi, NC represents the nasal column, Nos represents the nostril, UL represents the upper lip, LL for lower lip, and MD for the mandible.
Although the professional skills of obstetricians have been greatly improved with the popularity of prenatal ultrasound diagnosis and standardized training of ultrasound doctors in recent years, there are still some factors affecting the fetal ultrasound in the daily ultrasound work, such as the influence of the resolution of ultrasound equipment, the experience, concentration, energy, and sense of responsibility of the ultrasound doctors. This study is aimed at improving the recognition and classification efficiency of standard planes of fetal facial ultrasound and reducing the impact of human factors on the quality of fetal ultrasound. From the perspective of how to identify and obtain various types of standard ultrasonic planes of fetal facial, we should take measures to minimize the dependence of obtaining standard ultrasonic planes on ultrasonic doctors’ qualifications and the influence of different ultrasonic devices to improve the efficiency of prenatal ultrasonic examination.
In the prenatal ultrasound examination, many types of planes need to be used, and doctors usually acquire the standard fetal ultrasound planes manually. Because it is challenging to acquire the fetal ultrasound planes, and there are differences among different ultrasound doctors in clinical work experience, as well as different levels of cognition on the anatomical structure and characteristics of fetal planes, there are problems of small interclass differences and large intraclass differences among the obtained various planes [
With the application of artificial intelligence (AI) in various fields, AI has made outstanding achievements in medical image recognition and analysis in recent years. The primary research to realize AI and ultrasonic scanning mainly focus on the automatic or semiautomatic identification and classification methods of ultrasonic standard planes in different parts. The challenges are as follows: first, the imaging principle of ultrasonic images makes ultrasonic images have high noise and low contrast [
Image recognition and classification based on traditional manual features are mainly divided into three steps: feature extraction, feature coding, and feature classification [
After 2012, deep learning (DL) began to emerge, and automatic recognition and classification technology based on deep learning was gradually introduced into the task of automatic recognition and classification of the standard ultrasonic plane. The deep learning method is mainly divided into two steps: first, the image is trained by the depth network model, the depth features of the image are extracted, and then the trained depth network is used to identify or classify the image. In 2014, Chen et al. [
The above work has achieved good results in the corresponding research fields. Still, there are also one or more shortcomings, such as
The research method is low in universality and not suitable for positioning other types of fetal standard planes The adopted method needs manual intervention and has a low automation level and limited clinical practical value Due to the model’s defects, the accuracy of standard plane positioning is easily affected by accumulated errors The convolutional neural network model is challenging to train, complicated in-process, and slow in operation
Given the current research status of ultrasonic planes of fetal facial, and considering the characteristics of FFUSP, that is, the number of standard planes is small, and the characteristics of the three types of standard planes are quite different, we propose an ultrasonic standard plane recognition and classification method that is relatively simple in process, fast in operation speed, and suitable for other parts of the fetus. In this study, a method based on image texture feature fusion and Support Vector Machine was used to identify and classify the prenatal FFUSP. This proposed method was evaluated in terms of classification accuracy, precision, recall, and
Process flow chart of this proposed method.
This study was approved and audited by the Ethics Committee of School of Medicine, Huaqiao University, and all the relevant topics were notified of approval. The data of three types of standard ultrasound planes (OAP, MSP, and NCP) and the nonstandard plane (N-SP) of fetal facial involved in the experiment were provided by Three Grade A hospitals (Quanzhou First Hospital Affiliated to Fujian Medical University). With the pregnant women’s permission under examination, the professional ultrasound doctors recorded and saved the ultrasound scanning video through Philips EPIQ5 ultrasound instrument and GE Volusen E8 ultrasound instrument and further screened the pictures in the scanning video to ensure the accuracy of the experimental data to the greatest extent.
Image inclusion criteria:
The image was clear, and the target structure located in the center of the image accounted for more than 1/2 of the whole image. The background was pure and free of artifacts No superimposed color flow image in the image, no measurement caliper, text identification, and other artificial comments Postpartum confirmed fetal without facial and other structural abnormalities
Image exclusion criteria:
The images were blurred and smeared due to the obesity of pregnant women, image jitter, and other reasons. The target structure was not displayed Ultrasound or postpartum confirmed fetal abnormalities
Finally, 943 pieces of data from the three types of standard planes and 350 pieces of data from nonstandard planes of fetal facial ultrasound were added to the experiment. The data proportion distribution of the four types of planes and the number of data sets randomly divided by five-fold cross-validation are shown in Table
Data distribution in this lab set.
Class | Total | 1 | 2 | 3 | 4 | 5 |
---|---|---|---|---|---|---|
OAP | 221 | 45 | 45 | 45 | 45 | 41 |
MSP | 298 | 60 | 60 | 60 | 60 | 58 |
NCP | 424 | 85 | 85 | 85 | 85 | 84 |
N-SP | 350 | 70 | 70 | 70 | 70 | 70 |
Figure
As all experimental data in this study were obtained from the ultrasonic scanning video of fetal facial, the nonstandard planes shown in Figure
Image of N-SP. (a) Images similar to a standard plane shape. (b) Other forms of images.
This study is aimed at realizing the recognition and classification of the standard ultrasound planes of fetal facial based on the simple process and fast operation of the experimental method model. In this study, Local Binary Pattern (LBP) [
In the original ultrasound images, in addition to the speckle noise inherent in the ultrasound images and the differences between images caused by different instruments, the different sizes of images, the shooting angles of the images, and the scaling of vital anatomical structures can also interfere with the judgment of the standard plane. We cut out the target pictures from the original experimental pictures by customizing the edge detection in the collected original experimental pictures to solve this problem. The picture obtained in the step has the advantages: (1) subject information in the picture is eliminated; (2) the vital anatomical structure is more prominent. Further, given the speckle noise inherent in the image and the difference in the gray distribution of different pictures, we perform the gray normalization on the target picture, which effectively balances the image’s gray distribution and minimizes the image distortion. After pretreatment, the picture’s size finally added into the experiment was further reduced to
In our experiments, the specific calculation formula is as follows:
Where
For each central pixel, the idea of HOG is to convolute the image with gradient operators
In the phase of texture feature extraction, we divide the target image into
Texture feature fusion schematic diagram.
In the texture feature extraction step, the cells of LBP and HOG are two-element vectors specified in pixel units. To capture large-scale spatial information, we can appropriately increase the cell size, but it may lose small-scale details while increasing the cell size. Therefore, we defined the LBP and HOG cell size (CellSize) in the experiment as [72,72] through parameter optimization. Due to the diversity of images, the normalization of feature vectors can effectively remove background information and improve the image recognition rate. At the same time, normalization makes the feature vectors between different dimensions have a certain numerical comparison, which greatly improves the accuracy of classification. This step is a routine step in the process of extracting texture features. Performing L2 normalization on the histogram corresponding to each cell unit, and reshaping the LBP feature vector and the HOG feature vector obtained from each picture to form
The Support Vector Machine’s (SVM) [
The specific hardware configuration of the computer equipment used in this experiment is as follows: Intel(R) Core (TM) i7-7700 is used for CPU, NVIDIA GeForce GTX-1080Ti is used for GPU, and the video memory is 11G and the memory is 32 G. The computer’s operating system is 64-bit Windows 10, and the programming software is MATLAB R2018b.
This paper evaluates the model by calculating the precision, recall,
In the formula, TP means the number of positive cases predicted as positive cases, FP means the number of negative cases predicted as positive cases, TN means the number of positive cases predicted as negative cases, and FN means the number of negative cases predicted as negative cases.
Through the experimental process in Figure
The results of this experimental method.
Method | Group | Precision (%) | Recall (%) | Accuracy (%) | |
---|---|---|---|---|---|
The proposed (LH-SVM) | A | 97.44 | 97.17 | 97.30 | 97.31 |
B | 93.74 | 93.10 | 93.38 | 94.23 | |
C | 92.79 | 91.58 | 92.06 | 93.08 | |
D | 92.02 | 91.71 | 91.85 | 92.69 | |
E | 95.39 | 95.82 | 95.54 | 96.05 | |
AVG | 94.27 | 93.88 | 94.08 | 94.67 |
To further illustrate the advantages of choosing the fusion of LBP [
Comparative experimental results of different texture features and classifiers.
Methods | AVG-Pre (%) | AVG-Re (%) | AVG- | Accuracy (%) | |
---|---|---|---|---|---|
Texture | Classifier | ||||
LBP | SVM | 93.45 ( | 93.15 ( | 93.25 ( | 93.97 ( |
HOG | SVM | 89.87 ( | 89.22 ( | 89.45 ( | 90.72 ( |
LBP + HOG | SVM | 94.27 ( | 93.88 ( | 94.03 ( | 94.67 ( |
LBP | KNN | 88.96 ( | 87.08 ( | 87.66 ( | 89.33 ( |
HOG | KNN | 89.31 ( | 88.07 ( | 88.42 ( | 89.78 ( |
LBP + HOG | KNN | 90.32 ( | 89.77 ( | 89.95 ( | 90.87 ( |
LBP | NB | 70.29 ( | 70.65 ( | 69.91 ( | 72.68 ( |
HOG | NB | 73.73 ( | 73.17 ( | 73.25 ( | 76.33 ( |
LBP + HOG | NB | 78.08 ( | 77.34 ( | 77.24 ( | 79.81 ( |
It can be seen from Table
After the effect of a single texture feature on the experimental results is verified, we further explore the effect of different classifiers on the experiment’s efficiency. In this stage, we introduce the
The data shown in Table
In our experiment, the number of neighbors corresponding to
Comparative experimental results of different (
( | AVG-Pre (%) | AVG-Re (%) | AVG- | Accuracy (%) |
---|---|---|---|---|
(8, 1) | 94.27 (±3.17) | 93.88 (±3.29) | 94.03 (±3.27) | 94.67 (±2.64) |
(16, 2) | 93.72 (±2.52) | 93.40 (±2.37) | 93.52 (±2.44) | 94.20 (±1.95) |
(24, 3) | 92.37 (±2.20) | 91.98 (±1.68) | 92.11 (±1.81) | 93.04 (±1.58) |
94.07 (±2.08) | 93.60 (±2.17) | 93.77 (±2.18) | 94.52 (±1.63) | |
93.90 (±2.92) | 93.64 (±2.77) | 93.74 (±2.82) | 94.44 (±2.52) | |
93.52 (±2.35) | 93.23 (±1.98) | 93.31 (±2.16) | 94.13 (±1.64) |
Looking at Table
In the texture feature extraction stage, we need to divide the target image into cells to access the histograms on each cell, and the size of the cell will directly affect the formation of feature vectors, thus, affecting the image recognition and classification results. In the experimental data shown in this paper (Table
If the cell size is changed, will the experimental results show serious deviation and directly indicate that the model’s performance is not excellent? To verify this problem’s existence, we compared the experimental results corresponding to each group of parameters in the parameter optimization process. The average accuracy of the five-fold cross-validation experiment under 41 groups of parameters is in the range of 92.73%-94.67%. The average precision is in the range of 92.25%-94.34%, the average recall is in the range of 91.65%-93.87%, and the average
Scatter plot of experimental results corresponding to different cell sizes.
We can conclude that the setting of cell unit size affects the experimental results to a certain extent. Still, it is not the most crucial factor that affects the classification effect of the FFUSP using texture features in this study. The method used in this study has a certain stability.
Prenatal ultrasound is one of the essential means to screen for fetal abnormalities. Clinically, doctors have found that 32–39 classes of planes of the fetus [
The experimental results show that the traditional method of texture feature fusion with mainstream classifier can effectively and automatically identify and classify FFUSP images. In particular, for the recognition and classification problems involving fewer categories, the traditional texture features largely overcome the difficulties in training the convolutional neural network model, the complexity of the process, the slow operation, and other problems. In this paper, the fusion of LBP and HOG and the adoption of SVM recognition and classification have achieved excellent results.
In the process of predicting and classifying the ultrasonic planes of fetal facial by this research method, we performed index transformation
Prediction and classification process of FFUSP.
Experimental data involved in this paper were obtained from the fetal ultrasound scanning video. The added nonstandard planes included images similar to the standard plane morphology and other morphological images, which increased the difficulty in identifying the standard plane and interfered with the classification experiment in this study. However, the experimental data used in this study, to a certain extent, represent the real-time images obtained by ultrasonic scanning. The composition of the experimental data in this study and the final experimental results indicates the possibility of real-time detection of FFUSP by this method, further reflecting this method’s clinical potential.
It is concluded from the experiment that the accuracy of the proposed method for the classification of FFUSP is 94.67%, indicating that the texture feature fusion method can effectively predict and classify FFUSP, which provides an essential basis for clinical research on the automatic detection method of FFUSP. At the same time, there are still some shortcomings in this study. First, although the method used in this study performs well in the classification of FFUSP, it can only be used for rough classification. It cannot identify the specific anatomical structures in the standard planes. Second, the method used in this study still misclassifies some ultrasound images. Third, the experimental data in this study were all from pregnant women with a healthy fetus, and the recognition challenge of the standard plane was relatively small. Fourth, this study’s experimental data volume is relatively small, so the proposed method cannot be compared with the deep learning model with the same data set.
In the next stage of our work, we will strive to overcome the above shortcomings. First of all, we will continue to establish a relatively standardized and sizeable ultrasound image database. We can compare and evaluate the performance of more different method models. Further, efforts are made to detect the standard plane’s fundamental anatomical structures; we will use the similarity of plane prediction as the breakthrough point to identify and classify the images through the standard plane quality control. Fetal ultrasound images of more different pathological cases will be collected on experimental data. Besides, we will look for more effective ways to overcome the image differences caused by different ultrasonic instruments and different scanning techniques to pass the external examination as soon as possible.
To solve the problem that the traditional method of obtaining FFUSP is highly dependent on the doctor’s seniority, energy, and other aspects, and save time and human resources; in this study, we used the fusion of LBP and HOG to extract the texture features of the image and SVM classifier for recognition and classification to achieve the rapid classification of FFUSP. We first collected a certain amount of data on standard and nonstandard planes of ultrasound of the fetal face. A senior sonographer strictly screened each ultrasound image. In this experiment, we have obtained the FFUSP recognition accuracy of 94.67%. To verify the stability of the experimental method, we performed experiments under different parameters. The results showed that the experimental method could still achieve excellent results under different parameters. The results obtained under 41 groups of parameters were stable. Besides, with the addition of nonstandard planes, which were very similar to the standard planes, the experimental results were still significant, which strongly verified this experimental method’s clinical application potential. The proposal of the concept of prediction similarity lays the foundation for the next stage of work. The experimental results showed that this research method was a very effective method for the classification of FFUSP. It could further effectively solve the problem of the dependence of clinical acquisition of FFUSP on the doctor’s seniority, energy, and other subjective factors.
The Fetal Facial Ultrasound Image data used to support the findings of this study were supplied by Quanzhou first hospital in Fujian, China, under license and so cannot be made freely available.
The authors declare that there is no conflict of interest regarding the publication of this paper.
This work was supported by Quanzhou scientific and technological planning projects (Nos. 2019C029R and 2018N042S) and the grants from Fujian Provincial Science and Technology Major Project (No. 2020HZ02014) and Promotion Program for Young and Middle-aged Teacher in Science and Technology Research of Huaqiao University (No. ZQN-PY518).