Complexity
Volume 2021 (2021), Article ID 5597168, 9 pages
https://doi.org/10.1155/2021/5597168

Research Article

Multidirection Object Detection in Aerial View of Traffic Target under Complex Scenes

Zeqing Zhang,1,2 Weiwei Lin,3,4 and Yuqiang Zheng¹

¹School of Information Science and Engineering, Xiamen University, Xiamen, China
²West Yunnan University of Applied Sciences, Dali, China
³School of Electronic and Information Engineering, Fujian Polytechnic Normal University, Fuqing, China
⁴Engineering Research Center for ICH Digitalization and Multi-source Information Fusion, Fujian Province University, Fuqing, China

Correspondence should be addressed to Yuqiang Zheng; zhengyuqiang@stu.xmu.edu.cn

Received 30 January 2021; Revised 21 February 2021; Accepted 31 March 2021; Published 12 April 2021

Academic Editor: Abd E.I.-Baset Hassanien

Copyright © 2021 Zeqing Zhang et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

Focusing on DOTA, the multidirectional object dataset in aerial view of vehicles, CMDTD has been proposed. The reason why it is difficult for applying the general object detection algorithm in multidirectional object detection has been analyzed in this paper. Based on this, the detection principle of CMDTD including its backbone network and multidirectional multi-information detection end module has been studied. In addition, in view of the complexity of the scene faced by aerial view of vehicles, a unique data expansion method is proposed. At last, three datasets have been experimented using the CMDTD algorithm, proving that the cascaded multidirectional object detection algorithm with high effectiveness is superior to other methods.

1. Introduction

With the development of deep learning, rapid progress has been made in remote sensing or aerial image processing and analysis [1–5]. However, these methods cannot deal with multidirectional object detection. Different from the traditional object detection, which the detection frame is generally horizontal or vertical giant, the detection frame given by multidirectional object detection can be rectangular in any direction. Recently, a plurality of popular general object detection algorithms has been ineffective on vehicle object detection datasets with the best result reaching only 52.93%. Compared with the experimental result on the DOTA dataset [6], the rotation of the detection object boundary box, a large number of small vehicle objects in aerial view of images, and insufficient utilization of data information are main causes for the poor performance.

To improve the effect of multidirectional aerial view of the vehicle object detection dataset, the CMDTD (a cascaded multidirectional object detection algorithm) algorithm has been utilized with FasterRCNN as the benchmark method through using the cascade idea of Cascade RCNN for reference. Moreover, vehicle objects are classified in a coarse-to-fine manner for fitting boundary prediction via a multi-information cascade output end. Data augmentation training is performed on the classes with few samples in statistical data in order to address the problem of sample imbalance. By using the method, the effect of multidirectional object detection can be effectively improved. Hence, it has certain research value for implementing multidirectional vehicle object detection in complex scenes.

2. Research Models

2.1. ResNeXt Backbone Network

Compared with land vehicles, although there are few classes of aerial vehicles objects, big differences between the classes can be witnessed. In order to extract superior object features, ResNeXt [7], which is more powerful than ResNet, is selected in the backbone network of CMDTD with the adoption of the feature fusion method of FPN [8]. The submodule structures of ResNet and ResNeXt are shown in Figure 1. The submodule of ResNet is composed of three convolutional layers. Regarding the input feature map X, X will extract new features, i.e., F(X) upon three convolution operations. And then, residual blocks are output upon the superimposition of new and old features. The process can be expressed as

Figure 1: Comparison of residual block structures of ResNet and ResNeXt.

According to equation (1), the feature obtained by F(X) is the difference between Y and X, which is also called residual characteristic, and the submodule is called residual block. Based on ResNet, the residual was expanded using ResNeXt.

As shown in Figure 1, three convolutional layers are divided into 32 groups of convolutional combinations with consistent parameter sizes, and the sum is equal to the size of the three original convolutional layers. The feature map X is superimposed, or , after 32 groups of convolution features. On this basis, new features are superimposed with the old features for acquiring output features, which is shown as

Although parameters are not increased in the new combination, the complexity of feature transformation is increased, strengthening the expressive ability of network towards features.

2.2. Multi-Information Cascade Output

The output end of the general detector and the multidirection and multi-information detection output end proposed in this paper are demonstrated in Figure 2. Based on Figure 2(a), the output end of the general detector should undergo a class determination and border prediction after acquiring the region proposal in RPN (regional proposal network). Being consistent with the output end of the general detector in the first half part, the multi-information cascade output end proposed in this paper will regress to the horizontal outer border of the object. However, it only determines whether the object is foreground or background in the second stage of class determination. Moreover, the length and width information (length, width, and aspect ratio) of the object can be calculated upon obtaining the horizontal frame of the object. Next, RoI pooling performs a new object feature extraction based on the acquired horizontal frame. Finally, a fine classification of the object is performed in the second FCN (fully connected network) based on the length and width information of the object and extracted features. At the same time, the third FCN predicts the positions of four vertices of the object according to the extracted features, acquiring the quadrilateral bounding box of the object.

(a)

(b)

Figure 2: Comparison of general detection end and multidirectional multi-information detection end: (a) the general detection end; (b) the multidirection and multi-information detection end.

Compared with the detection of land vehicles, sizes of objects in aerial view of vehicles are diverse, causing that it becomes more difficult to regress the positions. Preciser object boundary can be obtained in the second stage in comparison to the object proposal region, contributing to the boundary positioning in the future. Regarding class determination, objects of different classes may have the same texture and color features yet in varied sizes. For example, small vehicles and large vehicles with similar color characteristics can be distinguished as per the size and aspect ratio of the object. Hence, introducing the aspect ratio information of the object during the fine classification of the object might improve the accuracy of classification. During implementation, the length, width, and aspect ratio calculated by the horizontal frame can be obtained at the detection end, which can be introduced to the FCN together with features upon reducing by 1000 times.

During training, the end is composed of four losses, including two position losses and two class losses. The position loss is the prediction losses of horizontal boundary and the vertex, while the class loss is the foreground classification loss and the fine classification loss. Among them, L1 smoothing loss and cross entropy loss are applied in the position loss and the classification loss, respectively.

2.3. Prediction of Vertex Information

The positioning object boundary at the detection end is presented in Figure 3. In this paper, the object box has been regressed for three times at the detection end. It is performed in chronological order, namely, object candidate box, object horizontal bounding box, and rotating rectangular box, which are corresponding to the dashed box in light blue, the dotted box in blue, and the rectangular box in green.

Figure 3: Cascading boundary regression process.

The dashed box in light blue is far away from the real boundary of the object. It is because the region proposal network regresses to the object boundary according to the anchor. If the anchor point position deviates from the object boundary, a poor regression effect will be achieved. In the experiment, the aspect ratio of anchor is [1 : 1, 1 : 2, 2 : 1, 1 : 3, 3 : 1] with the base size of 6 pixels, so as to adapt to dense small objects and long objects. The dashed box in blue is the horizontal bounding box of regression in the second stage. This regression is approaching to the real boundary of the object as the proposed position of the object is the horizontal rectangle that is externally connected to the regression object. The rectangular box in green is obtained by regressing the four vertices of the horizontal bounding box, and the predictive equation is shown as follows:

where x, y, , and h are the center coordinates, length, and width of the horizontal predictive frame, respectively. And the fully connected network can obtain the first vertex of the object through regressing tx1 and ty1. During training, the point closest to the upper left corner of the horizontal bounding box of the object is taken as the first vertex, and the second, third, and fourth vertices are obtained in clockwise order. The real predicted amount can be calculated as

where and are the real coordinates of the vertex; and are the real predictions. And calculation is conducted with the smoothing L1 loss during training. Other coordinate points are calculated in a similar manner.

2.4. Unbalanced Data Distribution and Data Augmentation

As can be observed from Figure 4, intensive vehicles and ships can be found due to mass of scenes such as ports and parking lots in aerial view of vehicles images. In the training set after cutting, ships and vehicles have high proportions. In contrast, other classes have extremely proportions.

Figure 4: Distribution statistics of the DOTA training dataset.

Images of the 5 classes (including football field, athletic field, rugby field, baseball field, and roundabout) with the least proportions among data are expanded, so as to cope with the problem of unbalanced training data. According to the expanded process presented in Figure 5, horizontal flip, vertical flip, and simultaneously horizontal and vertical flip are performed on images. At this point, data of the class with few proportions have been expanded by 3 times. Since other classes can be witnessed in the augmented image, these classes will be augmented appropriately. And the augmented ratios of all classes are shown in Figure 6.

Figure 5: Data augmentation diagram.

Figure 6: Data augmentation chart. Augmented data ratio: ship, car, big car, airplane, port, storage tank, bridge, swimming pool, tennis court, helicopter, roundabout, baseball field, football field, athletic field, and football field.

3. Experimental Results

3.1. Experimental Dataset and Evaluation Indicators

Verification has been performed on the vehicle-based dataset: DOTA [6] and HRSC2016 [9] in this paper. Moreover, CMDTD also performs verification on the multidirectional scene and ICDAR2015 [10]. As a large-scale aerial view of the vehicle image dataset, the DOTA dataset contains 1,411 training images, 937 test images, and 458 verification images. These image sizes are ranged from 800 × 800 to 4000 × 4000, including 15 classes and 188, 282 instance objects. The dataset provides horizontal rectangle labeling and vertex labeling with two detection tasks opened. The first task is a multidirectional object detection task, and the second task is a horizontal object detection task.

HRSC2016 is a dataset of aerial view of maritime vehicle images that are originated from 6 main ports. To be specific, the numbers of training set, verification set, and test set are 436, 181, and 444, respectively, and the image size is ranged from 300 × 300 to 500 × 900. The ICDAR2015 dataset is originated from a detection task of the ICDAR 2015 Robust Reading Competition. It is a task that collects images taken in reality. Specially, 1000 out of 1500 images are training images, while the remaining 500 images are test images with the size of 720 × 1280.

To compare with other methods, CMDTD adopts the calculation standard evaluation method of mAP in DOTA and HRSC2016 and evaluates with F-measure in ICDAR2015. The index is calculated by the recall rate and precision, shown as follows:

where is generally set as 0.5.

3.2. Experimental Setup

The experimental environment configured for CMDTD is shown in Table 1.

Table 1: Configuration of experimental environment.

Operating system
Ubuntu16.04

Kernel version
4.15.0–70-generic

Processor model
Intel(R)core(TM) i7-6700K CPU @ 4.00GHz

Graphics card model
GeForce GTX 1080

Programming language
Python

Frame
Pytorch 1.0.0, mmdetection

Concerning the ICDAR2015 dataset, CMDTD is processed through cutting the image into 1088 × 1088 in a way similar to the DOTA dataset. Besides, the network input size is set to 1088 × 1088 during training, with training frequency and parameter details consistent with HRSC2016. The input size of the network is set to 720 × 1280 for testing, so that the image can be input in its original size for testing.

3.3. Result Comparison

The comparison results of five methods such as CMDTD and RRPN [11], ICN [12], and SCRDet [13] in the DOTA dataset task 1 (multidirectional detection) are displayed in Table 2. The mAP indexes of CMDTD are higher than that of other methods, reaching up to 72.81%. In addition, concerning small object detection, the first and the second positions in the detection accuracy of CMDTD belong to ships and cars with 85.5% and 69.51%, respectively.

Table 2: Comparison results of DOTA task 1 (%).

RRPN [11]
ICN [12]
Ding et al. [14]
R3Det [15]
SCRDet [13]
CMDTD

Aircraft
88.52
81.40
88.64
89.54
89.98
88.15

Baseball field
71.20
74.30
78.52
81.99
80.65
84.46

Bridge
31.66
47.70
43.44
48.46
52.09
50

Athletic field
59.30
70.30
75.92
62.52
68.36
69.98

Car
51.85
64.90
68.81
70.48
68.36
69.51

Big car
56.19
67.80
73.68
74.29
60.32
73.68

Ship
57.25
70.00
83.59
77.54
72.41
85.5

Tennis court
90.81
90.80
90.74
90.80
90.85
90.35

Rugby field
72.84
79.1
77.27
81.39
87.94
81.09

Storage tank
67.38
78.20
81.46
83.54
86.86
83.52

Football field
56.69
53.60
58.39
61.97
65.02
59.1

Roundabout
52.84
62.90
53.54
59.82
66.68
65.33

Port
53.08
67.00
62.83
65.44
66.25
71.64

Swimming pool
51.94
64.20
58.93
67.46
68.24
66.48

Helicopter
53.58
50.20
47.67
60.05
65.21
54.28

mAP
61.01
68.20
69.56
71.69
72.61
72.81

The comparison results of 5 methods in CMDTD and RFCN [6], ICN [12], and SCRDet [13] in the DOTA dataset task 2 (horizontal detection) are shown in Table 3. The detection result of CMDTD in the horizontal direction is obtained by taking the smallest external rectangle from the result of the rotating rectangle. Of which, the method proposed in this paper ranks the second with mAP reaching 73.7%, in surpass of most of the other methods.

Table 3: Comparison result (%) of DOTA task 2.

R-FCN [16]
FRH [17]
FPN [8]
ICN [12]
SCRDet [13]
CMDTD

Aircraft
79.33
80.32
88.70
90.00
90.18
87.38

Baseball field
44.26
77.55
75.10
77.70
81.88
83.96

Bridge
36.58
32.86
52.60
53.40
55.30
51.33

Athletic field
53.53
68.13
59.20
73.30
73.29
70.01

Car
39.38
53.66
69.40
73.50
72.09
70.06

Big car
34.15
52.49
78.80
65.00
77.65
74.47

Ship
47.29
50.04
84.5
78.20
78.06
85.99

Tennis court
45.66
90.41
90.60
90.80
90.91
90.36

Rugby field
47.74
75.05
81.30
79.10
82.44
71.49

Storage tank
65.84
59.59
82.60
84.80
86.39
83.60

Football field
37.92
57.00
52.50
57.20
64.53
58.26

Roundabout
44.23
49.81
62.10
62.10
63.45
65.57

Port
47.23
61.69
76.60
73.50
75.77
75.60

Swimming pool
50.64
56.46
66.30
70.20
78.21
73.31

Helicopter
34.90
41.85
60.10
58.10
60.11
54.14

MAP
47.24
60.46
72.00
72.50
75.35
73.70

The comparison results of CMDTD in 6 methods including the ICDAR2015 dataset, CTPN [18], and RRPN [11] are shown in Table 4. These methods are detection methods based on one stage or two stage. The comprehensive score of the method proposed in this paper ranks the second, reaching 83.88%. An excellent performance can be observed.

Table 4: ICDAR2015 comparison results (%).

Methods
Recall rate
Accuracy
Comprehensive scoring

CTPN [18]
51.56
74.22
60.85

RRPN [11]
82.17
73.23
77.44

EAST [19]
78.33
83.27
80.72

R2CNN [20]
79.68
85.62
82.54

FOTS RT [21]
85.95
79.83
82.78

R3Det [15]
83.54
86.43
84.96

CMDTD
80.21
87.92
83.88

The comparison results of CMDTD in 7 methods including the HRSC2016 dataset, RRD [22], and R2CNN [20] are demonstrated in Table 5. The mAP index of the method proposed in this paper ranks the top, reaching 89.68%.

Table 5: ICDAR2015 comparison results (%).

Methods
mAP

R2CNN [20]
73.07

RRPN [11]
79.08

RetinaNet-H [10]
82.89

RRD [22]
84.30

RetinaNetR [15]
89.18

RoI-Transformer [14]
86.20

R3Det [15]
89.14

CMDTD
89.68

3.4. The Influence of Different Modules on Model Results

Influences of different settings on the model results, including the influence of cascade, the influence of location information on classification [23], and the influence of data augmentation are investigated by CMDTD on DOTA task 1. The results are shown in Table 6. Experimental results are presented as follows:

Table 6: Comparison results of different module settings (%).

No cascade
No location information
No data augmentation
CMDTD

Aircraft
83.60
88.18
88.73
88.15

Baseball field
65.05
82.99
81.17
84.46

Bridge
37.09
50.13
48.87
50.00

Athletic field
66.29
71.63
71.79
69.98

Car
64.09
70.20
69.03
69.51

Big car
60.49
74.11
74.43
73.68

Ship
76.00
85.08
85.39
85.50

Tennis court
86.43
90.65
90.65
90.35

Rugby field
60.67
77.53
77.76
81.09

Storage tank
67.40
83.74
84.07
83.52

Football field
44.23
53.32
53.30
59.10

Roundabout
51.23
66.01
65.29
65.33

Port
59.55
72.02
71.80
71.64

Swimming pool
54.19
66.36
65.89
66.48

Helicopter
37.73
51.38
58.03
54.28

MAP
60.94
72.22
72.48
72.81

For the influence of data augmentation, the mAP of the model trained by CMDTD on the dataset without data augmentation achieved only 72.48%. Furthermore, the detection effects of football field, rugby field, and baseball field are reduced by 5.8%, 3.33%, and 3.29%, respectively.

Detection results of CMDTD in two aerial views of image datasets can be found in Figures 7 and 8. It can be observed from the figure that there are a large number of objects in the DOTA dataset with significant differences in size and aspect ratios. The object of the HRSC2016 dataset is long rectangle. CMDTD can effectively capture objects in various directions [24], which also presents a satisfactory detection effect on small objects [25].

Figure 7: DOTA dataset results.

Figure 8: HRSC2016 dataset results.

4. Conclusion

Firstly, an aerial view based on aerial view of vehicle image object data set is proposed to overcome the shortcomings of the General Object Detection Algorithm in aerial view. On this basis, CMDTD has been proposed. The reason why it is difficult for applying the general object detection algorithm in multidirectional object detection has been analyzed in this paper. Based on this, the detection principle of CMDTD, including its backbone network, multidirectional multi-information detection end module, and aerial view of vehicle data augmentation method has been studied. At last, three datasets have been experimented using the CMDTD algorithm, proving that the cascaded multidirectional object detection algorithm with high effectiveness is superior to other methods.

Data Availability

The data used to support the findings of this study are included within [1, 2].

Conflicts of Interest

The authors declare that they have no conflicts of interest.

References

D. Hong, L. Gao, N. Yokoya et al., “More diverse means better: multimodal deep learning meets remote-sensing imagery classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 99, pp. 1–15, 2020.

D. Hong, L. Gao, J. Yao, B. Zhang, A. Plaza, and J. Chanussot, “Graph convolutional networks for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, pp. 1–13, 2020.

D. Hong, N. Yokoya, J. Chanussot, and X. X. Zhu, “An augmented linear mixing model to address spectral variability for hyperspectral unmixing,” IEEE Transactions on Image Processing, vol. 28, no. 4, pp. 1923–1938, 2019.

X. Wu, D. Hong, J. Tian, J. Chanussot, W. Li, and R. Tao, “ORSIm detector: a novel object detection framework in optical remote sensing imagery using spatial-frequency channel features,” IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 7, pp. 5146–5158, 2019.

X. Wu, D. Hong, J. Chanussot, Y. Xu, R. Tao, and Y. Wang, “Fourier-based rotation-invariant feature boosting: an efficient framework for geospatial object detection,” IEEE Geoscience and Remote Sensing Letters, vol. 17, no. 2, pp. 302–306, 2019.

G. S. Xia, X. Bai, J. Ding et al., “DOTA: a large-scale dataset for object detection in aerial view of images,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3974–3983, Salt Lake City, UT, USA, June 2018.

S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1492–1500, Honolulu, HI, USA, July 2017.

T. Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2117–2125, Honolulu, HI, USA, July 2017.

Z. Liu, L. Yuan, L. Weng, and Y. Yang, “A high resolution optical satellite image dataset for ship recognition and some new baselines,” in Proceedings of the International Conference on Pattern Recognition Appli- Cations and Methods, pp. 324–331, Porto, Portugal, February 2017.

D. Karatzas, L. Gomez-Bigorda, A. Nicolaou et al., “ICDAR 2015 competition on robust reading,” in Proceedings of the International Conference on Document Analysis and Recognition, pp. 1156–1160, Tunis, Tunisia, August 2015.

J. Ma, W. Shao, H. Ye et al., “Arbitrary-Oriented scene text detection via rotation proposals,” IEEE Transactions on Multimedia, vol. 20, no. 11, pp. 3111–3122, 2018.

S. M. Azimi, E. Vig, R. Bahmanyar, M. Körner, and P. Reinartz, “Towards multi-class object detection in unconstrained remote sensing imagery,” in Proceedings of the Asian Conference on Computer Vision, pp. 150–165, Perth, Australia, December 2018.

X. Yang, J. Yang,, J. Yan et al., “Scrdet: towards more robust detection for small, cluttered and rotated objects,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 8232–8241, Seoul, Korea, November 2019.

J. Ding, N. Xue, Y. Long, G.-S. Xia, and Q. Lu, “Learning Roi transformer for detecting oriented objects in aerial view of images,” 2018, arXiv preprint arXiv:1812.00155.

X. Yang, Q. Liu, J. Yan, and A. Li, “R3DET: refined single-stage detector with feature refinement for rotating object,” 2019, arXiv preprint arXiv:1908.05612.

J. Dai, Y. Li, K. He, and J. Sun, “R-FCN: object detection via region-based fully convolutional net- works,” in Proceedings of the Advances in Neural Information Processing Systems, pp. 379–387, Barcelona, Spain, December 2016.

S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: towards real-time object detection with region proposal networks,” Advances in Neural Information Processing Systems, vol. 39, no. 6, pp. 91–99, 2015.

Z. Tian, W. Huang, T. He, P. He, and Y. Qiao, “Detecting text in natural image with connectionist text proposal network,” in Proceedings of the European Conference on Computer Vision, pp. 56–72, Amsterdam, The Netherlands, October 2016.

X. Zhou, C. Yao, H. Wen et al., “EAST: an efficient and accurate scene text detector,” in Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, pp. 5551–5560, Honolulu, HI, USA, July 2017.

Y. Jiang, X. Zhu, X. Wang et al., “R2CNN: rotational region CNN for orientation robust scene text detection,” 2017, arXiv preprint arXiv:1706.09579.

X. Liu, D. Liang, S. Yan, D. Chen, Y. Qiao, and J. Yan, “Fots: fast oriented text spotting with a unified network,” in Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, pp. 5676–5685, Salt Lake City, UT, USA, June 2018.

M. Liao, Z. Zhu, B. Shi, G.-S. Xia, and X. Bai, “Rotation-sensitive regression for oriented scene text detection,” in Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, pp. 5909–5918, Salt Lake City, UT, USA, June 2018.

X. Xue, X. Wu, and J. Chen, “Optimizing biomedical ontology alignment through a compact multiobjective particle swarm optimization algorithm driven by knee solution,” Discrete Dynamics in Nature and Society, vol. 2020, no. 7, Article ID 4716286, pp. 1–10, 2020.

X. Xue and J. Chen, “Using compact evolutionary tabu search algorithm for matching sensor ontologies,” Swarm and Evolutionary Computation, vol. 48, pp. 25–30, 2019.

Z. Du, J. Pan, S. Chu, H. Luo, and P. Hu, “Quasi-affine transformation evolutionary algorithm with communication schemes for application of RSSI in wireless sensor networks,” IEEE Access, vol. 8, pp. 8583–8594, 2019.