Vehicle detection is one of the important technologies in intelligent video surveillance systems. Owing to the perspective projection imaging principle of cameras, traditional two-dimensional (2D) images usually distort the size and shape of vehicles. In order to solve these problems, the traffic scene calibration and inverse projection construction methods are used to project the three-dimensional (3D) information onto the 2D images. In addition, a vehicle target can be characterized by several components, and thus vehicle detection can be fulfilled based on the combination of these components. The key characteristics of vehicle targets are distinct during a single day; for example, the headlight brightness is more significant at night, while the vehicle taillight and license plate color are much more prominent in the daytime. In this paper, by using the background subtraction method and Gaussian mixture model, we can realize the accurate detection of target lights at night. In the daytime, however, the detection of the license plate and taillight of a vehicle can be fulfilled by exploiting the background subtraction method and the Markov random field, based on the spatial geometry relation between the corresponding components. Further, by utilizing Kalman filters to follow the vehicle tracks, detection accuracy can be further improved. Finally, experiment results demonstrate the effectiveness of the proposed methods.
With the rapid development of intelligent traffic control, computer vision has attracted much attention from designers of intelligent transportation systems (ITSs) [
At present, video-based vehicle monitoring systems can be divided into two categories according to two different kinds of vehicle features: vehicle appearance and vehicle moving character. In the method based on vehicle appearance [
The method based on a 3D vehicle model [
The method based on features [
In this paper, we aim to solve how to
Under the conditions of normal light illumination and the basic rules of the road, it is not a problem for a human being to locate the target vehicle’s headlights and its color information quickly. However, if a computer vision method is utilized to design a robust algorithm to fulfill these tasks, it becomes extremely challenging to detect vehicle headlights, license plates, and taillights accurately.
In this paper, we mainly study the target recognition algorithm in traffic monitoring systems. Based on the probabilistic model of the spatial relation, the detection of the target vehicle’s components is exploited instead of detection of the entire vehicle.
There are two main innovations in our paper: on one hand, the shape and size of the vehicle components are distorted due to camera projection transformation. In this paper, therefore, an inverse projection algorithm is proposed to construct an inverse projection map. The inherent shape and size of the vehicle can be obtained on the inverse projection map, and the saliency components of the vehicle can be detected using these inherent attributes. Details of this public detection of vehicles during both day and night are presented in Section
In the evening, since sunlight is limited, only the headlight information of the vehicles can be used. The target vehicle components at night are detected by the following steps: Search the real data according to the car design and manufacturing standard, and then establish a Gaussian mixture model (GMM) model based on the real data [ Detect the dominant information of the vehicle components by using the inverse projection map. Use the GMM model to train the dominant information, and obtain the GMM stochastic value. Judge the stochastic value to achieve final vehicle detection.
During the daytime, solar illumination is adequate, and thus the color information of the vehicle’s parts can be utilized to detect the vehicle license plate and taillights. In the daytime, the detection of the vehicle’s license plate and taillight can be realized by using background subtraction and a Markov random field (MRF) along with the spatial geometry relation between the corresponding components.
The main flow of the proposed algorithm is shown in Figure
Main flow of proposed algorithm.
In this paper, an inverse projection plane is preliminarily determined in the traffic scene that has been calibrated [
The proposed method relies on the space information with three dimensions; therefore, the calibration procedure for traffic scenes is necessary, and there are many calibration methods for traffic scenes. In this paper, the direct linear transformation (DLT) as proposed by Abdel-Aziz and Karara [
The inverse projection plane [
The part of the surface of the vehicle can be approximated as a plane with some geometric features. If the vehicle in three-dimensional (3D) space is regarded as a polyhedron, when the characteristics of different faces of the vehicle body are selected as the detection objects, the reverse projection plane is attached to the corresponding plane of the vehicle body, making the data after the construction of the inverse projection able to effectively show some of the apparent characteristics of the body (see Figure
Schematic of inverse projection plane.
According to the above-described method, an inverse projection plane that can be fitted with a certain local surface of the target is arranged in the space and divided into a grid with a certain resolution (such as 1 cm × 1 cm). The camera’s perspective relation is that the information contained in the grid is projected onto each pixel of the corresponding projection area on the image, where the inverse projection relationship from the image projection area to the reverse projection plane is determined, that is, a small inverse of the inverse projection plane in the space. The grid corresponds to a pixel on the image.
The inverse projection map is a pixel representation of the inverse projection plane, which means that a small grid in the inverse projection plane is represented by a pixel in the inverse projection map. The process of building inverse projection map data is as follows:
Suppose that
Imaging model of cameras and inverse projection transformation.
It can be seen from Figure
In this paper, the road traffic scene calibration during the course of the experiment is designed as two inverse projection planes, respectively, perpendicular to the road, as shown in Figure
Original image: (a) inverse projection map 1 (tail) and (b) inverse projection map 2 (side).
Nighttime vehicle detection can use the center surround extreme as in [
Flowchart of nighttime vehicle target detection algorithm.
The background difference method [
The mathematical representation of the background subtraction method is as follows: the hypothetical image size is
The background extraction environment chosen for this paper is nighttime traffic scenes, so the impact of weather and shadows is relatively small, and it is used to reveal a significant difference in the brightness of nighttime vehicle headlights upon block segmentation of the background, so the background extraction requirements are not too difficult to meet. As shown in Figure
(a) Set inverse projection plane, (b) inverse projection map, (c) background extraction, and (d) foreground segmentation and binarization.
As the most obvious characteristic of nighttime vehicle detection [
Headlights of the foreground segmentation region and its geometric features.
Calculating the foreground object block of these characteristics can exclude the nonheadlights block. The mathematical definitions and expressions of these geometric features, as well as the threshold values, are set as follows, which expresses the computing method for area
The method of computing the degree of circularity
The method of computing the geometric similarity of headlights is
Using the above judgment conditions to detect headlights, the rough matching results are shown in Figure
Frame 567: left, inverse projection plane set; right top, building the inverse projection map; right bottom, rough matching.
At this stage of the proposed method, rough matching of the vehicle headlights has been completed. Now, using the headlight parts and the spatial relation of the headlights for recognition and positioning and for alternative vehicle identification and localization is a very important part of the process. According to the actual life of the spatial relationship features of the vehicle components, one can extend the spatial relationship of the headlight features: it belongs to the center of gravity of the vehicle if the same pair of target headlights is presented in the same horizontal line and at the same vertical distance with parameter dimensions for practical production of vehicles. The spatial relationship is shown in Figure
Schematic of the spatial relationship between parts of vehicle headlights.
The mathematical expression of the spatial relationship between the headlights is as follows:
In this paper, vehicle headlight distances of samples of two headlights were statistically calculated in the
Values of variables of the spatial relationship of headlight components.
|
126 | 148 | 136 | 141 | 128 | 139 | 110 | 133 | 132 |
|
|||||||||
|
1 | 2 | 0 | 2 | 1 | 0 | 2 | 1 | 0 |
|
51 | 73 | 58 | 64 | 70 | 80 | 79 | 62 | 52 |
|
133 | 149 | 212 | 198 | 147 | 146 | 157 | 145 | 215 |
|
|||||||||
|
0 | 1 | 3 | 0 | 2 | 1 | 3 | 1 | 2 |
|
|||||||||
|
79 | 80 | 89 | 97 | 51 | 52 | 70 | 67 | 91 |
|
|||||||||
|
160 | 142 | 134 | 154 | 135 | 144 | 133 | 146 | 223 |
|
|||||||||
|
3 | 2 | 2 | 1 | 0 | 3 | 2 | 1 | 4 |
|
|||||||||
|
59 | 86 | 52 | 73 | 69 | 83 | 57 | 81 | 96 |
From the table, we can see that the difference of headlights on samples of the
Spatial relationship model of headlight components for different types of vehicles.
Using the blue component of vehicle license plates and taillights, respectively, we studied color detection of the target vehicle components. In RGB color mode, the blue component of the license plate background is larger than the red and green components, and the red and green components are very small. The red weight of the rear lights is greater than the blue and green component weights, and the blue and green component weights are also small. Based on these features, after processing, the target video sequence color space conversion can locate the license plate and the taillights. The detection flowchart is shown in Figure
Flowchart of daytime vehicle target detection.
Blue license plate RGB component histogram statistics.
RGB color pattern original image
R component histogram
G component histogram
B component histogram
Therefore, by analyzing the apparent characteristics of the license plate, we convert the video to a special color space:
Color conversion results.
In this step, our algorithm utilizes the blue information of the license plate, and thus color video is adopted (gray-level video has no color information).
After transformation of the image sequence, a single channel image is obtained, and the pixels of the license-plate-region characteristics are quite obvious. The brightness is prominent, but brightness also exists around the target region. In order to exclude the interference of the surrounding pixel brightness, the histogram of the single channel image can be used, as shown in Figure
Threshold segmentation results.
Results of gradient extraction.
In the video image sequence, except for the license plate area, there are still some similar license areas that also have a gray gradient, but that can also match a color feature, which may lead to a license plate detection error. Through observation, it is found that the license plate region has not only a significant gray gradient, but also a more uniform distribution. Therefore, we further improve the validity of the algorithm via the texture consistency of the vehicle license plate region [
License plate location results.
Inverse projection map
Positioning results
Taillights comprise a significant feature of a vehicle, and note that it is obvious that the taillight color is red in our experiment. We therefore use the red color to detect taillight.
Taillight RGB component histogram statistics.
RGB color pattern original image
R component histogram
G component histogram
B component histogram
After the calculation of the color conversion model, the vehicle taillight area is enhanced, and the nontaillight area is suppressed. As observed, in the RGB color mode the red component of the vehicle taillight is greater than the blue and green components, and the difference between the green and blue components is small. Upon analyzing this feature, we obtained the color conversion model, as shown in Figure
Color conversion results.
Extraction of taillight and closed operation.
Considering the space geometry relationship between the license plate and headlight, the Markov random field (MRF) model was used to detect the target.
A MRF model mainly includes a Markov property and a random field: A Markov property is a given sequence of random variables arranged with time; the state of In a random field, a value is assigned according to a certain distribution of each phase location. It mainly contains two elements: the location and phase space.
A 1D Markov stochastic process is a sequence of random variables
In the application of image processing, for the convenience of modeling, the MRF model introduces the concept of a system and group in the field, which is used to define the relationship between a pixel and the surrounding pixels in the image. Common areas of the system include the first-order field system and the second-order system. The first-order system is the current pixel of the upper and lower and left and right four positions of pixels. In a second-order system, except for a first-order system in four positions of pixels and diagonal four pixels, as shown in Figure
Field system.
We define the image of
Figure
Relationship among graph nodes.
In this paper, we use the probability distribution expression of the MRF model:
Vehicle tracking can be used to predict the position of the vehicle and to match the video image of the vehicle, and the Kalman filter [
In Section
Assuming that the spatial probability model of the headlights is
When the probability of the candidate headlight component is greater than a certain threshold in a given space probability model, then we can take these candidate headlight parts to be the target vehicle. The nighttime vehicle detection results are shown in Figure
Nighttime vehicle detection results.
After building the model as described in Section
The energy function of the node is expressed as
The energy function of the edge is expressed as
Using the GMM method to determine the angle between the taillight and the license plate, as well as the horizontal angle between the two taillights, the distribution of the taillight and license plate is obtained and shown in Figure
License plate and taillight relationship model
Two-taillight relationship model
The confidence of nodes and edges determines the MRF. In this paper, the maximum a posteriori (MAP) can be calculated by all labels
Daytime vehicle detection results.
If it is a blue car, detection is handled as follows: through analysis, after color model transformation, the gray gradient of the blue body is significantly enhanced, and there is a large target area on the transformation image.
According to the results of several experiments, if the total area of the target is greater than the total area of inverse projection map, which is 1/5, we can view the target as the rear of the vehicle, as shown in Figure
Processing method for blue car.
Furthermore, the vehicle taillight is red, and the threshold segmentation results are shown in Figure
Taillights of blue car.
Inverse projection map
Taillight segmentation
In the nighttime scene, we selected the midpoint of the headlights’ connection and vehicle speed as the state vectors, and the experimental results are shown in Figure
Tracking results.
312th frame
315th frame
381st frame
385th frame
This paper mainly introduces a method for realizing the recognition of a target component by using the spatial relationship model. The recognition result of the target was replaced by the result of the recognition of some parts of the target. In the real world, some inherent characteristics of the spatial relationship among the components of the target exist that would enhance the ability to describe and identify the content of the target. Based on the components of the vehicle detection algorithm, and by using multiple local components, a better detection effect was realized, but some missed and false detection phenomena still existed. Therefore, further improvement of the proposed method is required, mainly in the following aspects: In the evening, detecting the headlight targets using the GMM, if only one headlight is turned on, or none are turned on, a detection error would be caused. Since the model is established based on the amount of statistical samples, more samples are needed in order to adapt the method to more models. In the daytime, the vehicle is detected by the taillights and license plate color feature of the target. If the license plates are the same color as the car body, the detection results would be affected. Similarly, during taillight detection, a red-colored vehicle body would affect the test results. Therefore, the detection algorithm is still relatively imprecise and must be improved.
The authors declare that they have no conflicts of interest.
This work was funded by the Project of Shaanxi Provincial Science and Technology Program (Grant nos. 2014JM8351, 2015JZ018, and 2016JQ6011), the Fundamental Research Funds for the Central Universities (Grant nos. 2013G1241109 and 310824173603), and the National Natural Science Foundation of China (Grant nos. 61501058 and 61572083).