Study on Leading Vehicle Detection at Night Based on Multisensor and Image Enhancement Method

the original


Introduction
According to the statistics, the number of traffic accidents was up to 196812 in 2014; they caused 58523 fatalities, 211882 injuries, and 1075.42-milliondirect property loss [1].While rear collision accidents represented 17.27% of total traffic, the represented 9.84% of fatalities, 10.7% of injuries, and 20.8% of direct economic loss in 1995 in China [2].
With the development of processor technology and sensor technology, more and more security systems are applied to the field of the vehicle.To reduce rear accident, a preceding vehicle detecting method at night based on multisensor and image enhancement method are proposed.Owing to a lack of enough light at night, most of vehicle feature information during the daytime is not available, so daytime vehicle detection algorithm is basically ineffective.Vehicle taillight is obvious vehicle features at night; at present, the studies on leading vehicle detection and recognition at night mainly use single vision sensor to obtain preceding vehicle's visual information and identify preceding vehicle based on image information extracting taillight features.Liu et al. combine vehicle taillight color and brightness to detect taillight in [3]; Wu et al. track vehicles through using a pair of headlights [4]; Tang et al. extract the region of interest by using the frame difference method [5]; Wang present an image segmentation method based on fuzzy theory, extracting the license plate and taillights feature in [6]; Qi and Chen distinguish the vehicle position based on the HSV color model to segmental taillight color information in [7]; Zhou segments image based on R channel histogram in RGB color space by the use of adaptive threshold, and the effect is not very satisfactory [8].Digital camera is an effective sensor for detecting a vehicle, but it has some limitations, considering digital camera, and laser radar information to detect target vehicle was put forward in the literature [9,10].To a certain extent, although laser radar and digital camera are complementary, laser radar is very sensitive to weather, lighting, and surface smoothness of obstructions, so it is not suitable for complex road environment.Since the millimeter wave radar is less susceptible to outside interference and the distance measurement accuracy is high, it also can get the exact preceding vehicle speed, angle, and so forth.We present a technique for leading vehicle detection at night by the use of millimeter wave radar and digital camera fusion getting multisensor data, screening obstacles data that was detected by using millimeter wave radar through prior knowledge, establishing initial dynamic region of interest (ROI) by the use of radar data and image information, and extracting the vehicle feature within the narrow range based on vision sensors; we use D-S evidence theory information to reduce the amount of calculation and subjective threshold impact on detection accuracy and improve the accuracy of vehicle location and execution efficiency.
While the detection system of preceding vehicles can reduce the risk of rear vehicle, it cannot reduce the driver psychological pressure during driving at night.Since the distance that the system detected is in the range of 60 m-70 m and rear collision may be in an emergency situation when we detected vehicle, in this case, using this system cannot completely avoid rear collision accidents.The statistics show that more than 80% of the road environment information is acquired by the driver vision.Thus, in order to avoid traffic accidents fundamentally, we need to improve the driver visual perception in the traffic scene at night.Driver visual is limited at nighttime and visual range is short, so it is easily prone to fatigue and it is hard to observe road traffic conditions.According to the statistics, the driver is more intense and the reaction time is longer during driving at nighttime than at daytime when emergency braking, which may result in a serious accident.The night image enhanced algorithm remains to be study.
This paper is organized as follows.Section 2 presents the preceding vehicle detecting method at night based on multisensor.Section 3 presents image enhancement theories.The result of image enhancement methods is illustrated in Section 4. Finally, we conclude the paper in Section 5.

Nighttime Vehicle Detection Algorithm
The approach consists of hypothesis generation (HG) and hypothesis verification (HV).In hypothesis generating process, we can obtain candidate target distance, angle, speed, and other pieces of information by radar and then get the world coordinates of the candidate target, on the basis of an inverse, the camera calibration principle; we obtain conversions relationship between the world coordinates and image pixel coordinates, initially identified region of candidate targets on the image, namely, the region of interest (ROI).Hypothesis verification process for image segmentation is processed through improved OTSU and then we use the image processing method, prior knowledge, and D-S evidence theory to detect the presence of the vehicle features in ROI.Algorithm flow chart is shown in Figure 1.

Processing Radar Data and Determining Candidate Target.
Millimeter wave radar receives hexadecimal data; according to the agreement, we can calculate radar data and extract useful information that can be used to detect a vehicle; the data include angle of preceding vehicle relative to our vehicle, distance, speed, and reflection intensity.In actual measurement, a part of millimeter wave radar signal is empty target signal, the inactive target signal, and stationary target signal.First, we remove the interference of three target signals (empty target signal, the inactive target signal, and stationary target signal).If the signal value is in the range of distance threshold and the relative velocity threshold, the data are stored.According to the longitudinal width threshold, we judge the target vehicle and own vehicle in the same lane and then further record preceding goal vehicle that has been screened in accordance with from near to far principle.Radar scan plane is shown in Figure 2, in which  indicates the radar scan radius that detected vehicle (the distance threshold is 50 m and the relative velocity threshold is 30 m/s and the width of the lane is 4 m; | × sin | ≤ 2).
The valid targets are shown in Table 1.Assuming the front vehicle is stationary, relative velocity threshold is the speed of our vehicle relative to leading vehicle; it is positive or negative.If relative velocity threshold is a negative value, it indicates that our vehicle speed is higher than the front vehicle; a positive value indicates that our vehicle speed is lower than the leading vehicle.

Fusion Digital Image and Radar Data.
Coordinate systems of radar sensors and digital image are different; we must establish conversion model of two sensor coordinate systems to achieve spatial integration of radar and machine  radar vision so that it can convert from the radar coordinates to image pixel coordinates [11].We established coordinate system in accordance with the principles of the right-handed coordinate system, and then we can establish the spatial relationship between the radar coordinate system and the image pixel coordinates by formulas (1)- (2).
(  ,   ,   ) of the world coordinate system is converted to (, V) of the image pixel coordinates.Let  be focal length: distance between the image plane and the projection center.(  ,   ,   ) expressed the camera coordinate system; let ( 0 , V 0 ) be principal point, let 1/  and 1/  be the distance adjacent pixels in the horizontal and vertical directions of the image sensor, and   expressed the optical axis of the camera.The conversion equals the following: ] . ( The relationship between the world coordinate system and the camera coordinate system radar scan plane.
Among them, there is a relationship shown in Figure 3 coordinates between the world coordinate system and the camera coordinate system: let  be rotation matrix, and  expressed translation matrix.
The transform between the camera coordinates and image physical coordinates is shown: The transform between the physical coordinates and image pixel coordinates is shown: Figure 4: Positional relationship between the radar coordinate system and the world coordinate system:  0 − 0  0  0 expressed millimeter wave radar coordinate system and  0 −      expressed the world coordinate system.
Preceding obstacle information we acquired by millimeter wave radar is two-dimensional information in polar coordinates; the information of the barrier  (Figure 2) is converted from polar coordinates under a two-dimensional Cartesian coordinate system to rectangular coordinate system.
0  0  0 plane of radar coordinate system and     plane of the world coordinate system are parallel (Figure 4), the distance between the two planes is  0 , and center point of preceding vehicle can be projected to radar plane as the point ; we can obtain relative distance  and angle  that point  relative to the radar and we can determine the coordinates of the point  in the world coordinate system; its conversion relationship is as follows: We get center point of the preceding vehicle by the radar as input.By the use of the relationship between the radar coordinates and image pixel coordinates, we can get projection of the preceding vehicle on the pixel plane.Based on a common shape of the vehicle (aspect ratio) being projection on the pixel plane, we can establish a dynamic region of interest which will change according to the change of distance, so we can shorten search time on the image and reduce the amount of calculation.Dynamic region of interest in pixel coordinates was shown in Figure 5.
According to the statistics, we found that in general the aspect ratio of the vehicle is in the range of 0.7 to 2.0; for example, the aspect ratio of car, SUV, van, and commercial vehicle models is in the range of 0.7 to 1.3 [12].In order to avoid missing target vehicle taillights in the subsequent taillight detection process, in our paper we select 1.3 as maximum aspect ratio of several common models.According to (3), the recognized dynamic region of interest was shown in Figure 3.
It is assumed that ℎ expressed the height at which the object is projected onto the image plane, the scan radius of the radar , and vehicle height , in order to adapt to every vehicle.We assume it as 2 m, so we obtained ℎ and dynamic region of interest changing according to the distance to the radar scans preceding vehicle, so that the identification frame of the front vehicle becomes larger with the distance becoming smaller and becomes smaller with the distance becoming larger.Figure 6 shows derivation of ℎ.
(  ,   ), (  ,   ), respectively, represent the pixel coordinates of the top left corner point and bottom right corner point of the rectangular area in dynamic region of interest, (, ) represent the pixel coordinates of the vehicle center point, and / is aspect ratio of the vehicle common shape.Target vehicle of radar coordinates is shown as the region of Region of interest was shown in Figure 8.

Image Segmentation.
There is significant difference between vehicle taillight and the road surface background in gray scale; using threshold segmentation method, we can segment taillights quickly and accurately.In this paper, we use the improved OTSU algorithm to segment image to highlight taillight section that can represent features of the vehicle.The improved OTSU algorithm is based on traditional OTSU algorithm; it traverses every pixels from the minimum gray value to maximum gray value, according to the following equation: , where  0 expressed after image segmentation percentage of foreground pixels representing the total image pixels points,  1 expressed, after image segmentation, percentage of background pixels representing the total image pixels points,  0 expressed average gray of foreground pixels, and  1 expressed average gray of background pixels.When  2 is maximum, the distinction between vehicle taillight and the road surface background is maximum, so we get threshold value  and initial vehicle taillight image.By use of conventional OTSU algorithm again, we get threshold value  0 that is bigger than  and binary segment region of interest with  0 ; 1 represents target gray and 0 represents background gray, as shown in Figure 9.

Image Processing Based on Prior Knowledge and Image
Morphology.Due to the effect of noise, the image boundaries after threshold are not very smooth.There are some noise holes in object region and the background areas are dotted with small noise objects, so, after image segmentation, we process image by the use of morphological opening operation process and closing operation process to eliminate small objects, isolated slim object point, and smooth border of larger objects, but it does not significantly change the area.Erosion operation will remove the object edge point.All the points of small objects will be regarded as an edge point, so the points will be entirely omitted.Then, we perform dilate process; the large object left behind will be changed to its original size and small objects that have been deleted will be gone forever.Dilation operation causes outward expansion to the object boundary.If there are some small holes inside the object, these holes will be filled up through the dilation operation, so it is no longer a border.Then, we perform erosion operation again, external border will be changed back to its original appearance, and these internal voids will be gone forever [13].Operational rules of opening operation and closing operation are as follows: OPEN (, ) = ( ⊝ ) ⊕  CLOSE (, ) = ( ⊕ ) ⊝ .
expressed the input image and  expressed structural element, ⊝ expressed morphological dilation operation, and ⊕ expressed morphological corrosion operation.
We collect 253 pieces of images with 768 * 576 pixels at different distances, and after processing we conclude that areas of vehicle bright block are not less than 10 and not exceeding 300; for the same vehicle, horizontal distance between left vehicle taillight bright blocks and right vehicle taillight bright blocks is not less than 20 and not greater than 300.At the same time, the literature [14] presents that, in the range of 0-100 meters, bright block area of vehicle taillight image collected at different distances is not less than 10.Therefore, according to the area threshold level of the bright spots and horizontal distance threshold, we can remove some interference bright spots on the region of interest, as shown in Figure 10.

D-S Evidence Theory Fusion Characteristic Information.
We label connected region on the region of interest after the image processing and extract vehicle features such as area ratios of connected area and overlap rates in the vertical direction.We can obtain the total confidence value by fusion of D-S evidence theory and vehicle feature information.
Definition 1. Assuming that the elements of collection  are incompatible, the basic probability function value () is a mapping that, from collection 2  to [0, 1], the following conditions are met: In the paper, we defined identified framework as  = {true, false}, the area ratio of connected area and overlap ratio in the vertical direction were two propositions under the frame of discernment, respectively, representing  1 ,  2 , and the evidence probability functions of  1 ,  2 , respectively, represent  1 ,  2 .When area ratio of connected region is close to 1, the probability that the two connected regions belong to the same vehicle is relatively large.If overlap rate in the vertical direction is closer to 1, there is a greater probability that two connected regions belong to the same level, so the basic probability function values of two propositions are determined by the following formula: where AR represents area ratio of connected region and OR represents overlap rate of connected region in the vertical direction.If probability of two car taillights area ratio AR is close to 1, the probability value 1−AR is close to 0. Probability of two car taillights vertical overlap ratio OR is close to 1 and the probability value 1 − OR is close to 0. Results are in line with formula (8) D-S evidence theory.
According to D-S evidence theory combination rules, we integrate probability distribution values for compatible proposition, so we get the probability distribution values of these intersection propositions of compatible proposition.We assume that the focal element of two basic probability functions  1 ,  2 , respectively, represents  1 ⋅ ⋅ ⋅   ,  1 ⋅ ⋅ ⋅   , by the use of orthogonal rules; the two bodies of evidences were combined output: where  = ∑   ∩  =⌀  1 (  ) 2 (  ), () represents integrated probability value of  1 ,  2 ,   represents ( 9), and   denotes (10).In this article, if () is more than 0.9, we believe that two taillights come from the same car.Eventually, we establish trust threshold to verify the vehicle, as shown in Figure 11.

Results.
In the paper, we present a method to fuse data based on millimeter wave radar and digital image, focusing on researching the detection method of how to identify the preceding vehicle under complex environment at night.The hardware operating environment is Intel Pentium E6500CPU, software environment including Windows XP system, VC ++ 6.0 integrated development environment, and Opencv open source computer vision library [15].We transfer data between the millimeter wave radar system and the visual system to achieve preceding vehicle identification at night as shown in Figure 12.

Color Image Enhancement
By leading vehicle detection system, the real-time gray images are captured to detect the preceding vehicle.Although nighttime vehicle detection system can effectively identify the leading vehicle, the statistics show that driver's reaction time at night was significantly longer than during the day.Figure 13 shows that different braking distance is caused by different reaction times at different brake initial velocity.When vehicle detection system detects leading vehicle, the braking time was extended and increases the risk of rear-end.
In the meanwhile, the human eye is more sensitive to color images than gray images.Currently, most of image enhancement methods are used to enhance the gray image.Based on the human perception of color, while the human visual system can perceive about twenty different gray levels, camera, such as automobile data recorder.In this paper, we enhance color images under nighttime conditions.There are several methods to enhance image degraded by irregular illumination, including image contrast enhancement, histogram equalization [16], and retinex [17,18].These methods usually enhance an input image by increasing its contrast.Retinex can process color images, which can improve image quality caused by insufficient lighting at night.It has become a hot research field of image enhancement processing.This part will discuss how to enhance overall nighttime image by McCann99 Retinex, Frankle-McCann Retinex, and single-scale retinex (SSR).
3.1.The SSR Algorithm.Retinex theory plays an important role in the development of image enhancement.The color of the object is determined by reflectivity that is the inherent property of the object within some band, and it does not depend on the light source.In the SSR, based on the where * represents convolution operator,   (, ) represents the input color component image,   (, ) represents the illumination, and   (, ) represents the reflectance component,  = {, , }.The illumination is estimated by applying a Gaussian function to the input color component image as follows: where    (, ) represents the estimated illumination and (, ) represents the Gaussian filter function as follows: where  represents the scale parameter; in Figure 14, (a In this paper, we select 110 as scale parameter and  represents normalized factor.Finally, the output color component image is as follows: (, ) represents the output of Retinex.By using SSR algorithm to enhance nighttime image, we can overcome the situation that leading vehicle cannot be recognized properly at night, and ultimately we obtain a similar visual effect as in daytime.including taken points, compare and average iterate operating.But retinex mccann99 algorithm is more time-consuming than Frankle-McCann Retinex algorithm.

McCann99 Retinex Algorithm and Frankle-McCann
McCann99 Retinex selects pixels by using the image pyramid model layer by layer.Topmost layer image resolution is the minimum size of rows×cols and the bottom layer image resolution is the maximum size of rows ⋅ 2  × cols ⋅ 2  , rows ≥ cols, and 1 ≤ cols, rows ≤ 5, where  represents the number of layers.In the calculation process, each pixel compares with its eight neighboring pixels to obtain estimated reflectance component from top layer to bottom layer.We use the estimated reflectance component interpolating operation at the previous layer.So the size of the upper layer of the pyramid image is the same size as the image size of the lower layer by interpolating operation.Repeat interpolation and comparison operation until the end of the bottom of the pyramid image.Eventually, we can get the final color enhanced image after comparing with the original image (Figure 15).
Frankle-McCann Retinex uses spiral path to select the gray value of the estimated pixel to estimate and remove luminance component of the image.The closer to the prediction the center point is, the more points should be selected, because the point that is near the center point is more relevant to the center point.Each step will be relatively rotated 90 degrees clockwise; the distance is halved until it reaches the unit pixel distance (Figure 16).

Image Enhancement Evaluation
Looking at the results of the enhanced image, we can see that the nighttime color image is really well restored.We realize the restoration from nighttime image to the daytime image.Although the human visual system is an effective image evaluation standard, it is a subjective standard.The distribution of visual effect is proposed as shown in Figure 21; the region of gray average between 100 and 200 and the standard deviation between 35 and 80 is the optimal visual [19].
It is difficult to acquire a reference image of normal daytime in the same scene.Therefore, we evaluate enhanced image by the use of no-reference objective quality evaluation methods.In order to illustrate the results objectively, we use objective evaluation criteria to evaluate the image quality and the effectiveness of the algorithms.Five simple and effective indicators are proposed, including time, average gray, standard deviation, average gradient, and color image information entropy.Specifically, average gray denotes the quantity of lighting.Standard deviation indicates the contrast of the image.Average gradient indicates the structural features of the image.Color image information entropy indicates the image information [17] (the larger value contains the more information).The results are shown in Tables 2-6.
The evaluation results are presented in Tables 2-6.Specifically, (1) the results of the average gray show that the entire image is excessively bright by McCann99 retinex algorithm and Frankle-McCann Retinex algorithm, compared to the original image, the average gray of the SSR algorithm is improved significantly, the overall image brightness is moderate, and it is consistent with human visual experience.(2) For standard deviation, Table 6 shows that the SSR algorithm is better than the other two algorithms.It indicates that by the SSR algorithm image contrast is significantly enhanced and image detail is obviously restored.(3) For average gradient, the value after image enhancement is significantly the nighttime color image and restore the image details and image enhancement results satisfy the human visual habits.

Conclusion
In this paper, in order to reduce rear accident at night, a leading vehicle detection method at night based on millimeter wave radar vision is proposed, using the fusion of data from multisensor to detect the preceding vehicles.Our results show that radar can determine the preceding vehicle distance, speed information, and form a region of interest.
In the region of interest, we verify vehicle based on digital image information, which not only reduces the interference of the external environment, but also reduces the scope of inspection and the amount of calculation.Test results show that the method based on fusion of millimeter wave radar and digital image can be used to identify preceding vehicle effectively at night.The method for vehicle taillight with the other shapes also has good recognition results, since taillights overlap or block will cause verifying fault, which is focus of our study in the future.By the image enhancement algorithms, night images are enhanced and results have been assessed by objective evaluation and subjective evaluation.The evaluation results show that the image enhancement results satisfy the human visual habits.We believe that SSR algorithm is the best compared to Frankle-McCann and McCann99.In this paper, visual enhancement algorithms have the following disadvantages.The image enhancement algorithms cannot apply to real-time process and all nighttime images.In particular, the efficiency of the algorithm needs to be optimized and improved based on the actual the application process.Visual image enhancement algorithms applied to general image are worth to be further studied.

Figure 5 :Figure 6 :
Figure 5: Dynamic region of interest in pixel coordinates.

Figure 7 :
Figure 7: Radar target on the image of ROI.From top to bottom: the distance between detected leading vehicle and our vehicle was 5.6 m, 7.6 m, and 18.3 m.

Figure 10 :
Figure 10: Image after image processing based on a priori knowledge and morphology.

Figure 11 :Figure 12 :Figure 13 :
Figure 11: Detection of the vehicle taillight by the use of the D-S evidence theory.
) Gaussian template was very smooth, dynamic range of enhanced image was compressed, and image become locally blurred.(b) Gaussian function is relatively smooth; although the pixel dynamic range is smaller than (a), image fidelity is better.(c) Gaussian function is sharper, the central pixel receives more impact from neighbor pixels, and the details of enhanced image are better and include greater dynamic range, but enhanced image is dark and has more distortion.

3. 3 .
Comparison and Analysis on Results.We used software environment including MATLABR2009a development environment and Windows XP operating system.The camera system provides images with a resolution of 640 * 480 pixels.The image is obtained under night conditions.They compared the performance of the SSR, that of the Frankle-McCann Retinex, and that of McCann99 algorithm as shown in Figures 17-20.

Figure 17 :Figure 18 :Figure 19 :Figure 20 :
Figure 17: The comparative results of nighttime images with different algorithms.From left to right, respectively, the following are represented: original image, result image of the SSR, result image of Frankle-McCann, and result image of McCann99.

Table 1 :
Primary effective target signal.
Retinex Algorithm.McCann99 Retinex algorithm and Frankle-McCann Retinex algorithm come from Retinex algorithm based on multiple iteration strategy.Essentially, there is no difference between McCann99 Retinex algorithm and Frankle-McCann Retinex algorithm.The two algorithms,