Ring Fusion of Fisheye Images Based on Corner Detection Algorithm for Around View Monitoring System of Intelligent Driving

,


Introduction
In the past decades, because of the rapid growth of road transportation and private cars, road traffic safety has become an important problem in society [1][2][3].The statistics results of the national traffic accidents show that the proportion of accidents caused by drivers' vision limitation, the reflection of delay, judgment error, and improper operation accounted for up to 40% of the total accidents [3][4][5].In order to solve the above problem, the advanced driving assistance system (ADAS) received more and more attention, such as lane departure warning system (LDWS), pro-collision warning system (FCWS), blind spot monitoring (BSD), and around view monitor (AVM) [6,7].Among them, AVM is used to provide the driver with the 360-degree video image information around the body, in the parking, and in crowded city traffic conditions to reduce the user's visual blind area, to help users to better judge the road traffic conditions around the vehicle, to avoid collision with pedestrians and vehicles around, and to make the driving process safer and more convenient [8][9][10].
The key technology of AVM is fisheye image correction and image fusion.In this paper we focus on image fusion, which includes image registration and blending.There are three main types of registration methods, which are region matching method, transform domain based method, and feature matching method.Among them, feature based registration method is fast and accurate, and some features are robust to image deformation, illumination change, and noise.It is a common method of image registration.In the document, the overlap points of two correction images are extracted by matching the SFIT features with scale and direction invariance [11,12].The feature operators usually used to extract overlapping corner points include Harris [13,14], Canny [15,16], and Moravec [17,18].These feature operators are mainly used in the registration process of general image mosaics.However, in AVM system, we match images by scene calibration.Documents [19,20] detect the corner points of checkerboard patterns by quadrilateral join rules.In documents [21,22], the initial corner set is obtained by the improved Hessian corner detector, and the initial concentration of false points is eliminated by the intensity and geometrical features of the chessboard pattern.But the above two methods are mainly designed for the chessboard pattern used in camera calibration, which do not meet the requirement of calibration pattern used in scene calibration.Due to the influence of inherent difference of camera and the installation position, there is a difference of exposure between the images, which leads to the obvious stitching trace.Therefore, the image blending is necessary after registration.In the documents, the optimal seam is searched by minimizing the mean square variance of pixel in the region, and the adjacent interpolation method is used to smooth the stitching effect, but this method is not suitable for the scene with too large difference of exposure [23,24].The documents fuse several images captured by the same camera with different parameters from the same angle of view by weighting method, but this method can only adjust the brightness and cannot reduce the color difference [25,26].In the documents, seamless stitching is achieved by tone compensation near the optimal seam, but the stitching seam of around view system is fixed, so the AVM image cannot be fully fused by this method [27,28].
Therefore, in order to fully consider the needs of scene calibration, we propose an integrated corner detection method to automatically detect all corners in the process of image registration.In order to fully consider the influence of inherent difference of camera and the installation position, we propose ring fusion method to blend images from 4 fisheye cameras.The main contribution of this paper lies in the following aspects: (1) limitation condition of minimum area and shape are used to remove redundant contours for corner detection successfully.We also improve the corner positions extraction accuracy, by detecting corners in fisheye images at first and calculating the corresponding position in corrected image.(2) The color matching method and ring shape scheme are used in image blending for a more smooth transition successfully, which makes it possible to seamlessly fuse images with large differences in exposure.(3) A Matlab toolbox for image blending in AVM system is designed.
The rest of this paper is organized as follows: Section 2 introduces AVM architecture.Section 3 describes the methodology of image registration and blending in detail.Section 4 describes the experiment result of our method.Conclusions are offered in Section 5.

AVM Architecture
The algorithm flow of the AVM is shown in Figure 1.Firstly, we input fisheye images of the calibration scene and detect corner points.Then, positions of these corner points in corrected images are calculated by a correction model.Meanwhile, we use Look Up Table (LUT) to correct fisheye images and obtain the corrected images.Secondly, the target positions of corner points in output image are calculated by size data in calibration scene.Then, positions of corner points in corrected images and their target positions are used to compute homography matrix H. Finally, we project corrected images into the coordinate of output image by using homography matrix H. Then we use ring fusion method to blend them, which is the emphasis of this paper.
In our experiment, the Volkswagen Magotan has been used.The length of the vehicle is 4.8 m, and the width is 1.8 m.We use fisheye camera with a 180-degree large view angle, with a focal length of 2.32 mm. 4 fisheye cameras are mounted on the front, back, left, and right sides of the vehicle separately.The size of the image captured by fisheye camera is 720 * 576.The size of AVM output image is 1280 * 720.This paper develops the proposed method on a PC.The adopted simulation processor is the Intel(R) Core(TM) i7-6700HQ CPU at 2.60 GHz and the simulation software is MATLAB.

Scene Calibration.
Calibration scene is set up for image registration in next step.The distance between vehicle body and calibration pattern in the front and rear positions is 30 cm and the right and left is 0 cm.The reference point of front pattern is A, and the rear is F. We made the point A collinear with the left vehicle body and F collinear with the right vehicle body.And there are 12 point positions we need in every view angle, as shown in Figure 2.
The size data which need to be measured include the following: (1) Car length: the line length of AE.
(2) Car width: the line length of AB.
(3) Offset: the line length of AC or the line length of BD.After the measurement of size data, the target positions of corner points in the coordinate of output image are calculated by the following parameters: the size data measured above, the size of output image defined by users, and the size of calibration pattern and vehicle.The calculation process of the target position of all points is the same.We take the target position of point 1 (as shown in Figure 1) as an example.Firstly, we calculate the position of point 1 in the calibration scene, as shown in

Journal of Robotics
where the origin of calibration scene is located at the center of the calibration scene, as shown in Figure 1.( where scale denotes the scaling factor from calibration scene to coordinate of output image. img denotes the width of output image. real denotes the width of calibration scene. ( 1 , V 1 ) denotes the position in calibration of output image.
( 1 ,  1 ) denotes the position in calibration scene.Algorithm steps of integrated corner detection method are as follows:

Image Registration Based on
(1) Input fisheye images of calibration scene from all 4 cameras.
(2) Use the Rufli corner detection method to detect the corners in the chessboard array.
(3) Based on the relative position between the black box and the chessboard array, use the detected corners from step (2) to obtain the Region Of Interest (ROI) of the big black box.
(4) Preprocess the ROI by adaptive binarization using "adaptivethreshold" function in Opencv and a morphological closing operation to denoise.
(5) Obtain the contour of big black box from ROI and the positions of contour vertex by "findContours" function in Opencv.Then we use the following method to remove redundant contours.
(1) Limit the minimum area of the contour: according to the size ratio of the chessboard array to the big black box and their relative positions,

Coordinate of output image
Coordinate of corrected image where  area min denotes the threshold of big black box area and  areaavg denotes the average area of small box in chessboard array.(2) Limit contour shape: according to the location of the big black box and the imaging features of fisheye camera, the big black box should be in a fixed shape.The shape restrictions are shown in where  1 and  2 denote the diagonal length of the contour,  3 and  4 denote the length of adjacent side of contour,  denotes perimeter of contour, area contour denotes the area of contour, and area rect denotes the area of envelope of contour.
(6) Use the SUSAN method to locate the exact positions of the contour vertex around positions obtained from step (5).

Image Registration and Coordinate Unification.
After the calibration of Figure 3 and corner detection, the corner positions in coordinates of corrected images and their target positions in coordinates of output images are obtained.Then we need to unify the coordinate of 4 corrected images into the coordinate of output image, as shown in Figure 3.The specific process is as follows.Firstly, we calculate the homography transform matrix, as shown in (5).The form of this matrix is shown in (6).
where (  ,   ) denotes the pixel position in the coordinate of corrected image.(  ,   ) denotes the pixel position in the coordinate of output image.

Image Blending.
As the corrected images from 4 cameras are different from each other in brightness, saturation, and color, we blend them to improve the visual effect of output image by using ring fusion method.The detailed process is shown as follows.
(1) Equalization Preprocessing.The "imadjust" function in Matlab is used for equalization preprocessing to reduce (

2) Ring Color Matching
Step 1 (spatial transformation).As RGB space has a strong correlation, it is not suitable for image color processing.So we transform RGB space to the l space where the correlation between three channels is the smallest.The space conversion process includes three transformations, namely,  →   →  → .
Secondly, from CIE XYZ space to LMS space, one has Since the data are scattered in the LMS space, it is further converted to a logarithmic space with a base of 10, as shown in (10).This makes the data distribution not only more converging but also in line with the results of the psychological and physical research of human feeling for color.
Finally, from LMS space to l space, one has (11).This transformation is based on the principal component analysis (PCA) of the data, where l is the first principal component,  is the second principal component, and  is the third principal component.
[ [ After the above three steps, the conversion from RGB to l space is completed.
Step 2 (color registration).Firstly, the mean and standard deviations of every channel in l space are calculated according to where  denotes the mean value,  denotes the total number of pixels, V  denotes the value of the pixel , and  indicates the standard deviation.Secondly, the color matching factors are calculated according to where   denotes the factor that matches the color of V2 image to V1 in channel . ,V1 denotes the variance of V1 image in channel . ,V2 denotes the variance of V2 image in channel l.And the rest is similar.Finally, we match the color of images, as shown in where   V2 denotes pixel value of image V2 after color matching in channel .  denotes the factor of color matching in channel . V1 denotes pixel value of image V1 in channel . V1 denotes average pixel value of image V1 in channel . V2 denotes average pixel value of image V2 in channel .And the rest is similar.
Step 3 (global optimization).Then, we match the color of images from 4 cameras anticlockwise as follows to reach a global optimization result.Firstly, we match the colors of  4 to  3 , then  3 to  2 , then  2 to  1 , and finally  4 to  1 , which forms a ring shape, as shown in Figure 5.The processing result of left view is shown in Figure 4(c).
(3) Weighted Blending.After color matching, the visual effects of output image have been greatly improved.But around the stitching seam between different corrected images, the visual effect is still not enough.Therefore, we use (15) to ensure smooth transition.The interpolation result of left view angle image is shown in Figure 4(d).
where (, ) denotes the pixel value in output image and (, ) is the position index of pixel. 1 (, ) and  2 (, ) denote the corresponding pixel value in corrected images  1 and  2 . denotes the distance from pixel to the seam. max denotes the width of transition field, as shown in Figure 5.

Experiment Result
Some details of the experiment have been provided in part 2 of this paper.So, in this part we only introduce the result.The fisheye images captured from 4 cameras are shown in Figure 6.And their corresponding corrected images are shown in Figure 7.The corner detection and calculation result are shown in Figure 8, where Figure 8(a) shows  the corner positions detected in the distortion image and Figure 8(b) shows the corresponding positions calculated in the corrected image.The integrated corner detection algorithm is compared with several other corner detection algorithms, in Table 1.
In Table 1, C c denotes the number of corner points detected correctly in chessboard.C b denotes the number of corner points detected correctly in big black box.N denotes all the number of corner points detected in the calibration scene.The Rufli method cannot detect vertices of the big black box.The Harris and Shi-Tomasi methods cannot detect all target vertices and generate a lot of corner redundancy.And the integrated corner detection algorithm can accurately extract all the target corner points of calibration pattern in the scene.As a result, the integrated corner detection algorithm proposed by us is effective.
The output image result is shown in Figure 9. Figure 9(a) is the result before image blending, and Figure 9(b) is the result after image blending.The experimental results show that the proposed algorithm has visual effect around the stitching seam, which proves that our ring fusion method is effective.

Conclusion
This paper has proposed a ring fusion method to obtain a better visual effect of AVM system for intelligent driving.To achieve this condition, an integrated corner detection method of image registration and a ring shape scheme for image blending have been presented.Experiment results prove that this designed approach is satisfactory.100% of the        required corner is accurately and fully automatically detected.The transition around the fusion seam is smooth, with no obvious stitching trace.However, the images we processed in this experiment are static.So, in the future work, we will transplant this algorithm to development board for dynamic real-time testing and try to apply the ring fusion method to more other occasions.

Figure 1 :
Figure 1: Flow chart of image fusion.

Figure 2 :
Figure 2: The illustration of calibration scene.

Corner Detection 3 . 2 . 1 .
Detect and Calculate the Corner Point Positions.Firstly, the corners are detected automatically in the fisheye image by the integrated corner detection method.Secondly, the corresponding positions of these corners in the corrected image are calculated using the correction model.Finally, we save the positions in the corrected image for the next computation of homography matrix.
Firstly, from RGB space to CIE XYZ space, one has

Figure 5 :
Figure 5: The illustration of ring color matching method.

Figure 6 :
Figure 6: Fisheye images from each camera.
(a) Front view (b) Right view (c) Back view (d) Left view
(a) Corner positions in distorted image (b) Corresponding positions in undistorted image

Figure 8 :
Figure 8: The corner detection and calculation result.

Figure 9 :
Figure 9: Stitched bird view image of AVM.
1 ,  1 ) denotes the position of point 1,  denotes the vehicle width,  denotes the vehicle length,   is the white edge width, and  1 is the width of big black box.
denotes the corner position in the coordinate of corrected image.  = [    1]  denotes the target position in the coordinate of output image.Secondly, we project every pixel of 4 corrected images into the coordinate of output image, as shown in  =  31   +  32   +  33   =  11   +  12   +  13    =  21   +  22   +  23  , where  = [    1]

Table 1 :
Comparison of different corner detection algorithms.