A Rapid UAV Image Georeference Algorithm Developed for Emergency Response

The image collection system based on the unmanned aerial vehicle plays an important role in the postearthquake response and disaster investigation. In the postearthquake response period, for hundreds of image stitching or 3D model reconstruction, the traditional UAV image processing methods may take one or several hours, which need to be improved on the efficiency. To solve this problem, the UAV image rapid georeference method for postearthquake is proposed in this paper. Firstly, we discuss the rapid georeference model of UAV images and then adopt the world file designed and developed by ESRI to organize the georeferenced image data. Next, the direct georeference method based on the position and attitude data collected by the autopilot system is employed to compute the upper-left corner coordinates of the georeferenced images. For the differences of image rotation manners between the rapid georeference model and the world file, the rapid georeference error compensation model from the image rotation is considered in this paper. Finally, feature extraction and feature matching for UAV images and referenced image are used to improve the accuracy of the position parameters in the world file, which will reduce the systematic error of the georeferenced images. We use the UAV images collected from Danling County and Beichuan County, Sichuan Province, to implement the rapid georeference experiments employing different types of UAV. All the images are georeferenced within three minutes. The results show that the algorithm proposed in this paper satisfies the time and accuracy requirements of postearthquake response, which has an important application value.


Introduction
The high-resolution image collection system based on the unmanned aerial vehicle has the advantages of lightweight, low cost, and flexibility.It has to be a vital tool for rapid collecting and viewing of the disaster information in postearthquake response and reaction and attracts more attention [1][2][3][4][5][6].At present, the main processing methods of UAV images are image stitching [7][8][9][10][11] and 3D point cloud model reconstruction [12][13][14][15] based on the computer vision theory.The image stitching can get the panoramic image of the parts of a disaster area even the whole disaster area.It generally needs steps such as feature extraction, feature matching, image transformation, bundle adjustment, and image fusion, which usually consumes one or several hours to achieve the image stitching.The 3D point cloud reconstructed from the UAV images can acquire the disaster information from different sides of the concerned objects.It generally needs steps such as sparse match, dense match, adjustment, and point cloud calculation.We need much time to achieve these steps.All the methods mentioned above are implemented in the commercial software, such as PhotoScan (http://www .agisoft.com/),an image stitching software, and Smart 3D (http://www.acute3d.com/),a three-dimensional model reconstruction software.The image map and threedimensional model output from the mentioned software could be provided to the postearthquake commanders as the next level results.However, in the postearthquake response period, one to several hours is quite long for the emergency response commanders; they want to know the disaster information as soon as possible and make the correct decision to muster rescue force and distribute necessary supplies.To improve the image processing speed, [16][17][18][19] adopt the SLAM (simultaneous localization and mapping) framework to create a panorama image, which uses the incremental manner to mosaic the UAV images.For large-scale image mosaics which contain thousands of images, it becomes computationally infeasible to use the filtering approach since the size of the state vector grows very large.The commercial solutions such as DroneDeploy Live Map produces mosaics instantly during the flight, which is designed to create and view a low-resolution map on the mobile device and, if you want to get the high-resolution map, you have to upload the images to the cloud through the network and download the result after image stitching.Processing the high-quality data can take up to a few hours for a very large job with highresolution imagery (https://support.dronedeploy.com/docs/live-map).It depends on the image network speed, and it is hard to achieve the real-time georeference.To satisfy the postearthquake response application requirements, the rapid georeference method of UAV images for postearthquake response is proposed in this paper, the technique flow (see Figure 1).Firstly, the rapid georeference model is established combined with the position and attitude data collected from the autopilot system.And then, we deduce the image rotation transformation method, which converts the image rotation around the image center to the image rotation around the upper-left corner.Considering the limitation of GPS accuracy, which does not have the differential function, the feature extraction and matching method are employed to reduce the systematic error and improve the georeference accuracy.Finally, we validate the algorithm proposed in this paper through the rapid georeference experiments, which collected the UAV images from Danling County and Beichuan County, Sichuan Province, using different types of UAV.The results indicate that the algorithm could achieve rapid georeference within three minutes for up to hundreds of UAV images.It saves much more time and provides important technical support in the early postearthquake response period.

The Rapid Georeference of UAV Images
2.1.The Rapid Georeference Model of UAV Images.The disaster information collecting system based on the UAV is usually equipped with a high-resolution digital camera and an autopilot system, which could record the time, position, and attitude of the camera exposure.The UAV images could achieve the georeference by synchronizing the images and POS (positioning and orientation system) data.Because of the digital camera and the autopilot system which is installed along with the main optical axis of the camera are installed tightly, the lever arm and boresight angle will be ignored at the situation of emergency response.It satisfies the accuracy requirement.We can acquire the position and attitude data of the camera theoretically, which output from the autopilot system.In order to avoid the distortion of georeferenced images in the process of spatial transformation, which affects the disaster  In the above formulas (1) and ( 2), x′y′ T is the georeferenced image pixel coordinate.xy T is the image pixel coordinate before georeference.λ is the scale factor, which indicates the image spatial resolution.θ is the camera rotation angle around the optical axis.T is the translation matrix, T x is the translation amount along with the x-axis, and T y is the translation amount along with the y-axis.In formula (3), x ′ y ′ T is the georeferenced image pixel coordinate.xy T is the image pixel coordinate before georeference.A is the scale factor on the x-axis.B and D are the rotation parameters.E is the scale factor on the y -axis, and it is negative.C and F are the longitude and latitude coordinates of the georeferenced image upper-left corner.All the rotation and scale factors could be deduced from formula (2).Each pixel will acquire the geographic coordinates, when the Geographic Information System software loads the georeferenced image data as Figure 2. When x equals zero and y equals zero, the georeferenced pixel x′ equals C and y′ equals F. The key problem of UAV image rapid georeference is how to quickly calculate parameters C and F. Since the autopilot system collects the angle data, which indicates that the image rotates around its symmetry center but the world file adopts the upper-left as its rotation center, we have to change the center rotation to the upper-left rotation and reduce the error caused by rotation.

The Parameters of Rapid
Georeference Determination A direct and simple way to obtain the coordinate parameters is to calculate the numbers of pixels between the center and the upper-left corner, then multiply the image resolution, and finally plus the center coordinates.The calculated results are the coordinates of the upper-left corner.But it does not take the error brought from the position and the attitude data change into consideration.To solve this problem, the method based on the collinear equation is employed to calculate the image upper-left corner coordinates.Ground point A is given and its geographic coordinates are X, Y, Z as shown in Figure 3, and the coordinates of correspondence pixel a in the camera frame are x, y, −f ; the transformation from pixel a to A is shown (1) Camera frame: the z-axis is pointing up (aligned with gravity), the y-axis is defined by how the user has mounted the IMU (inertial measurement unit), and the x-axis is defined by how the user has mounted the IMU (2) IMU body frame: the z-axis points up through the roof of the vehicle perpendicular to the ground, the y-axis points out the front of the vehicle in the direction of travel, and the x-axis completes the righthanded system (out the right-hand side of th vehicle when facing forward) (3) Navigation frame: it is a local frame, which is tangent to the reference ellipsoid.Its origin is at the center of the reference ellipsoid, and the x-axis points to the equator intersection of the Greenwich meridian; the y-axis points to the equator with a 90 °meridian intersection and the z-axis through the North Pole.
(4) Earth-centered earth-fixed frame: a coordinate system X, Y, Z is selected which is earth-centered earth-fixed (ECEF); with the X-Y plane containing the equator, X intersects, in the positive direction, the Greenwich Meridian (from where the longitude is measured; longitude equals 0 degrees at Y equals zero); Z is parallel to the Earth's rotation axis and has a positive direction toward the North Pole, and Y is in the equatorial plane and perpendicular to X and completes a right-handed coordinate system; i.e., the cross-product of X and Y is a vector in the direction of Z.
In formula (4), R b c is the transformation matrix from the camera frame to the IMU body frame.R g b is the transformation matrix from the IMU body frame to the navigation frame.R e g is the transformation matrix from the navigation frame to the earth-centered earth-fixed frame.
In formula ( 5), H, P, and R indicate the heading pitch and roll angles, gained from the autopilot system.
In formula (6), L and B, respectively, indicate the longitude and latitude, which are outputs from the autopilot system.The rotation transformation matrix could be deduced according to the Euler transformation referred in [20][21][22].When x y − f T equals − w/2 h/2 −f T , the image upper-left corner coordinates could be obtained through formula (4).
When the UAV collects the images in the postearthquake areas, it flies about hundreds of meters above the ground and there is no signal blocking from the GPS receiver to the satellite.If the signal discontinuity problem happens, we could adopt the linear interpolation method to calculate the lost position and the attitude information within a short time interval.The UAV collects image 1 at time t 1 , and the autopilot records the position and the attitude information p 1 x 1 , y 1 , z 1 , roll 1 , pitch 1 , yaw 1 .It collects image 2 at time t 2 , and the autopilot records the position and the attitude information p 2 x 2 , y 2 , z 2 , roll 2 , pitch 2 , yaw 2 .However, it collects image i without recording the position and attitude information at time t i between t 1 and t 2 ; we can calculate the position and attitude information p i from the flowing formula: In the above formula (7), A i indicates the parameters x i , y i , z i , roll i , pitch i , yaw i and i is the image number.

The Transformation between Different Rotation Systems.
The remote sensing images collected based on the unmanned aerial vehicle rotate around their symmetry center theoretically, but the images georeferenced based on the world file rotate around their upper-left corner.Therefore, they have the different positions after the rotation.Figure 4(a) shows that the image rotates around the upper-left corner in different angles from position 1 to position 8, and Figure 4(b) indicates that the image rotates around the symmetry center according to the same angle form position 1 to position 8.
In Figure 4, when the image rotates around its upper-left corner and the image rotates around its symmetry center according to the same angle, the correspondent edges of images after rotation are parallel to each other.It indicates that the objects of the images have the same directions after the rotation.However, they have a translation gap between the images, in which we need to correct and improve the position accuracy.According to formula (2), after the image rotates around its symmetry center, the coordinates of the image are calculated as In formula (8), j is the pixel coordinate of the x direction, and i is the pixel coordinate of the y direction.θ is the rotation angle of the image rotating around the symmetry center.w is the width of the raw image, and h is the height of the image.When the image rotates around its symmetry center, i, j is calculated from − w/2 , h/2 .When the image rotates around its upper-left corner, i, j is calculated from 0, 0 .

Journal of Sensors
The translation amount caused by image rotation could be deduced from formula (8); the detailed form is as follows: In formula (9), Δx and Δy are, respectively, the translation amounts on the x and y directions which indicate the position deviation that the image rotates around its symmetry center relative to the image rotating around the upperleft corner.θ is the image rotation angle that the image rotates around its symmetry center.w is the raw image width, and h is the raw image height.The translation amount is closely related to the rotation angle, image width, and image height.

The Rapid Georeference Accuracy Improvement Based on the Reference Image
The UAV image rapid georeference accuracy is dependent on the position data accuracy which is the output from the autopilot system.Generally, different types of UAV have different GPS modules, and the accuracies of GPS modules are different, especially the GPS module without differential positioning function.In order to improve the georeference accuracy and time efficiency further, the whole georeference accuracy improvement method based on the  The feature extraction and matching could be up to subpixel accuracy.We do not discuss more details about the SURF features; the details could be referred in [23][24][25][26].
Since the reference image always covers a big area, the SURF features extracted from the UAV images to match the features extracted from the reference images will exhaust much time.To solve this problem, the features are extracted from the reference image in the given area which is initialized according to the boundary area of the georeferenced UAV images and match with the features extracted in the UAV image.When x equals − w/2 and y equals h/2, we put x, y into formula (4); the upper-left corner coordinates of the initially georeferenced UAV image will be acquired.When x equals w/2 and y equals to −h/2, we put x, y inputs into formula (4); the lower-right corner coordinates of the initially georeferenced UAV image will be acquired.Simultaneously, considering the initial georeference error δ, the searched feature area in the reference high-resolution image is as follows: Considering the time efficiency, the Gaussian probability model is employed in this paper to delete the wrong matched points.The distance from the feature in the UAV image to the feature in the reference image is D i i = 1, 2, 3, ⋯, n ; all the distances D i i = 1, 2, 3, ⋯, n between the matched features will be the same in the ideal conditions.All the distances D i between matched features will satisfy the Gaussian normal distribution, the mean of distance is μ, and the variance is σ; the probabilities of samples which drop into the areas μ − σ, μ + σ , μ − 2σ, μ + 2σ , and μ − 3σ, μ + 3σ are, respectively, 68.3%, 95.5%, and 99.7%, according to the Gaussian normal distribution model.If the distance D i drops out of the confidence interval, we could recognize them as wrong matched features and delete them from the matched feature list.In Figure 5, we use the SURF feature extraction and matching to achieve the UAV image matching with a google image.It fulfils the purpose of the position error calculation between the initial georeferenced image and the reference image.
The i th feature in the UAV image is given, and its geographic coordinates are X i , Y i .After the UAV image matches with the reference image, it gets the new geographic coordinates X i ′, Y i ′ .The UAV image georeferenced error was computed as follows:

Experiments and Analysis
Experiments adopt C# language combined with GDAL (Geospatial Data Abstraction Library) function to compile the rapid geographic reference software and achieve the UAV image rapid georeference.The rapid georeference software is installed on the IBM T430 computer as the experimental platform, with an Intel Core i5 2.60GHz CPU and a 4 GB size memory.7 Journal of Sensors www.feimarobotics.com/)was used to collect the image data as shown in Figure 6.The specifications of the UAV are shown in Table 1.The UAV was equipped with a SONY DSC-RX1R II microsingle camera with a 20 mm focal length, of which CCD size is 23.5 × 15.6 mm.The resolution of the collected image is 7952 × 5304.The UAV autopilot system can output the longitude, latitude, elevation, pitch angle, roll angle, and heading angle.The GPS module of the autopilot system is without data differential function.
Taking the UAV remote sensing image data obtained from the Danling County in Sichuan Province, as an example, the average flying height is 250 m.Finally, 1025 images are obtained.The single image size is about 18 MB, and the total image size is about 18 GB.The attitude and position data are shown in Figure 7.All the images achieved rapid georeference within 3 minutes.
In Figure 8, the UAV images achieved the georeference without considering the image rotation and the error correction.The UAV image rapid georeference meets the time requirement, but we could not get the ground object distribution situation of the entire flight area.It is difficult to quickly select suitable images for disaster information submission and the interest object monitoring.
In Figure 9, it indicates the result of UAV image rapid georeference based on the algorithm proposed in this paper.All the images get the spatial reference.If they are loaded in  If we adopt the software such as PhotoScan which will make the perfect result with a small stitching gap to generate the panoramic image, it will take more than three hours to generate the panoramic image.In the emergency response period, saving time is very important.If the emergency response commanders get the disaster information submitted through the rapid georeference, they will know the disaster location and make the correct rescue decision as soon as possible.In this situation, dozens of meter accuracy totally satisfy the emergency response requirements.And the panoramic image generated from PhotoScan could make the image map, which could be the next level image production and submitted to the emergency response commanders.

The Rapid Georeference Experiment Based on the
Quadrotor UAV.In this experiment, we use the quadrotor UAV (http://www.flightwin.com/) to collect images as shown in Figure 10.The specifications of the quadrotor UAV are shown in Table 2.The UAV equipped with the Sony ILCE-6000 camera, with a 20 mm focal length, of which CCD size is 23.5 × 15.6 mm.The resolution of the collected image is 6000 × 4000.
Taking the UAV remote sensing image data obtained from the earthquake site in Beichuan County, Sichuan Province, as an example, the average flight height of the UAV is 450 m.Finally, 226 images are obtained.The single image size is about 10 MB, and the total image size is about 2.2 GB.The attitude and position data are shown in Figure 11.Achieve rapid geographic reference of all image data in less than 1 minute.
From Figure 12, we could know the whole situation of the area.All the images are separate layers.For example, the building is damaged by an earthquake.We select the suitable image and submit it to the response commanders before the panoramic image generated from the PhotoScan software.It is very important in the early period of emergency response.According to the rapid georeferenced result, the commanders will arrange workers to investigate the disaster information.

Conclusions
In this paper, the method and model of UAV image rapid georeference are discussed in detail and the algorithm is implemented through C# language combined with GDAL.The rapid georeference software can achieve one flying sortie UAV image referencing in a short time.It provides important technical support for grasping the overall disaster information and monitoring the key objects in the disaster area.At the same time, it can generate a vector outline of each georeferenced image, and then the vector outline is overlaid on the reference image to quickly select the image containing the disaster information to submit.It has an important application value in the postearthquake response period.In the future work, the image rapid georeference for the UAV with differential GPS module will be studied to improve the localization accuracy and reduce the relative error between adjacent images.

2. 2 .
The Data Organization Form of Georeferenced UAV Images.Considering the processing efficiency of the UAV image rapid georeference, we do not implement the pixel resampling in the process of the rapid georeference but adopt the world file to organize the georeference parameters.The world file is a data organization format, designed and developed by ESRI, a Geographic Information System company.It comprises six parameters, which are organized in a file with the specified suffix name and text encoding.By this data organizing manner, the Geographic Information System software could automatically identify the georeferenced images.All of these parameters consist of the affine transformation matrix as flows.

3. 1 .
The Position Parameter Calculation.The image upperleft corner coordinates are key parameters in the UAV image rapid geographic reference based on the world file.

Figure 2 :Figure 3 :
Figure 2: The imaged georeference based on the world file.

Figure 4 :
Figure 4: (a) The image rotates around the upper-left corner, and (b) the image rotates around the symmetry center.

Figure 5 :
Figure 5: The SURF feature extraction and matching.

Figure 7 :
Figure 7: The attitude and trajectory of fixed wing UAV.

Figure 8 :
Figure 8: The rapid georeference without considering the angle information.

Figure 11 :
Figure 11: The attitude and trajectory of quadrotor UAV.

Figure 12 :
Figure 12: The image rapid georeference and selected damaged image submitted.

Table 1 :
The specification of the fixed wing UAV.

Table 2 :
The specification of the quadrotor UAV.