Panoramic imaging is information-rich, low-cost, and effective. In panoramic image acquisition, unmanned aerial vehicles (UAVs) have a natural advantage that owes to their flexibility and relatively large observation ranges. Using a panoramic gimbal and a single camera may be the most common means of capturing gigapixel panoramas. In order to manage the constraints of UAV power and facilitate the use of a variety of camera lenses, an effective and flexible method for planning UAV gigapixel panorama acquisitions is required. To address this need, a panoramic image acquisition planning method is proposed in this paper. The method defines image overlaps via a ray casting procedure and then generates an acquisition plan according to the constraints of horizontal and vertical overlap thresholds. This method ensures the completeness of the panorama by maintaining the overlap between adjacent images. Two experiments, including simulated and field cases, were performed to evaluate the proposed method through comparisons with an existing panorama acquisition plan. Results showed that the proposed method can capture complete panoramas with fewer images.
A panorama is a single wide-angle image of the environment around a camera [
Current unmanned aerial vehicles (UAVs), particularly multirotors, are relatively inexpensive and have a high degree of mobility and maneuverability. UAVs have a natural advantage in panoramic image acquisition, owing to their flexibility and large observation range. Combining UAVs and panoramic imaging makes macroscopic landscape analysis possible [
Multiple methods and tools have been developed for stitching gigapixel panoramas. The pipeline starts with the correction of lens distortion and interior elements using a pinhole camera model [
Compared with the number of studies on image stitching, there are fewer studies on capturing gigapixel panoramas. Camera settings, rotation axes, and image overlap have been discussed in previous studies. Krishnan and Ahuja [
In this paper, a gigapixel panorama acquisition planning method for multirotors is proposed. This method is based on a ray casting image overlap calculation method. The ray casting-based overlap restricts the panoramic image capture planning. The plan is designed for panoramic gimbals that can rotate horizontally and vertically; thus, it is generated considering pan and tilt angles separately. Two experiments, including a simulated case and a field case, are proposed to compare the planning result with existing panoramic acquisition plans. The image stitching result shows that the proposed acquisition plan can ensure the completeness of the panorama while using fewer images.
A spherical panorama is represented using a spherical coordinate in the horizontal angle and vertical angle. It is an equirectangular projected sphere [
We assume that the center of perspective remains during the capture of a panorama, and that the camera only changes its pan and tilt angles to capture different views. The pan and tilt angles are denoted by
Axis and rotation of panoramic imaging.
It is assumed that an image
In equation (
The coordinates of the point on the image,
The pixels that fall within the overlap of two images
Ray casting diagram of two overlapping images.
Note that the ground object, the projection on the image, and the center of perspective are collinear, as shown in
Let
The input of equations (
In practice, the accuracy and stability of a panoramic imaging system may affect the orientation of the image. Thus, the pixels corresponding to the same ground point on two adjacent images may not be strictly collinear. The collinear condition of equation (
The value of
An optimal planning process for UAV panoramic image acquisition should ensure the overlap between adjacent images with a minimum number of images. The overlap can be calculated via equations (
A panoramic gimbal with a single camera should allow exposures at
The input for tilt angle determination (see Algorithm
The determination of pan angles is similar to that of tilt angles (see Algorithm
This section discusses simulation and field experiments designed to evaluate the proposed panoramic image acquisition planning method in terms of completeness and number of images. In order to evaluate the range of adaptation of the proposed method, it was tested under three different camera and lens scenarios. The 35 mm equivalent focal length ranged from 30 mm to 90 mm. The proposed method was compared with the plan by HDRpano [
1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14:
1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15:
Panoramic acquisition plans for comparison.
Scenario # | Camera and equivalent focal length | Plan | Specifications tilt angle/number of images | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | DJI X5s, 30 mm | Proposed, 25 images | -30° |
5° |
39° |
70° |
90° | ||||||
HDRpano, 28 images | -30° |
0° |
30° |
60° |
90° | ||||||||
2 | DJI X5s, 90 mm | Proposed, 155 images | -30° |
-18° |
-5° |
7° |
20° |
32° |
43° |
55° |
66° |
78° |
90° |
HDRpano, 161 images | -30° |
-18° |
-6° |
6° |
18° |
30° |
42° |
54° |
66° |
78° |
90° | ||
3 | SONY A6000, 50 mm | Proposed, 49 images | 0° |
21° |
40° |
57° |
75° |
90° |
The proposed plan was generated with both overlap constraints (
Moreover, it is often postprocessed. Thus, the completeness
To avoid the influence of the platform, all scenarios were simulated before the field experiment. The synthetic images were rendered from an existing panorama shown in Figure
Existing panorama for synthetic image rendering.
Panorama stitching completeness of the simulation case.
Scenario # | Plan | Completeness | Number of images |
---|---|---|---|
1 | Proposed | 100% | 25 |
HDRpano | 100% | 28 | |
2 | Proposed | 100% | 155 |
HDRpano | 99.97% | 161 | |
3 | Proposed | 100% | 49 |
Considering the number of images of both plans in all scenarios, the proposed plan needs three fewer images in Scenario 1 and six fewer images in Scenario 2. As shown in Table
Incomplete panorama from HDRpano plan in Scenario 2 of the simulation case.
In the field case, all plans in three scenarios were evaluated. Two different platforms were selected to evaluate the panorama acquisition plans. The platform used in Scenario 1 and 2 was a DJI Inspire 2 quadcopter with an X5s camera gimbal (M4/3 system); in Scenario 3, a custom ArduPilot [
Panorama stitching result from the field case. (a) Proposed plan of Scenario 1. (b) HDRpano plan of Scenario 1. (c) Proposed plan of Scenario 2. (d) HDRpano plan of Scenario 2. (e) Proposed plan of Scenario 3.
Panorama stitching completeness of the field case.
Scenario # | Plan | Panorama size (pixel) | Completeness | Number of images |
---|---|---|---|---|
1 | Proposed | 28520 |
100% | 25 |
HDRpano | 28524 |
100% | 28 | |
2 | Proposed | 88280 |
100% | 155 |
HDRpano | 88340 |
94.13% | 161 | |
3 | Proposed | 56096 |
100% | 49 |
In Figure
Figure
Enlarged views of gigapixel panoramas planned by the proposed method. (a) Scenario 2. (b) Scenario 3.
In this paper, a gigapixel panoramic image acquisition planning method for UAVs is proposed. This method is based on a ray casting overlap constraint, which ensures panoramic stitching with sufficient overlap between images. During generation of the panoramic acquisition plan, pan and tilt angles are considered separately; the resulting plan can be executed by three-axis gimbals mounted on multirotor UAVs. The proposed method was compared with existing panoramic acquisition plans in different scenarios with equivalent focal length ranged from 30 to 90 mm. Results showed that the proposed method could acquire complete and seamless gigapixel panoramas with fewer images than existing plans.
The novelty of the proposed method is that the ray casting overlap is closely related to the panoramic image stitching process. A relatively low overlap (15%) can fulfil the panoramic stitching. On the other hand, the proposed method has a wide range of adaptation. It can generate panoramic acquisition plans for arbitrary perspective cameras and lenses. Its application is not limited to drones—it can support the planning of panoramic gimbals mounted on the ground or manual acquisition as well. The panorama captured by the proposed method with 90 mm equivalent focal length contains over 3.8 gigapixels. Moreover, the spatial resolution on the buildings near the skyline is higher than 1 cm. This type of panorama contains more information and has a wider range of applications in surveillance. In the proposed method, the overlap constraints
The data used to support the findings of this study are available from the corresponding author upon request.
The authors declare that there are no conflicts of interest.
This study was supported by the National Natural Science Foundation of China (grant No. 41771481) and the National Key R&D Program of China (grant No. 2018YFF0215304). The authors would like to thank Mr. Chen’guang Xu and Mr. Doudou Zeng for the help with the experiment.