^{1}

^{2}

^{1}

^{1}

^{1}

^{1}

^{2}

In the existing vision-based autonomous landing systems for micro aerial vehicles (MAVs) on moving platforms, the limited range of landmark localization, the unknown measurement bias of the moving platform (such as wheel-slip or inaccurate calibration of encoders), and landing trajectory knotting seriously affect system performance. To overcome the above shortcomings, an autonomous landing system using a composite landmark is proposed in this paper. In the proposed system, a notched ring landmark and two-dimensional landmark are combined as an R2D landmark to provide visual localization over a wide range. In addition, the wheel-slip and imprecise calibration of encoders are modeled as the unknown measurement bias of the encoders and estimated online via an extended Kalman filter. The landing trajectory is planned by a solver as a convex quadratic programming problem in each control cycle. Meanwhile, an iterative algorithm for adding equality constraints is proposed and used to verify whether the planned trajectory is feasible or not. The simulation and actual landing experiment results verify the following: the visual localization with the R2D landmark has the advantages of wide localization range and high localization accuracy, the pose estimation result of the moving platform with unknown encoder measurement bias is continuous and accurate, and the proposed landing trajectory planning algorithm provides a continuous trajectory for reliable landing.

Micro aerial vehicles (MAVs) are highly agile and versatile flying robots. Recent work has demonstrated their capabilities in many different applications including but not limited to surveillance, object transportation, agriculture, and aerial photography. This work is focused on the specific task of a fully autonomous landing on a moving platform. Some work has focused on autonomous landing on stationary platforms (SPL systems) such as reference [

2D landmark localization: to detect the landing zone, most state-of-the art works exploit computer vision from onboard cameras. In those studies, 2D landmark localization is the most common approach for providing camera position relative to landing zone. Generally, the 2D landmark is arranged on the top of the moving platform, so its size is often strictly limited. To provide a large range of localization data under this limitation, researchers have adopted landmarks such as 2D codes, ring landmarks, or character landmarks [

Pose estimation of the moving platform: it is difficult to guarantee that the moving platform will be visible throughout the entire duration of the landing. To address missing visual information, state estimation and multisensor fusion methods have been introduced to predict the motion of the moving platform [

Online landing trajectory planning: for the MAV to be truly autonomous, the landing trajectory planning must be performed on the onboard processor in real time [

In this paper, a MAV system capable of autonomously landing on a moving target using mainly onboard sensing and computing is presented. The only prior knowledge about the moving landing target is its real-time measurement of the encoders. The proposed R2D-MPL system uses a composite landmark comprising a notch ring (NR) landmark and 2D landmark to provide accurate visual localization data on the moving platform over a wide spatial range. To deal with temporarily missing visual information, the unknown measurement bias of encoders caused by wheel-slip and imprecise calibration is taken into consideration in the target’s dynamical model, and the state of the moving platform is estimated online based on an extended Kalman filter (EKF). Meanwhile, the R2D-MPL system computes trajectories based on the energy necessary to execute an efficient landing, which takes into account the dynamic constraints of the MAV and optimizes the equality constraints of waypoints through an iterative algorithm. The proposed system is validated by simulation as well as in real-world experiments using low-cost, lightweight consumer hardware. The experimental results show that the approach can realize the autonomous and reliable landing of the MAV when the encoders have measurement bias.

The rest of this paper is organized as follows. Section

Landing on a moving platform is more complex than landing on a stationary platform. Recognition errors for a 2D landmark, inaccurate motion models, and maneuvering by moving platforms will seriously affect landing results. To overcome the above problems, a general and hierarchical R2D-MPL system is proposed in this paper, which is shown in Figure

Flow chart of the R2D-MPL system.

The R2D-MPL system shown in Figure

Ring landmarks and 2D landmarks are the most commonly used landmarks in visual localization systems. A ring landmark has the advantage of providing good estimation results from larger distances since it uses the projection information of the whole contour. However, it has the problems of a lack of freedom and singular solution. A 2D landmark allows estimation of the camera’s 6Dof pose, but its localization accuracy is affected by the vertex detection result; therefore, it is only suitable for close-range localization. To accurately estimate the MAV’s pose, a composite R2D landmark is designed and placed on the top of the moving platform. The R2D landmark comprises a notched ring (NR) landmark and a 2D landmark; thus, the R2D landmark has the potential to realize precise and continuous localization in a wide range space. As shown in Figure

R2D landmark.

For the R2D landmark recognition, the original image is processed by the 6

The elliptical oblique cone [

In equations (

To obtain the nonsingular solutions, the following two kinds of constraints are added. First, two simple constraints (

Ideally, the 2D landmark and NR landmark can simultaneously provide the 6Dof camera pose. The localization of the NR landmark is obtained based on equations (

Pose estimation of the moving platform can improve the robustness of the MPL system since the visual localization results are not always available. To obtain continuous and accurate pose estimation of the moving platform, a precise motion model and appropriate multisensor fusion algorithm are necessary. In this section, a general motion model for the MPL system is proposed, which considers the unknown encoder measurement bias. Meanwhile, the precise and continuous position, velocity, and orientation of the moving platform are estimated online based on the extended Kalman filter (EKF). Here, a three-wheeled omnidirectional car is used as the moving platform, and its coordinate system is shown in Figure

Description of the moving platform coordinate system.

As seen from Figure

Based on equations (

It makes the proposed R2D-MPL system more general to introduce other sensors to measure

It makes the proposed R2D-MPL system more general to handle the transmission delay of encoder data with the ring queue and timestamp methods [

A nonlinear function is used to describe the motion model of the moving platform in this paper.

Define

The R2D landmark localization result and the encoder data are utilized to perform the measurement update here.

Define

The residual vector

The Kalman filter gain can be calculated as follows.

Finally, the state estimation and covariance matrix of the moving platform are given as follows.

To achieve the precise, continuous, and minimum-energy cost landing process, an online two-stage landing trajectory planning algorithm based on the minimum jerk rule is proposed in this section, which includes the visual tracking stage and planning landing stage. In the visual tracking stage, the MAV approaches the circular boundary

Top view of the landing process.

The visual tracking method is utilized to guide the MAV to approach the boundary

As shown in Figure

The waypoint

Above all, to generate a smooth and minimum-energy landing trajectory, an online trajectory planning algorithm is proposed here. For simplification, the three dimensions of the trajectory are planned independently. The fifth-order polynomial is used to describe the landing trajectory with the polynomial parameter vector

Several equality constraints can be established considering the waypoint constraints and the continuity of the position, velocity, and acceleration at the waypoint

Therefore, the proposed IAEC algorithm first plans the landing trajectory based on equation (

1: Initialize:

2:

3: Calculated

4:

5: Define

6:

7: Add new position equality constraints:

8:

9:

10: Add new velocity equality constraints:

11:

12:

13: Relaxation of adjacent waypoints constraints:

14: Extended matrix

15:

16:

17:

To verify the proposed R2D-MPL system, an X450 quadrotor is used in this section. The quadrotor uses an 8-inch propeller and four 980 kV brushless motors as the drive system. Its weight is 1.35 kg, and the flight time is 15 minutes. The proposed R2D landmark localization algorithm, pose estimation algorithm, and trajectory planning algorithm are tested on the onboard ODROID-XU4 processor. The height

The localization performance of the R2D landmark is verified, and the result is shown in Figure

Localization results of the NR landmark, 2D landmark, and R2D landmark.

Localization result

Localization error

In addition, an experiment comparing the localization performance of 2D, NR, and R2D landmarks with the same size is proposed, the results of which are given in Table

Landmark comparison experiment results (landmark size is 0.7 m).

Landmarks | Localization range |
Localization error |
---|---|---|

2D landmark | 1.2–4.5 | 0.35–1.2 |

NR landmark | 1.5–6.0 | 0.5–0.6 |

R2D landmark | 0.5–6.0 | 0.1–0.6 |

To verify the pose estimation performance of the proposed system (method 1), the following simulation experiments are designed. The Gazebo and ROS simulation software are used to build the experimental environment. The quadrotor hovers at the coordinate point (0,0,3). Meanwhile, the moving platform runs along a circular trajectory centered at (0,3,0) and radius of 3 m at a speed of 1 m/s. In addition, the pose estimation algorithm (method 2) proposed in reference [

Estimate results of the moving platform.

Position estimate result

Velocity estimate result

Figure

In addition, the position estimate error is shown in Figure

Position estimate error.

To simulate wheel-slip of the moving platform, random walk noise is added to the encoders’ measurements. The estimated result of the encoder measurement bias is given in Figure

Encoder measurement bias estimate result.

First, the following experiment is used to verify the IAEC algorithm. Five desired waypoints, [2, 1, 2], [3, 5, 3], [5, 2, 5], [10, 8, 4], and [12, 2, 5], are given in this experiment. In addition, the velocities and accelerations at the start and end points are zero. The proposed IAEC algorithm is also compared with the algorithm using only

Trajectory planning experiment.

As seen in Figure

To verify the performance of the proposed R2D-MPL system, several simulation landing experiments are performed, which are compared with the system proposed in reference [

Landing comparison experiment results (20 experiments).

Motion model | Parameters | Bias | AVT | AVT | LR/SR | LR/SR |
---|---|---|---|---|---|---|

(Encoder 1) | (R2D-MPL) | (Method 3) | (R2D-MPL) | (Method 3) | ||

Line | 0.00 m/s | 12.4 s | 14.2 s | 0.18 m/90% | 0.25 m/85% | |

Line | 0.00 m/s | 13.2 s | 15.7 s | 0.21 m/85% | 0.27 m/75% | |

Line | 0.15 m/s | 13.6 s | 22.4 s | 0.22 m/80% | 0.38 m/25% | |

Line | 0.30 m/s | 14.1 s | 25.1 s | 0.25 m/85% | 0.40 m/5% | |

Circle | 0.00 m/s | 15.7 s | 23.2 s | 0.21 m/80% | 0.42 m/40% | |

Circle | 0.00 m/s | 17.4 s | 27.4 s | 0.24 m/85% | 0.41 m/30% | |

Circle | 0.15 m/s | 18.1 s | Fail | 0.28 m/75% | Fail | |

Circle | 0.30 m/s | 19.5 s | Fail | 0.32 m/80% | Fail |

As seen in Table

Furthermore, the actual landing experiments for linear motion and circular motion are tested, and the video of the experiment can be found at

Figure

Linear motion landing experiment.

Landing experiment

3D trajectory

Circular motion landing experiment.

Landing experiment

3D trajectory

In this paper, a MAV composite landmark guidance system capable of autonomously landing on a moving platform using only onboard sensing and computing is presented. This system relied on state-of-the-art computer vision algorithms, detection and motion estimation of the moving platform, and path planning for fully autonomous landing. No external infrastructure, such as motion-capture systems or an ultra-wideband system, is needed. The only prior knowledge about the moving platform is its real-time measurement of the encoders; meanwhile, the unknown measurement biases of encoders are considered in the dynamical model of the moving platform. The proposed system is validated by simulation as well as with real-world experiments using low-cost and lightweight consumer hardware. Finally, the proposed approach achieved a fully autonomous MAV system capable of landing on a moving target with wheel-slip and bias in encoder measurements, using only onboard sensing and computing and without relying on any external infrastructure.

The data used to support the findings of this study are available from the corresponding author upon request.

The authors declare that they have no conflicts of interest.

This work has been supported by the National Natural Science Foundation (NNSF) of China under Grants 61603040 and 61433003, Yunnan Applied Basic Research Projects of China under Grant 201701CF00037, and Yunnan Provincial Science and Technology Department Key Research Program (Engineering): 2018BA070.