^{1}

^{2}

^{3}

^{1}

^{1}

^{2}

^{3}

Large-screen human-computer interaction technology is reflected in all aspects of daily life. The dynamic gesture tracking algorithm commonly used in recent large-screen interactive technologies demonstrates compelling results but suffers from accuracy and real-time problems. This paper systematically addresses these issues by a switching federated filter method that combines particle filtering and Mean Shifting algorithms based on a 3D sensor. Compared with several algorithms, the results show that the one-hand and two-hand large-screen gesture tracking based on the switched federated filtering algorithm work well, and there is no tracking failure and loss of target. Therefore, the switching federated tracking positioning algorithm can be well applied to the design of human-computer interaction system and provide new ideas for future human-computer interaction.

The definition of human-computer interaction technology is based on a certain software program through the corresponding input and output devices, and the organic combination of computer and human operations is employed to realize the related technology of interactive communication [

Hand gesture recognition methods based on computer vision are mainly divided into two categories: one is an analysis method based on a 3D model, and the other is based on a 2D image. The analysis method with the 3D model needs to establish a parametric model that describes gestures. Because it can provide 3D data, a more accurate gesture model can be established. However, the method has many parameters and high computational complexity, and it is difficult to achieve real-time results in current technologies. Also, with the two-dimensional image method, it is mainly to analyze the performance of the image and extract the effective hand features for identification. Because of the loss of the third 3D space information, the gesture model cannot be effectively established for the described features. Besides, its parameters are less, so it can meet the requirements of real-time processing. These two methods can hardly guarantee the balance between system parameters and real-time performance.

This paper proposes a large-screen interactive imaging system with switching federated filter method based on 3D sensor and validates the tracking results with the independently developed gesture position tracking platform. The main innovations of this paper are as follows: (1) An improved switching federated filter algorithm combining particle filtering and Mean Shift is introduced into the field of large-screen gesture tracking to track dynamic gestures; (2) the self-developed gesture tracking platform and 3D interactive software are combined to observe the gesture tracking effects; (3) the large-screen single-hand and two-hand gesture tracking based on the switching federated filtering algorithm work well and there is no phenomenon of tracking failure and losing the target.

The structure of this paper is as follows: An improved switching federated filter algorithm combining particle filtering and Mean Shift is employed to track and locate the dynamic gestures in Section

Common gesture tracking algorithms cannot handle the dynamic gesture tracking problem in complex environments. This paper has developed an improved switching federated filter algorithm that combines particle filtering and Mean Shift algorithm. In the case of slow movement of the human hand, after the uniform displacement of the particles, the particles drift to the dynamic gesture area except for a small number of particles. The gesture position can then be obtained without subsequent prediction of the particles, so that the running time of algorithm can be saved. The average value of the particles will drift when the movement of the hand is fast or there is occlusion. If the region of the gesture cannot be detected, the particle will return to the condition before the drift, waiting for the algorithm to perform corresponding processing on the next frame. Therefore, how to select the corresponding filtering algorithm under different conditions is the key problem. In this paper, the switching system scheme is introduced into the filter for the first time, and applied to the large-screen interactive imaging system.

A switching system consists of a series of sequential or discrete differential equations subsystems and switching rules or a switching strategy [

The current common particle filter implementation framework is based on resampling and sequential importance sampling, which can be called sampling importance resampling filter [

Based on the Bayesian posteriori estimation and state transfer equation at the previous moment to achieve the purpose of updating the particle state, the predicted distribution

Based on the latest observed information

where

Based on the principle of identity distribution, the resampling operation is performed on the updated particle combinations after the weights update operation is completed. A new set of particles with most of the particle weights can then be obtained. The number of times

Meanwhile, the Mean Shift algorithm as the other filter subsystems is a process that uses nonparametric density estimation, which was used to perform iterative search based on feature spatial gradient directions and then obtain sample data with local density maximum [

For n sample points

where

The form of Mean Shift has a fundamental problem: in the region of

Among them,

The above Mean Shift vector can be rewritten as

Mean Shift vector

where

Let

Among them, the second square brackets mean the Mean Shift vector, which is proportional to the probability density gradient. Mean Shift vector correction results are as follows:

Considering

The process after the integration of the specific algorithm is as follows.

In the initial frame, all the particles are distributed in the gesture area according to the Gaussian distribution.

In the next frame, all particles are mean shifted, the Pasteurian after shifting is taken as the weight of the particle, the weights are normalized, and the number of effective particles

Determine whether the number of effective particles is greater than the threshold

If the effective number of particles is greater than the set threshold

Define the seed as the center of the region calculated in

If the effective number of particles is less than the set threshold

Use Mean Shift algorithm on all particles, the Pasteurian after shifting is taken as the weight of the particle, the weights are normalized, and the number of the effective particles

If the effective number of particles is greater than the set threshold

If the effective number of particles is less than the set threshold

If the weight of some particle increased, it means that a small part of the particles have spread to the gesture area. Resample the significance of the particles and then go to

The algorithm flowchart is shown in Figure

The algorithm flowchart with switching federated filter method.

In this paper, two experiments are designed to verify the gesture tracking and positioning effect of the proposed switched federated filtering algorithm. The first experiment is based on the self-developed human-computer interaction positioning software platform and observes the filtering effect of the switching federated filter algorithm. The second one is the gesture tracking and positioning experiment based on 3D interactive software. It compares the switching federated filter algorithm with several algorithms and observes the performance of the different algorithms in tracking accuracy and tracking time.

The self-developed human-computer interactive positioning software platform system generates an interactive operation interface through a projector on an arbitrary wall or curtain. The Kinect sensor is generally fixed at a range of 1-3 meters directly above the screen and 5-6 centimeters far from the wall surface [

The self-developed human-computer interactive positioning software platform.

Figures

Gesture tracking and positioning effect without filter algorithm.

Gesture tracking and positioning effect with the federated tracking filter algorithm.

From Figures

There are many commonly used target tracking algorithms. This paper uses the typical Cam-shift algorithm to compare the actual application with the algorithm on the large screen. Cam-shift algorithm is the commonly used gesture tracking algorithm, which is good for tracking solid objects in a black-and-white background. However, the contrast between the background color and the target is not obvious, and the tracking effect is poor [

This section compares the Cam-shift algorithm with the switching federated tracking algorithm presented in the simulation. Gesture trajectories tracked by two different algorithms are transmitted to the computer. The mouse function of trace tracking in the 3D interactive software can convert the gesture trace map into the trace of the mouse origin and display it on the mosaic screen. The tracking effect of different algorithms can be observed according to the mouse trajectory on the splicing screen. The interactive software based on the software platform can operate the position of the gesture as the mouse position, so the mouse track is the gesture tracking trajectory. Figure

The one-hand gesture tracking effect based on switching federated filter algorithm.

The computer screenshot of one-hand gesture tracking based on switching federated filter algorithm.

The one-hand gesture tracking effect based on Cam-shift algorithm.

The computer screenshot of one-hand gesture tracking based on Cam-shift algorithm.

As can be seen from Figure

The hands gesture track trajectory diagrams are shown in the following. Figure

The hands gesture tracking effect based on switching federated filter algorithm.

The computer screenshot of hands gesture tracking based on switching federated filter algorithm.

The hands gesture tracking effect based on Cam-shift algorithm.

The computer screenshot of hands gesture tracking based on Cam-shift algorithm.

To verify the accuracy and tracking time at different sampling points based on the federated tracking algorithm and several commonly used algorithms [

Table

Relationship between target tracking accuracy (%) and algorithms.

MP | Cam-shift | Particle Filter | Velocity-Adaptive Particle Filter | Federated Filter Tracking |
---|---|---|---|---|

20 | 4.8 | 5.2 | 5.8 | 7.1 |

60 | 52.5 | 55.3 | 58.6 | 63.1 |

100 | 82.5 | 84.7 | 89.1 | 91.7 |

200 | 90.1 | 91.2 | 92.7 | 94.9 |

Table

Relationship between tracking time (ms) and algorithms.

MP | Cam-shift | Particle Filter | Velocity-Adaptive Particle Filter | Federated filter tracking |
---|---|---|---|---|

20 | 18.3 | 16.5 | 15.7 | 14.7 |

60 | 30.2 | 27.4 | 23.2 | 16.9 |

100 | 48.5 | 39.2 | 31.5 | 21.8 |

200 | 82.8 | 64.3 | 52.1 | 26.2 |

From Tables

Recent gesture tracking algorithms of large-screen interactive imaging systems are faced with inaccuracies and low real-time performance. This paper proposed a switching federated gesture tracking algorithm combining particle filtering and Mean Shift algorithm. The federated filter algorithm first averages the particles to make most of the particles drift into the scope of the gesture region. The subsequent step of particle prediction can be omitted, thus saving the running time of the algorithm. In the aspect of experimental simulation, this paper first compares the effect graph before and after the filtering with the self-designed software platform and shows that the switching federated filter has good effect in removing the positioning interference. Then, the trajectory display function is invoked in the 3D interactive software. Compared with the Cam-shift algorithm for the gesture tracking image in actual large screen, whether it is tracking one-hand gesture or two-hand gesture, none of the federated filter algorithms will fail to track or lose the target. Therefore, the switching federated filter algorithm can solve the problem of low accuracy and real-time performance in dynamic gesture tracking. This algorithm effectively reduces the impact of complex environments on tracking effects, which can be applied to interactive imaging systems such as large screens.

The data used to support the findings of this study are included within the article.

The authors declare that they have no conflicts of interest.

The work is supported by the National Natural Science Foundation of China (61403268, 61873176); Natural Science Foundation of Jiangsu Province, China (BK20181433); Fundamental Research Funds for the Central Universities (30918014108); Natural Science Fund for Colleges and Universities in Jiangsu Province (16KJB120005); Open Fund for Collaborative Innovation Center of Industrial Energy-Saving and Power Quality Control, Anhui University (KFKT201806).

_{∞}control for discrete-time switched nonlinear systems with time delay