We propose a systematic framework for Intelligence Video Surveillance System (IVSS) with a multicamera network. The proposed framework consists of low-cost static and PTZ cameras, target detection and tracking algorithms, and a low-cost PTZ camera feedback control algorithm based on target information. The target detection and tracking is realized by fixed cameras using a moving target detection and tracking algorithm; the PTZ camera is manoeuvred to actively track the target from the tracking results of the static camera. The experiments are carried out using practical surveillance system data, and the experimental results show that the systematic framework and algorithms presented in this paper are efficient.
Moving target detection and tracking has a variety of applications in the field of computer vision, such as intelligence video surveillance, motion analysis, action recognition, environmental monitoring, and disaster response. Normally, it is quite easy and intuitive for humans to see and track targets and recognize their actions. However, establishing an automatic system without any intervention by humans is very challenging. Especially, as the size of the camera network grows with the development of the safe and smart city, it becomes infeasible for human operators to manually monitor multiple video streams and identify all events of possible interest, nor even to control individual cameras in performing advanced surveillance tasks, such as actively tracking a moving target of interest to capture one or more close-up snapshots. Therefore, an important task of the Intelligence Video Surveillance System (IVSS) is to design multicamera sensor networks capable of performing visual surveillance tasks automatically or at least with minimum human intervention. The design of an autonomous visual sensor network as a problem in resource allocation and scheduling can be found in [
To detect the moving targets from video frames of the static cameras, one of the widely used algorithms is background subtraction approach [
Moving target tracking is an important processing step in the field of computer vision and has been widely applied in some practical applications, such as video surveillance [
In this paper, we focus on the real time surveillance system with a multicamera network, which includes static and PTZ cameras, and the control system of active cameras. The target detection and tracking is done by fixed cameras using a moving target detection and tracking algorithm. Target coordinates are transformed to appropriate pan and tilt values using geometrical transformation, and then camera is moved accordingly. The contribution of this paper lies in that we design the real time control strategy of active cameras based on the target information obtained by detection and tracking algorithms.
The paper is organized as follows. The system framework is presented in Section
The work presented in this paper originates from a research project on video surveillance applications in the Digital Navigation Center (DNC) at Beihang University. The primary goal of the project lies in the development of an IVSS platform. Intelligence video surveillance in a large or complex environment requires the use of distributed multiple cameras. Since the focal length of static cameras is fixed, they cannot be used to realize some advanced surveillance tasks, such as capturing high-quality videos of moving targets of interest, actively tracking one or more moving targets of interest, and capturing close-up image. For this reason, plenty of researches have been dedicated to designing the combination of a PTZ camera with multiple PTZ or fixed cameras in a master-slave manner to complete some practical tasks [
The structure diagram of IVSS with a multicamera network.
In this paper, we focus on the design and application of a practical IVSS with a multicamera network which consists of low-cost static and PTZ cameras as well as algorithms. The low-cost static cameras are placed at the perimeter, indoor and outdoor areas, and used to realize targets detection and tracking by using moving target detection and tracking algorithm.
An experiment is carried out by using the video data. The Gaussian mixture model [
The experimental results are shown in Figures
The target detection results of video data 1.
Original video
Gaussian mixture model
Random background model
The target detection results of video data 2.
Original video
Gaussian mixture model
Random background model
The target tracking results of video data 1.
The target tracking results of video data 2.
The feedback signal is unavailable for low-cost PTZ cameras which can only implement one instruction within a certain time interval. In addition, the relationship between time and the variety of pan, tilt, and zoom is indeterminate. In order to solve this problem, we propose a PTZ control algorithm based on the target information feedback. The principle diagram of the acquisition of feedback signal is illustrated in Figure
The principle diagram of the acquisition of feedback signal.
The feedback signal of the PTZ control algorithm based on the target information feedback is obtained by computing the distance (e.g., the horizontal direction distance
When calculating the offsets of the centroid of target (COT) to the center of image (COI)
Based on the rotational speed of the PTZ camera, we adopt a linear approximation to map the relationship between the speed and the central offsets
The distance between COT and COI is chosen as the feedback and the corresponding up-down, left-right, and zoom in-out control instructions are sent according to the calibrated rotational speed.
In order to adjust the target to the COI in the first frame, after the position of the interesting target is obtained, the average speed of the target can be obtained by its historical moving information as follows:
The position
The state vector and observation vector for the Kalman filtering can be represented as [
According to the result of (
Regarding the zoom control of a PTZ camera, in order to alleviate the difficulties of detection and tracking in the process of rotation control due to the changing size of targets, we first realize the P/T rotation control of the PTZ camera and then realize zoom control only if the distance between the COT and COI is less than a predefined threshold.
In the process of the zoom control, the size of targets may change intensively if the camera zooms intensively. It brings great challenges for the algorithm of target matching and tracking. In order to solve the problem of zooming, we adopt a gradual type of control strategy. The control signal is sent every time in the minimal unit and the control process is repeated until the zooming time is satisfied. The feedback signal is computed by
During the continuous frame tracking, PTZ will adopt a slightly adjusted tracking plan and recalculate the shift of the target
The rotating direction of the PTZ camera.
Parameters | Rotating direction |
---|---|
|
Left |
|
Right |
|
Up |
|
Down |
Once the system sends a control instruction, the PTZ will respond within a certain interval. A whole package of the PTZ control needs 3 instructions at most and the response time is about 40 ms. Hence, the PTZ tracking system can be run in real time.
The control algorithm is tested in this paper. The parameters of the PTZ camera are listed in Table
The parameters of PTZ camera.
Parameters | Performance |
---|---|
Image sensor |
|
Effective pixels | PAL: 752 (H) × 582 (V) |
Focal length | 3.5~98 mm, 28× |
Zoom speed | Approx. 1.7 s (optical wide-tele) |
Angle of view | 55.8~2.1° (wide-tele) |
Aperture range |
|
Pan range | 360° endless |
Pan speed | Pan manual speed: 0.1°~300°/s |
Tilt range | −5°~185° (auto mirror) |
Tilt speed | Tilt manual speed: 0.1°~240°/s |
Number of preset | 256 |
The control results from zoom = 7 to zoom = 28.
Frame 42
Frame 48
Frame 55
Frame 66
The tracking result of the moving objects by controlling continuously PTZ camera.
Frame 14
Frame 15
Frame 17
Frame 20
From the experimental result, we can find that the zooming is smooth and the visual effect is in accordance with the law of human vision. The PTZ control can guarantee the camera to rotate with the moving target and keep the target in the center of the field of view. In the control process of the PTZ camera, the performance of detection and tracking algorithm strongly affects the result. If the detection and tracking algorithm performs unsatisfactorily, one will lose the target, which makes the PTZ camera have no feedback for the sampled video that hinders the control for the PTZ camera.
The system presented in this paper is tested, and the parameters of the PTZ cameras are shown in Table
In the area of surveillance, we set up some important regions, entrances and exits, and design a joint tracking system consisting of the PTZ cameras and static cameras. The regions of entrance and exit are the regions where the target arrives and departs. In order to track targets in the first time, those regions are set as the initial regions for the PTZ camera, as illustrated in Figure
The placement of static and PTZ cameras.
The area of surveillance is between the office building A and the wall. The regions “a,” “b,” “c,” “d,” and “e” are the regions covered by the static cameras (the corresponding number of cameras is camera 1, camera 2,…, camera 5), where region “a” is the start of the road which connects the gate and other roads and also the region that targets must cross when they are entering or departing. Therefore, the region “a” is set as the entrance region and is set as preset 1 with the initial preset of the PTZ camera. The region “c” is more important than other regions, and thus it is set as the important preset, that is, preset 2. The important preset possesses a higher monitoring authority than the other preset.
When a target enters the area of surveillance and turns up in the entrance region “a,” the static camera covering region “a” will detect the target and track it. Meanwhile, a “Call initial preset 1” instruction will be sent to the control system of the PTZ camera. The PTZ camera will turn to the initial preset 1 and the target will be actively tracked by the active control algorithm; following that the channel for the instruction of “Call initial preset 1” will be cut off to prevent the circumstances of unclear targets. When a target enters region “c,” the static camera covering region “c” will be in charge and a “Call preset 2” instruction will be sent to the control system of the PTZ camera. The PTZ camera will turn to preset 2 and the target will be actively tracked.
The relay tracking results of a single walking man in cameras 1 and 2 are illustrated in Figure
The continuous tracking results between camera 1 and camera 2.
The continuous tracking results between camera 2 and camera 3.
When a target enters the entrance region of the surveillance area, the static camera will detect the target and the PTZ camera will be adjusted from patrol state to initial preset 1. When the target turns up in the view of camera 3, namely, the important region, it will be detected, and corresponding instructions will be sent. The PTZ camera will be shifted to preset 2. The feedback instruction will be formed by the target information and the PTZ camera will be controlled to track the targets. The test result is shown in Figure
The active tracking results of PTZ camera.
Tracking target for static camera
Tracking target into important resign
PTZ automatic patrolling
PTZ from patrolling mode to preset 2
Frame 512 for PTZ camera tracking
Frame 879 for PTZ camera tracking
Frame 1143 for PTZ camera tracking
Frame 1866 for PTZ camera tracking
In this paper, the comprehensive design and implementation of the IVSS platform based on a multicamera network were presented. The system is composed of the low-cost static and PTZ cameras, the target detection and tracking algorithms, and the low-cost PTZ camera feedback control algorithm based on target information. The target detection and tracking is done by static cameras using a moving target detection and tracking algorithm; the PTZ camera is commanded to track actively the target from the tracking results of the static cameras, and the target information is transformed to the appropriate pan and tilt values using the geometrical transformation, such that the camera is moved accordingly. The test results of the target detection and tracking, active target tracking algorithm, and multicamera target tracking system were reported. Although the development of the multiple target active tracking based on a multicamera network is still challenging when there are more targets to be monitored in the scene than PTZ cameras, we believe that the developed low-cost PTZ control algorithm and scheduling strategy can be widely applied to IVVS and extended to other visual analysis systems.
The multicamera system that can realize the multitarget tracking and active target tracking was verified by a practical IVVS. In addition, the low-cost PTZ camera control algorithm and scheduling strategy were preliminary realized too. However, the algorithm of controlling and scheduling multiple PTZ cameras is undeveloped. Further research works will be required to develop and test these algorithms, and the tests of these algorithms in the practical IVVS will be carried out as well.
The authors declare that there is no conflict of interests regarding the publication of this paper.
This project is supported by the key program of the National Natural Science Foundation of China (Grant no. 61039003), the National Natural Science Foundation of China (Grant no. 41274038), the Aeronautical Science Foundation of China (Grant no. 2013ZC51027), the Aerospace Innovation Foundation of China (CASC201102), and the Fundamental Research Funds for the Central Universities.