Detection and Recognition of Abnormal Running Behavior in Surveillance Video

. Abnormal running behavior frequently happen in robbery cases and other criminal cases. In order to identity these abnormal behaviors a method to detect and recognize abnormal running behavior, is presented based on spatiotemporal parameters. Meanwhile, to obtain more accurate spatiotemporal parameters and improve the real-time performance of the algorithm, a multitarget tracking algorithm, based on the intersection area among the minimum enclosing rectangle of the moving objects, is presented. The algorithm can judge and exclude e ﬀ ectively the intersection of multitarget and the interference, which makes the tracking algorithm more accurate and of better robustness. Experimental results show that the combination of these two algorithms can detect and recognize e ﬀ ectively the abnormal running behavior in surveillance videos.


Introduction
In most of the existing video surveillance systems, moving objects only were detected and tracked, which lacked to detect and recognize their behaviors in the surveillance scene.However, the purpose of monitoring the scene is to detect and analyze the unusual event or person's abnormal behavior in real life.In a long video sequence, such works handled manually are neither practical nor efficient, and the video surveillance system has already lost its original intention for preventing and actively intervening and almost become a tool of providing video evidence afterwards.The intelligent detection of abnormal behavior not only can detect abnormal behavior and inform the staffs to prevent illegal activities in time, but also can save a lot of storage space and avoid the staffs finding and collecting massively evidence after the illegal actions had happened.

Definition of Running Behaviors
Abnormal running behaviors frequently happened in robbery cases and other criminal cases.In order to distinguish between the abnormal running and the normal running, we first present the definition of the two behaviors as follow.
Definition 2.1 normal running .The object gradually accelerates from the state of walking or being stationary and then reaches even greater than the speed of normal running after a certain long time, or the object's speed moving into the video scene is greater than the speed of normal running.We define the above-mentioned action as the Normal Running Behavior.It can be represented by the following equation: Definition 2.2 abnormal running .The object suddenly accelerates from the state of walk or stationary and then reaches even greater than the speed of normal running after a certain short time, which is defined as Abnormal Running Behavior.It can be written as: where V 0 , V t are the initial velocity and the instantaneous velocity of the interested object, respectively, and V walk , V run are the speed of walk and the speed of normal running, separately.In additional, t is the time interval of an object from the speed less than V walk to the speed V t .When the speed V t > V run , T run is a time threshold used to determine whether the motion of object is the normal running, T ab is also a time threshold used to determine whether the motion of object is the abnormal running, and there is T ab > T run .Diagrams of the behavior are shown in Figures 1 and 2.
In Figures 1 and 2, t 1 is the start time and t 2 is the moment that the speed of moving object reaches the V run value.The difference between Figures 1 and 2 is that t 2 − t 1 > T run existed in Figure 1, while there is t 2 − t 1 < T ab in Figure 2. From this we have two conclusions: the key to distinguish the run behavior from the nonrun behavior is the speed of moving target; while the key to differentiate normal running from abnormal running is the moment at which moving targets achieve the speed of running.

The Basic Idea of the Method
Between the two adjacent images, the position and the contour of the same object are only changing a little in general, so the object's region in the two images is often intersected with each other 16, 17 .It is an important feature in continuous video sequences.The feature is exploited to track object in continuous video sequences in this paper, which is also used to detect moving objects in the algorithm of frame subtraction.In the following we will discuss the fundamental ideas of this paper in detail.In this paper, a moving object is marked with the minimum enclosing rectangle which is represented as Rect I I.x, I.y, I.width, I.Height , and I.x, I.y refers to coordinates of the upper left corner.I.width, I.Height refers to the width and the height of the rectangle.Thus the moving object's centroid can be calculated as the following formula:

3.2
In accordance with 3.1 and 3.2 , we can obtain the centroid CI CI.x, CI.y of the moving object.We assume that J which is represented as Rect J J.x, J.y, J.witdth, J.Height is a moving object in the t − 1 frame.We consider that the moving object J intersects with I, a moving object, in the t frame if they satisfy the following formulas: Aea Rect I&Rect J min Aea Rect I, Rect J ≥ r.

3.5
Actually the object I intersects with the object J if they satisfy the 3.3 and 3.4 , while the object's position does not always have much change in the two adjacent images, so that the intersection area between the rectangle of Rect I and Rect J has the property that is represented as 3.5 .In this paper, Rect I and Rect J are considered as the intersection unless they meet 3.3 , 3.4 , and 3.5 simultaneously.Shelter often occurs in surveillance scenes, because there are always multiple moving objects, so 3.5 can exclude a small part of the shelter which always impacts on the object tracking.
The shelter often happens in multitarget tracking in which objects may be sheltered by others or themselves, or by the stationary object in the background.The degree of the shelter is always different.The object's shelter can be divided into two stages.Firstly, the shelter occurring means that the target information is lost more and more during this period, which is shown as two or more rectangle boxes merged.Secondly, the shelter begins disappearing, and the target information is gradually restored, which are shown as the rectangle box separated into two or more rectangle boxes.
Therefore, when the block is occurring, this approach is to merge the blocked objects into a new object tracking and to record the histogram information of the sheltered objects in the previous frame.When the shelter is disappearing, to recognize the separated object, the separated rectangular box matches the recorded histogram of the tracked target.

The Exclusion of Interference
The interference in the moving object detection phase generally has two features.One is small size, and the other is the short survival time.In the former case, since we are only interested in people in video images and the people size in video images is generally not too small, so we can use a threshold to remove small object.According to data from several experimental results, we remove the object if its target area is less than 30 pixels.In the latter case, we have designed the list of temporary tracked objects, m TempObjectList, and the list of tracked objects, m TrackedObjectList, which are shown in Figure 3.
The node structures of the two lists are same.Each node records the corresponding history information of the moving target, such as the information of tracking process or the tracking information of behaviors analysis.These nodes are called tracked objects.There are differences in the two lists.m TempObjectList records the moving object in scenes whose existence time does not exceed a certain threshold, and m TrackedObjectList records the stable moving object whose existence time reaches a certain threshold.According to many experimental results, only when the existence time of the moving object reaches 5 frames, then the moving object is inserted into m TrackedObjectList and deleted from m TempObjectList.This method can exclude short-term interference in the surveillance video.Meanwhile, in order to avoid the interference in which the objects appear in part, we only deal with the object that has entered completely into the scene.

The Proposed Algorithm
Algorithm 3.1.Multitarget tracking algorithm.Input.The list of moving objects extracted from the current frame.
Step 1.We get a node from the m TempObjectList or m TrackedObjectList.If there are moving objects in the list of moving objects with the node that satisfies the formulae 3.3 , 3.4 , and 3.5 , then those moving objects are recorded as associated objects in the node.The node is recorded as associated node in those moving objects too.If there are some nodes which are not handled in the m TempObjectList or the m TrackedObjectList, then go to Step 1. Otherwise, go to Step 2.
Step 2. We get a node from the m TempObjectList and the m TrackedObjectList and count the number of associated objects of the node.If n 0, then goto Step 3, else if n 1, then go to Step 4, else go to Step 5.If every node in the m TempObjectList and the m TrackedObjectList has been processed, then go to Step 6.
Step 3. n 0 shows the tracked objects has disappeared in the current frame, if it belongs to m TempObjectList, then it is deleted from m TempObjectList.Otherwise if it is inserted into m TempObjectList and deleted from m TrackedObjectList, go to Step 2.
Step 4. n 1 means that there is only one object associated with the node.If there are more associated nodes recorded in the associated object, then the shelter algorithm will begin.Otherwise, the node is updated with the information of the associated object.Go to Step 2.
Step 5. n > 1 indicates that there are more associated objects, so the approach of the shelter disappearing is utilized.Go to Step 2.
Step 6.If there are not objects associated with any node in the m TempObjectList and the m TempObjectList, then a new node is generated for the object, and the node is inserted into the m TempObjectList.Go to Step 7.
Step 7. Update the m TempObjectList and the m TrackedObjectList.Delete the node whose existence time is more than 5 frames from the m TempObjectList and insert it into the m TrackedObjectList.

Detection of Running Behavior
According to the conclusion in the second part, the key to distinguish the running behavior or nonrunning behavior is the speed of the moving object.The instantaneous speeds of the targets can be simply obtained from v video s k − s k−1 , where s k , s k−1 is the target's centroid in the frame of k and k − 1, respectively.But it has not taken into account the actual action.People may appear into the surveillance video scene from different angles, and the distance between man and the camera may be changing.In addition, the focal lengths of cameras may often vary.Although people were standing in the same position, if its location was relatively far away from the camera, his picture will be small.On the contrary it will be relatively large.Moreover, the focal length has the same impact on the picture size of man in the video images.
Thus we can see that v video is related with the camera focal length and the distance between man and camera, for which we use the following formula to revise v video : where Rect I, Rect I is the enclosing rectangle of the moving object I in two adjacent frames, respectively.And α 27 is a const, v video , v video is the instantaneous speed of object I in the corresponding conditions of Rect I, Rect I , respectively.As RectI is similar to Rect I , v video / area Rect I v video / area Rect I .From this we can see that if the actual speeds of an object into surveillance scene are same, v video / area Rect I is same as the target area, even if in different shooting conditions.Therefore, the speed revised by 4.1 is reliable.
However, there are many reasons leading to the instantaneous speed being not reliable.First of all, the human motions are a complex system with a high degree of freedom and nonlinear characteristics.Secondly, the position and the contour of the object will have little change between the two adjacent images in general.Nevertheless, there may be some interference in the phase of extracting moving targets.It will lead to the centroid position if the target have not changed, even in the opposite direction.It makes the centroid and the instantaneous speed not accurate with formula 3.1 .To reduce the influence of the unreliable factors, we use the average speed in a short time to distinct running or not.The average speed of the targets can be obtained from v where k 2 and k 1 is frame number.

Recognition between Normal and Abnormal Running
Distinction of abnormal running is carried out under the condition of, V t > V run .Only when the speed of the moving target has reached the V run , we determine whether the object running is normal running or not.According to Definition 2.2, if the speed of the moving target achieves the running speed, the key to distinguish whether it is abnormal running is the time of t.If it is abnormal running, there are t < T ab and V 0 ≤ V walk .According to the Newton Leibniz Theorem, there are where V run , V walk , and T ab are consts, so V run − V walk /T ab is a const too, which is abbreviated as AMIN in this paper.Therefore, it can be distinguished between abnormal and normal running by judging whether a is greater than AMIN.

The Proposed Recognition Algorithm
Based on the above analysis, Definition 2.2 reduces to the following formula: In accordance with the above formula, determining whether the target behavior is abnormal running, we only need to judge whether V and a of the moving target meet a certain condition.In the light of many experimental results, the threshold of V and a is set to 4.0 and 0.4, separately.Thus we get the criterion for detecting and recognizing abnormal.Algorithm 4.1.recognition algorithm for abnormal running.Input.The list of moving objects extracted from current frame.
Step 1.The moving targets in the current frame are tracked with Algorithm 3.1, and then m TrackedObjectList is got.Go to Step 2.
Step 2. Get a tracked object from m TrackedObjectList and calculate the average speed v of the object in 5 frames, if v > V run then go to Step 3, else go to Step 4.
Step 3. Calculating the average acceleration a of the object in 5 frames, if a > AMIN then the object is identified as abnormal, otherwise go to Step 4.

Experimental Results
Our algorithms are implemented using the OpenCV library with C interface, which has been tested and evaluated in simple surveillance scenes and complicated surveillance scenes from open surveillance datasets PETS 2007 20 .Objects are modeled as rectangular bounding boxes with two colors.If the object is an abnormal object, then we identify it with black box and red "running" will appear on the box above, otherwise with red box, no "running".Besides, the red font upper left corner of the image shows the frame number.and the number of the objects, what is more, the green line in the image stands for the trajectory of the objects.Figure 6 a shows the relationship between the performance of the proposed algorithm and the value of threshold V run .And Figure 6 b shows the relationship between the performance of the algorithm and the threshold value of AMIN too.According to Figure 6, we conclude that the best threshold of V run is 4.0, and the optimal threshold of AMIN is 0.4.

In Simple Surveillance Scenes
The first test case is to detect abnormal running in a simple scene with single person.Figure 6 shows the result.In Figure 7 a , the existence time of the target is less than 5 frame, so that its behavior is not judged in frame 425.But in Figure 7 b , the average speed of the person reaches v 5.34 which is more than 4.0, and the average acceleration of the person is a 0.889 which is more than 0.4 too, and we can see that v and a meet the criterion for judging abnormal running, so the person is an abnormal target in the frame of 433.

In Complicated Surveillance Scenes
Figures 8 and 9 illustrate two complicated cases of abnormal running detection, respectively.In Figure 8, although there is more than one object, but shelter did not occurred between the objects, yet shelter happened in Figure 8.In Figure 8 a , the average speed of the person referred to as object 1 is v 5.42 which is more than 4.0, but his average acceleration a −0.05, which is less than 0.4, so the person 1 is not an abnormal target in the frame 65.Meanwhile, the average speed of another person referred to as object 3 is v 7.50 and its average acceleration is a 1.87, so object 3 is an abnormal target.Besides, the existence time of object 4 and object 5 is both less than 5, so that their behaviors are not judged in the frame 348.In Figure 8 b , the average speed of object 1 is v 7.9, but its average acceleration is a 0.17, so object 1 is not an abnormal target.While the average speed of object 3 is v 7.00 and its average acceleration is a 1.40, the average speed of object 4 is v 4.85 and its average acceleration is a 1.21, and the average speed of the object 5 is v 5.00 and its average acceleration is a 1.69, so as we have seen, object 3, object 4 and object 5 are all identified as abnormal targets in frame 356.
In Figure 9 a , the average speed of the object 2 is v 3.49, and is less than 4.0, so it is not an abnormal target.Object 3 behavior is not judged because of its existence time which is less than 5 frames.In Figure 9 b , serious shelter happened, which results in object 2 and object 3 merged into a new object 4, so we only need judge object 4, at this time the average speed of object 4 is v 4.29, and its average acceleration is a 0.71, so object 4 is identified as an abnormal target.Experimental results show that this algorithm can accurately detect the abnormal running behavior in different scenes.

Conclusion
Abnormal running frequently happened in robbery cases and other criminal cases.In order to identity such abnormal behavior in real time, this paper proposed a method on the basis of spatio-temporal parameters which can detect accurately the abnormal running.Meanwhile, to obtain precise spatiotemporal parameters and improve the real-time performance of the proposed algorithm, this paper proposed a multitarget tracking algorithm that is based on the intersection area among the minimum enclosing rectangle of the moving objects.The simple and real-time algorithm can effectively judge the intersection among objects and exclude the interference.In addition, two means of excluding interference are adopted in the multitarget tracking, which can exclude the objects which are too small or stay too short in scenes.Thus, the complexity of multitarget tracking is reduced significantly, and the accuracy is improved.

Figure 3 :
Figure 3: The list of temporary tracked object and tracked object.

Figure 4 :
Figure 4: The main flow of multitarget tracking method.

Figure 4
Figure 4 shows the main flow chart of the algorithm.Mean shift and Particle Filter are the most popular tracking algorithms in the intelligent video surveillance system.Comparing with Mean shift tracking algorithm and Particle Filter tracking algorithm, the proposed multitarget tracking algorithm has the following advantages.

Step 4 .
If there are tracked objects in the m TrackedObjectList have not been access then go to Step 2, else end.

Figure 5
Figure5shows the main flow chart of the algorithm.

Figure 5 :
Figure 5: The main flow of recognition algorithm for abnormal running.

Figure 6 :
Figure 6: The detection of performance at different threshold values.

a
The frame of 425 b The frame of 433

Figure 7 :b
Figure 7: Detection of abnormal running in simple surveillance scenes.

1
In the tracking result, regardless of the shelter, Particle Filter tracking results are more accurate than Mean shift, and Particle Filter is less affected by the background.Mean shift can track fast moving targets, but it is vulnerable to the background that is similar to the tracking target.And it can easily cause the vibration of the tracking window, which results in the tracking result being not stable.But our algorithm is less affected by the background as well as Particle Filter, and our algorithm can exclude two typical interferences in the surveillance video.Particle Filter and Mean Mathematical Problems in Engineering shift cannot track the object which is entirely sheltered, but our algorithm can do this.2About the time complexity, Particle Filter is more complex than Mean shift.The time complexity of Particle Filter is O K 2 N 2 , where K is the number of moving objects in the current frame, and N is the number of particles that distributed to moving objects 18 .The time complexity of Mean shift is O Nk h C s , where N is the average number of iterations per frame, k h is the number of pixel of target in the window of nuclear function, and is C s the cost of arithmetic operations, such as an addition operation 19 .However, the time complexity of the proposed multitarget tracking algorithm is O K 2 , where K is the number of moving objects in the current frame.