Physical activity (PA) recognition has recently become important in activity monitoring for the public healthcare. Although body-worn sensors are well suited to collect data on activity patterns for long periods of time, users may forget to wear special microsensors. On the contrary, more and more people take smartphone with them almost anytime. At present, most popular smartphones have three built-in kinematic sensors (triaccelerometer, gyroscope, and magnetic sensor) which could be utilized to recognize PA. This study utilized three built-in kinematic sensors in a smartphone to recognize PA and found out which features derived from the three sensors were significant to different PA. We used a combined algorithm of Fisher’s discriminant ratio criterion and
Physical activity (PA) recognition has recently become important in activity monitoring for the public healthcare. The progressive decline in the PA level due to the adoption of sedentary lifestyles has been associated with the increasing incidence of obesity, diabetes, and cardiovascular diseases [
Body-worn sensors [
The built-in kinematic sensors of smartphones are used to recognize PA, which not only reduces the cost of hardware but also exploits existing communication module and ubiquitous monitor. The previous studies have demonstrated the effectiveness of using a smartphone to detect a fall [
Most of the previous studies [
There are several studies using the combination of accelerometer, gyroscope, and magnetic sensor. Zhang et al. [
The aims of this study were to utilize the built-in kinematic sensors (triaccelerometer, gyroscope, and magnetic sensor) of a smartphone to recognize PA and find out which features derived from the built-in kinematic sensors were significant to recognize PA. This work will make the assessment of activities of daily living and fall detection more ubiquitous and portable.
The rest of the paper is organized as follows: Section
The block diagram of the main data processing scheme was described in Figure
The block diagram of the main data processing scheme.
A smartphone (Samsung I9023 Nexus S,
The coordinate system and the placement of the smartphone at the chest of a subject.
The smartphone has built-in triaxial accelerometer (STM KR3DM) with 19.6 m/s2 maximum range and 0.019 m/s2 resolution, triaxial gyroscope sensor (STM K3G) with 34.9 rad/s maximum range and 0.0012 rad/s resolution, and triaxial magnetic field sensor (Asahi Kasei AK8973) with 2000
The standard sensor coordinate system of the smartphone is defined relative to the screen. As shown in Figure
The coordinate system of the accelerometer is the same as the standard sensor coordinate system. The accelerometer measures acceleration along
The coordinate system of the gyroscope is the same as the one used for the accelerometer. Rotation is positive in the counterclockwise direction. Gyroscope measures the rate of rotation around
The orientation sensor is software-based and derives its data from the accelerometer and the geomagnetic field sensor. The orientation sensor monitors the position of the smartphone relative to the earth’s frame of reference and measures azimuth (
In Android operating system, four different sampling frequencies (fastest, game, normal, and UI) of sensors can be selected. The values of the four frequencies are not constant and depend on the computational workload of the smartphone. The fastest sampling frequency is selected and can reach 50 Hz normally. The acceleration, gyroscope, and orientation signals were sampled at 25 Hz and stored on a Secure Digital (SD) card in the smartphone. Table
Description of activities performed by a subject.
Activity tasks | Key | Description |
---|---|---|
Sitting | Si | The subject sits up on an upholstered chair with armrest (seat height: 48 cm). |
Lying | L | The subject lies down on a bed on his back (bed height: 50 cm). |
Standing | St | The subject remains standing for 5 s. |
Lie-to-sit | L-Si | Initially lying on the bed, after 5 s, the subject sits up and remains sitting on the bed for 5 s. |
Sit-to-lie | Si-L | Initially sitting up in the chair, after 5 s, the subject lies down and remains lying on the bed for 5 s. |
Sit-to-stand | Si-St | Initially sitting up in the chair, after 5 s, the subject stands up and remains standing for 5 s. |
Stand-to-sit | St-Si | Initially standing, after 5 s, the subject sits down and remains seated for 5 s. |
Walking | W | Initially standing, after 5 s, the subject walks for 10 s at a normal pace. |
Walking upstairs | WU | Initially standing, after 5 s, the subject walks upstairs at a normal pace for 12 steps. |
Walking downstairs | WD | Initially standing, after 5 s, the subject walks downstairs at a normal pace for 12 steps. |
Running | R | Initially standing, after 5 s, the subject runs for 10 s at a moderate speed. |
Jumping | J | Initially standing, after 5 s, the subject jumps up along the gravity direction. |
Forward fall | FF | Initially standing, after 5 s, the subject simulates fall frontward onto a mattress (thickness = 10 cm). |
Right-side fall | FR | Initially standing, after 5 s, the subject simulates fall towards the right side onto the mattress. |
Backward fall | FB | Initially standing, after 5 s, the subject simulates fall backward onto the mattress. |
Left-side fall | FL | Initially standing, after 5 s, the subject simulates fall towards the left side onto the mattress. |
The sliding window approach was employed to divide the sensor signal into smaller time windows (Figure ( ( TA is defined as the angle between the positive
The sliding window of the data from the accelerometer, the gyroscope, and the orientation sensor during running. The window slides one time such as the blue-dotted rectangle moves to the blue solid one.
As shown in Table
Definitions of statistical features.
Statistical features | Key | Description |
---|---|---|
Average | Avg | The average value over the window. |
Median | Med | The median value over the window. |
Standard deviation | Std | The standard deviation value over the window. |
Skewness | SK |
|
the degree of asymmetry of the distribution over the window. | ||
Kurtosis | K |
|
the degree of peakedness of the distribution over the window. | ||
Interquartile range | IR |
|
Percentage of decline | PD | The percentage of point decline in the entire window. |
Definitions of features.
Features | Description |
---|---|
|
The average value over the window of |
|
The median value over the window of |
|
The standard deviation value over the window of |
|
The skewness value over the window of |
|
The kurtosis value over the window of |
|
The interquartile range over the window of |
|
The percentage of decline in the entire window of |
A combined algorithm of Fisher’s discriminant ratio (FDR) criterion and
Then the 10 highest-ranked features were selected. And the exhaustive search method with the
Furthermore, large values of
A hierarchical classifiers system was shown in Figure
The classifiers flowchart. All abbreviations in this figure were shown in Table
Moreover, Classifier 5 was employed to classify jumping and falls. Falls are said to have occurred if at least two consecutive peaks in the signal magnitude vector above a defined threshold are recorded [
Furthermore, there were no obvious features to classify some certain activities, such as walking, walking downstairs and walking upstairs. In a classifier, all possible combinations splitting the activities were exhaustively formed, and for each combination its
The scatter graphs of part classifiers were shown in Figure
The scatter graphs of part classifiers. (a) The scatter graph of C3. (b) The scatter graph of C4. (c) The scatter graph of C8. Pluses and circles indicated points into the two classes of classifiers, respectively.
The scatter graph of C3
The scatter graph of C4
The scatter graph of C8
As shown in Table
The selected features of all classifiers.
Classifier | Top 5 optimal features | Selected features |
---|---|---|
C1 |
|
|
| ||
C2 | TAAvg, TAMed, |
TAAvg, |
| ||
C3 |
|
|
| ||
C4 |
|
|
| ||
C5 |
|
|
| ||
C6 |
|
|
| ||
C7 |
|
|
| ||
C8 |
|
|
| ||
C9 |
|
|
| ||
C10 |
|
|
| ||
C11 |
|
|
| ||
C12 |
|
|
| ||
C13 |
|
|
| ||
C14 |
|
|
In these three sensors, accelerometer was dominant to PA recognition. In addition, gyroscope served well in recognizing the change of body posture such as transitions, and the orientation sensor was effective to detect falls. As illustrated in Figure
Data obtained during four kinds of falls (forward fall, left-side fall, right-side fall, and backward fall). The dotted red rectangles were amplified to observe details. The features from the orientation sensor were effective to detect falls.
Figure
We used hold-out method [
The confusion matrix in Table
Representative confusion matrix.
Classified as | a | b | c | d | e | f | g | h | i | j | k | l | m | n | o |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
a |
|
4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
b | 1 |
|
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
c | 0 | 0 |
|
3 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
d | 0 | 0 | 2 |
|
0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 4 | 1 |
e | 0 | 0 | 0 | 0 |
|
13 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
f | 0 | 0 | 0 | 0 | 10 |
|
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
g | 0 | 0 | 0 | 0 | 3 | 2 |
|
8 | 10 | 3 | 0 | 0 | 0 | 0 | 0 |
h | 0 | 0 | 0 | 0 | 0 | 0 | 7 |
|
6 | 1 | 0 | 0 | 0 | 0 | 0 |
i | 0 | 0 | 0 | 0 | 0 | 0 | 5 | 7 |
|
5 | 0 | 0 | 0 | 0 | 0 |
j | 0 | 0 | 3 | 0 | 0 | 0 | 3 | 2 | 1 |
|
7 | 0 | 0 | 0 | 0 |
k | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 2 | 8 |
|
2 | 0 | 0 | 0 |
l | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 |
|
1 | 2 | 1 |
m | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 |
|
0 | 3 |
n | 0 | 0 | 0 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 3 | 0 | 0 |
|
0 |
o | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 3 | 0 |
|
Key: lying = a; sitting/standing = b; lie-to-sit = c; sit-to-lie = d; sit-to-stand = e; stand-to-sit = f; walking = g; walking upstairs = h; walking downstairs = I; running = j; jumping = k; forward fall = l; right-side fall = m; backward fall = n; left-side fall = o.
The application of PA recognition implemented in the smartphone. The green line indicated the SVM value of the accelerometer real time, and the blue line presented the PA.
A PA recognition system based on a smartphone was developed. The optimal features derived from the built-in kinematic sensors of the smartphone were selected from 140 features for each classifier.
In the previous studies [
Data obtained during walking, walk downstairs, and walk upstairs.
The scatter graphs of three activities, employing a combination of GyStd, A3aStd, and G3aStd. Pluses, circles, and dots indicated points from walking, walking upstairs, and walking downstairs, respectively.
The PA recognition system based on the smartphone provided several advantages, such as existing communication module and ubiquitous monitor. The previous studies have used Short Messaging Service (SMS)[
The main limitation of our approach is that the smartphone must be worn on the chest. People put phones in the pocket of shirts or pants according to their respective habits, which would cause loose attachment between the body and smartphone resulting in the invalidness of the algorithm eventually. Khan et al. [
For this study, the optimal features derived from the built-in kinematic sensors of the smartphone were selected to recognize PA. The results of selected features suggested the accelerometer was significant to PA recognition, and gyroscope and orientation sensor were effective to recognize the change of body posture or detect falls, respectively. Despite the limitations of this system which requires firm attachment of the smartphone to a subject, the experimental results demonstrated the feasibility of utilizing the built-in kinematic sensors of the smartphone to recognize PA.
This work was supported in part by the National S&T Major Project of China (no. 2011ZX03005-001), Guangdong Province and Chinese Academy of Sciences Comprehensive Strategic Cooperation Project (no. 2011A090100025), and National Natural Science Foundation of Youth Science Foundation (no. 31000447).