Activity recognition is needed in different requisition, for example, reconnaissance system, patient monitoring, and human-computer interfaces. Feature selection plays an important role in activity recognition, data mining, and machine learning. In selecting subset of features, an efficient evolutionary algorithm Differential Evolution (DE), a very efficient optimizer, is used for finding informative features from eye movements using electrooculography (EOG). Many researchers use EOG signals in human-computer interactions with various computational intelligence methods to analyze eye movements. The proposed system involves analysis of EOG signals using clearness based features, minimum redundancy maximum relevance features, and Differential Evolution based features. This work concentrates more on the feature selection algorithm based on DE in order to improve the classification for faultless activity recognition.
Many researches in activity recognition and computer vision adopt gesture recognition as forefront [
The goal of activity recognition is to provide information that allows a system to best assist the user with his or her task. Activity recognition has become an important area for a broad range of applications such as patient monitoring, vigilance system, and human-computer interaction. Bulling is the first to describe and apply eye base activity recognition to the problem of recognition of everyday activities [
Bulling also described and evaluated algorithms for detecting 3 eye movement characteristics of EOG signals, saccades, fixations, blinks [
One of the possibilities to detect eye movements is EOG, which is a technique for measuring the resting potential of the retina [ Wijesoma et al. used EOG for guiding and controlling a wheelchair for disabled people [ Usakli and Gurkan used EOG for using virtual keyboard [ Deng et al. used EOG for operating a TV remote control and for a game [ Talaba used EOG for visual navigation metaphor [ Gandhi et al. used EOG for controlling multitask gadget [ Bulling et al. used EOG for activity recognition [
Feature selection is important and necessary when a real world application has to deal with training data of high dimensionality. There are many features selection approaches available in the literature. Some of the hybrid approaches are listed as follows. García-Nieto et al. presented a Differential Evolution based approach for efficient automated gene subset selection using DLBCL Lymphoma and Colon tumor gene expression datasets. The selected subsets are evaluated by means of the SVM classifier [ Li employed a DE-SVM model that hybridized DE & SVM to improve the classification accuracy of road icing forecasting using feature selection [ Xu and Suzuki proposed a feature selection method based on sequential forward floating selection to improve performance of a classifier in the computerized detection of polyps in CT colonography (CTC). In this work feature selection is coupled with SVM classifier and maximized the area under the receiver operating characteristic curve [ Kuo et al. proposed kernel based feature selection method to improve the classification performance of SVM using hyperspectral image datasets [ Güven and Kara employed artificial neural network analysis of EOG signals for the purpose of distinguishing between subnormal and normal eye [
From the literature review it can be observed that there are a number of EOG applications being developed. Still necessary feature selection algorithms need to be developed, evaluated, and used to produce substantial improvements in communications with disabled people by using eye movements and to make inference about person’s cognitive state.
This paper presents a hybrid feature selection technique based on DE for activity recognition using eye movements by EOG signals which can identify a subset of most informative, eye movement characteristics amongst all eye movement characteristics. This method is used as an optimizer before the classifier to EOG signal features for recognizing activities like read, browse, write, video, and copy. The benefits of the feature selection approach include improving the efficiency of activity recognition since only a subset of eye movement is used and assisting SVM to attain satisfied accuracy.
Here we explain the preprocessing, feature selection (with CBFS, mRMR, and DEFS), SVM classifier, and the model evaluation strategy used in this work.
The EOG data used in this study are collected from the Andreas Bulling’s “recognition-of-office-activities” dataset (
These experiments were carried out in a well-lit workplace during normal working hours. Participants were seated ahead of 2 seventeen inch flat screens with a resolution of
EOG signals were picked up using an array of five 24 mm Ag/AgCl wet electrodes from Tyco Healthcare placed around the right eye. The horizontal signal and vertical signal were collected using two electrodes for each and the fifth electrode was placed on the forehead for the signal reference. EOG data is captured employing a commercial EOG device known as TMSI (Twente Medical Systems International) Mobi8 that integrates instrument amplifiers with 24 bit ADCs. Mobi8 tends to have better signal quality. Mobi8 was worn on a belt around each participant’s waist and recorded four channels EOG at a sampling rate of 128 Hz. The behaviours of participants in specific phases (read, browse, write, video, copy, and rest) were observed by an annotated activity changes with wireless remote control and their nature in daily life during regular working hours. Context Recognition Network Toolbox (CRNT) was used for handling data recording and synchronization.
For preprocessing, this work adapts median filter and baseline drift removal 1D wavelet decomposition at level nine using Daubechies wavelets on the signal component [
Figure
Before and after preprocessing (EOG) signals.
Table
MSE and PSNR values.
Activity | Null | Read | Browse | Write | Video | Copy |
---|---|---|---|---|---|---|
Samples number | 320315 | 290963 | 302385 | 317092 | 304170 | 303441 |
MSE | ||||||
Before filtering | 4.1736 |
3.1071 |
7.550 |
2.1893 |
1.0267 |
3.1546 |
After filtering |
|
|
|
|
|
|
PSNR | ||||||
Before filtering | 30.4026 | 29.5266 | 30.3008 | 31.0553 | 30.5343 | 31.0118 |
After filtering |
|
|
|
|
|
|
Various activities using eye movements by EOG can be portrayed as a regular pattern by a specific sequence of saccades and short fixations of similar duration. The amplitude change in signals varies for various activities which can be used in identifying the regular office activities. The reading activity using EOG is patterned by small saccades and fixations. No large change in amplitude is included in reading. This is due to small eye movement between the words and fast eye movement between ends of previous line and beginning of next line. Writing was similar to reading, yet it required greater fixation duration and greater variance. It was best described using average fixation duration. Copying activity includes normal back and forth eye movements which involves saccades between screens. This was reflected in the selection of small and large horizontal saccade features, as well as variance in horizontal EOG fixations. In contrast, watching a video and browsing are highly unstructured. These activities depend on the video or website being viewed. These results propose that, for tasks that involve a known set of specific activity classes, recognition can be streamlined by only choosing eye movement features known to best describe these classes.
The steps in basic eye movement type detection for EOG based action recognition are
Eye movement characteristics like saccades, fixations, and blink patterns are separated from EOG signals using continuous wavelet transform (CWT) 1D wavelet coefficients using a Haar mother wavelet at scale 20 [
By applying a threshold
We used the same wavelet coefficients to detect blinks in EOGv. Features related to the eye movements using EOG signal were calculated separately for two EOG (horizontal and vertical) signals for each participant. Statistical features such as mean, variance, and maximum value, minimum value based on saccades, fixations, and blinks are extracted from this work. Different characteristics of EOG signals result in different changes in the coefficients. Totally 210 statistical features are extracted from this work.
Eye movement characteristics such as saccades and blink patterns of EOG (horizontal and vertical) signals are shown in Figure
(a) Blink detection from EOGv. (b) Saccades from EOGh.
The minimum redundancy maximum relevance (mRMR) [
CBFS [
Clearness based feature selection (CBFS) algorithm is a type of filter method. Clearness means the detachment between classes in a feature. If (clearness of feature
The centrist for read and write is calculated by average operation. This is the median point of a class. Med(
For each
Calculate
Calculate
The proposed method first identifies essential features by applying a threshold (
The reduced feature space is given to the DEFS algorithm for detecting the significant feature subset through machine learning algorithm such as maximum a posterior approach. Here for the projection of features into feature subspace we go for kernelized Bayesian structure. This method is used as an optimizer before the classifier to EOG signal features for recognizing activities like read, browse, write, video, and copy.
Figure
Differential Evolution operations.
In this work parameter
For each target vector mutant vector is created by using (
The parent vector is mixed with the mutated vector to produce a trial vector as in (
DE employs uniform crossover. Newly generated vector results in a lower objective function value (fitness). The randomly chosen initial population matrix of size (NP × DNF) containing NP randomly chosen vectors
Table
Performance summary.
Performance | Null | Read | Browse | Write | Video | Copy | All |
---|---|---|---|---|---|---|---|
Samples number | 320315 | 290963 | 302385 | 317092 | 304170 | 303441 | 1838366 |
Number of features | 232 | 211 | 218 | 225 | 218 | 218 | 210 |
Accuracy (all features) | 82% | 67% | 62% | 73% | 83% | 68% | 72.5% |
Accuracy (with DEFS-15 features) | 87% | 77% | 79% | 84% | 88% | 85% |
|
Activity classification involves a set of
SVM handles nonlinear data by using a kernel function [
The kernel function used in this work is linear kernel, meaning dot product which is shown in (
There is a class of functions
The classifier is defined by (
We adopted the following criteria to evaluate the performance of the classifier. We used Mat lab to calculate the accuracy of the classifier:
The sensitivity, specificity, true positive rate (TPR), false negative rate (FPR), and accuracy (ACC) were calculated by using (
The detailed performance of the proposed approach with DEFS features is listed in Table
Detailed performance of the proposed approach with CBFS and mRMR features.
Activity | Features | TP (%) | FP (%) | Precision (%) | Recall (%) |
|
---|---|---|---|---|---|---|
Null | DEFS |
|
|
|
|
|
mRMR | 65.50 | 15.41 | 80.95 | 93.32 | 86.70 | |
CBFS | 67.60 | 14.30 | 82.54 | 89.89 | 86.06 | |
|
||||||
Read | DEFS |
|
|
|
|
|
mRMR | 58.30 | 18.29 | 76.11 | 91.79 | 83.22 | |
CBFS | 57.29 | 16.58 | 77.56 | 82.27 | 79.84 | |
|
||||||
Browse | DEFS |
|
|
|
|
|
mRMR | 68.98 | 14.38 | 82.75 | 95.17 | 88.53 | |
CBFS | 54.00 | 16.50 | 76.60 | 81.82 | 79.12 | |
|
||||||
Write | DEFS |
|
|
|
|
|
mRMR | 79.10 | 08.85 | 88.94 | 94.56 | 92.19 | |
CBFS | 69.24 | 12.10 | 85.12 | 90.39 | 87.68 | |
|
||||||
Video | DEFS |
|
|
|
|
|
mRMR | 67.41 | 13.09 | 83.74 | 90.23 | 86.86 | |
CBFS | 62.37 | 15.44 | 80.16 | 86.16 | 83.05 | |
|
||||||
Copy | DEFS |
|
|
|
|
|
mRMR | 70.34 | 10.28 | 85.25 | 87.66 | 87.45 | |
CBFS | 56.44 | 11.17 | 83.48 | 81.88 | 82.67 |
Figure
Precision for each activity by proposed DEFS based features with mRMR, CBFS features.
Feature extracted addresses the problem of finding a more informative set of features. The resultant feature subset shows that the most informative features in EOG signals are saccades and blinks. The statistical measures for EOG signal analysis proved to be very useful in searching the feature space by using hybrid feature selection based on differential evolution. The SVM with linear kernel is used for classification in EOG datasets before and after feature selection. The results of 10-fold cross-validation are listed in Table
A hybrid feature selection method was employed in EOG signals based on the Differential Evolution and the proposed method is compared with CBFS and mRMR feature selection. The differential evolutionary algorithm is utilized to give powerful results in searching for subsets of features that best interact together through supervised learning approach. EOG dataset with high dimensionality and number of target classes (
The authors declare that there is no conflict of interests regarding the publication of this paper.