For an establishment of a skill evaluation method for human support systems, development of an estimating equation of the machine operational skill is presented. Factors of the eye movement such as frequency, velocity, and moving distance of saccade were computed using the developed eye gaze measurement system, and the eye movement features were determined from these factors. The estimating equation was derived through an outlier test (to eliminate nonstandard data) and a principal component analysis (to find dominant components). Using a cooperative carrying task (cc-task) simulator, the eye movement and operational data of the machine operators were recorded, and effectiveness of the derived estimating equation was investigated. As a result, it was confirmed that the estimating equation was effective strongly against actual simple skill levels (r=0.56–0.84). In addition, effects of internal condition such as fatigue and stress on the estimating equation were analyzed. Using heart rate (HR) and coefficient of variation of R-R interval (Cvrri). Correlation analysis between these biosignal indexes and the estimating equation of operational skill found that the equation reflected effects of stress and fatigue, although the equation could estimate the skill level adequately.
1. Introduction
With the development of science and technology in several decades, we have had more opportunities to operate various types of machines in our daily life. In order to elicit high performance of such machines, however, the users have to strive for mastery of the operation, and much time and effort are frequently needed. With that in mind, a concept of Human Adaptive Mechatronics (HAM) [1–3], which is an intelligent mechatronics to help the mastery of the user’s operation, was presented. Under the project, various kinds of system design theories and technologies for HAM, that changes its dynamic characteristics and the supporting strategies adaptively to the status of individual user in order to enhance the performance of whole human-machine system, have been studied [4–6]. The following two main functions are required to realize HAM: quantification of skill level of users and adaptive human-assisting mechanism according to the skill level. Researches about the quantification of skills, analyses of vehicle control characteristics of drivers or pilots [7, 8], studies concerning cognitive skill for human-computer interface interaction [9], and researches on perceptual skill on video game [10] are known. On the other hand, in order to design an adaptive human-assisting mechanism, a real-time estimation of participants’ skill and its feedback to the human-assisting mechanism are required. Utilization of biological information is one of effective approach for such real-time estimation. Especially, measurement of ocular motion is adequate for analysis of vehicle drivers’ behavior and their skill [11], because the motion reflects intention and thinking of a human in real time [12]. Effectiveness of a usage of the ocular motion is widely well known, and the present authors also have been studying the related themes such as an identification of a human controller on the vehicle driving [13] and a brain monitoring analysis on oculomotor cortex for the voluntary motion skill [14].
Fatigue or stress, however, also influences the ocular motion. Utilizing this property, many studies evaluating such internal status from the measurement of eye motion are reported [15, 16]. Therefore in this paper, the following three steps which were required to establish the skill evaluation method for HAM were treated.
Step 1.
Derivation of an equation to estimate a skill level of a machine operation using the measurement of eye motion.
Step 2.
Analysis of the fatigue and stress during the operation.
Step 3.
Investigation of an influence between the estimating equation and the fatigue and stress.
Concerning the evaluation of fatigue and stress, a method computing Cvrri on the VDT (video display terminal) operation is known, where Cvrri is a coefficient of variation of R-R interval which is a time sequence data of heart rate (HR) [17]. Other methods utilizing variation of HR and a chaotic property of the pulse wave are also used [18]. In reference to the previous studies, the fatigue and stress were also treated in this paper. In regard to the measurement of the eye motion, a gaze detection system which was developed in our previous study [19] was used here. As a machine operation task, a cooperative-carrying-task (cc-task) simulator was utilized [20]. These systems were integrated, and eye motion, machine operation, and biosignal of the participants were measured. Then analyses for Step 1 to Step 3 were performed.
The remainder of this paper is organized as follows. Section 2 explains the cc-task simulator and the experimental setup. In Section 3, findings about the eye movement and skill are mentioned, and development of an estimating equation of the skill level is explained (for Step 1). Section 4 explains properties of biosignal used in latter analyses. Next using the measured biosignal, adequacies of methods to evaluate fatigue and stress are verified (for Step 2). Section 5 shows the result of the correlation analysis between the obtained skill estimating equation and fatigue and stress (for Step 3). Last Section 6 is the conclusion.
2. Observation Experiment of Training Process on Cooperative-Carrying Task2.1. Cooperative-Carrying-Task Simulator
This simulator was designed as a VDT operation of the virtual vehicle, so that the participants would not have to move their body since the large body movement is not preferable for long hours measurement of the eye motion and biosignal. In the virtual cooperation task, three participants work in the same virtual space. Each participant sits in front of each monitor and uses a joy-stick for the manipulation, as shown in Figure 1. The participant manipulates a virtual mobile robot, cooperates with the other participants, and conveys three boxes to target places. Figure 2 shows an overhead view of the virtual space. The simulator was built by a real-time computer graphics using OpenGL and ODE (open dynamics engine) library. Motions of all robots and boxes are computed according to their physical dynamics, and collision among their objects can be also simulated.
Experimental scene of cooperative-carrying task.
Overview of virtual work space for cooperative-carrying task.
Each participant donned a head set of the gaze detection system which was developed by authors in previous studies [19, 21], as shown in Figure 3. Multivideo sources, which consist of the eye image captured by the eye camera (the top-left section in Figure 3(b)), the front camera view (the top-right), and the video source displayed to the monitor for the participant (the bottom-left), were compressed into one video signal using an image partition device. Heart rate of one participant of three was recorded using Polymate II (AP216, TEAC Corporation, Japan).
(a) Head set of gaze detection and (b) sample of recorded movie.
The procedure of this experiment was approved by the University’s Ethics Committee. Participants cooperated after he/she gave informed consent. Participants are 33 Japanese (31 males and 2 females, 20 years to 23 years), and eleven teams consisting of three participants executed the cc-task. Trials were repeated ten times for each team. Total 330 data files including information of eye movement and operation could be recorded. Trial time differed in each team; minimum and maximum of accumulative time of all ten trials were 78 min and 250 min, respectively. Seven persons among those 33 participants were, however, eligible for later correlation analysis between eye movement and stress and fatigue. That was because biosignal data from two members of three ones in one team were not measured from the beginning, as explained perviously. To avoid confusion to readers, data of those seven participants are treated in the following sections on ahead, and the reason of selection of those seven participants will be explained in Section 3.2. In later discussion, those seven participants are called an operators A–G.
2.2. Index of cc-Task Performance
Objective index is necessary to evaluate an ability to accomplish the cc-task in order to develop an estimating equation of the operation skill. Since all operators were asked to finish the task as soon as possible, it was expected that the trial time decreased as the trial increased. It was also expected that frequency of pause in a vehicle operation decreased since unnecessary pausing caused waste of time. Hence, the time required to finish one trial was defined as T-index, and the pause time per one minutes was defined as P-index. Further, since the experimenter asked operators to avoid collision against walls and other vehicles, the number of such collision was chosen as a third index and was defined as C-index. Transition of these indexes are shown in Figures 4, 5, and 6, respectively. Table 1 summarizes correlation coefficients of the linear regression lines for each transition shown in these figures.
Correlation coefficients of T-index, P-index, and C-index, against the number of trials.
Operator
A
B
C
D
E
F
G
Time: T-index
−0.77
−0.76
−0.83
−0.92
−0.73
−0.82
−0.90
Freq. of pause: P-index
−0.30
−0.60
−0.76
−0.59
−0.69
−0.73
−0.92
Freq. of collision: C-index
0.54
0.35
0.69
0.22
0.04
0.28
0.29
Transition of the trial time for all operators (T-index).
Transition of the frequency of pause for all operators (P-index).
Transition of the number of collision for all operators (C-index).
Figure 4 shows a monotonic decrease concerning T-index for any case. As shown in the second row in Table 1, correlation coefficients between T-index and the number of trial are all negative, and their absolute values are large as 0.73–0.92; hence, all operators achieved perfection in terms of the task time. Figure 5 for P-index indicates almost monotonic decrease, and their correlation coefficients are negative large (r=-0.92,…,-0.30), although there were individual differences. This means that the hesitation and waiting in the operation decreased as the trial increases, and it can be thought that the efficiency of the task performance increased. Figure 6 showing index-C may indicate the number of collision increases, but not monotonically. It is seemed that the number of trial can be used to express the degree of skill; hence, the number of the trial is treated as simple index and is used as a simple skill level S as follows.
With these facts, it can be guessed that almost all operators were getting skilled up in terms of the trial time and frequency of the pause time, but they became tired, and the operational error might increase. Based on this inference, an influence of the fatigue-stress condition to the estimating equation of the machine operation skill is analyzed in later sections.
3. Development of the Estimating Equation of Skill Using Eye Motion Measurement3.1. Eye Movement and Operational Skill
The well-known studies investigating a relation between an eye movement and a machine operation are for driving a car. Such studies elucidated the following facts.
Fact 1.
As the speed of the car increases, the driver tends to concentrate to the direction of movement of the car, and he/she does not notice events that are projected to the peripheral vision on retina [22].
Fact 2.
Superfluous saccadic eye movement decreases as the driver becomes an expert [23].
These facts come from the following biological properties of the human eye: a human eye has high resolution only around central visual field, and the saccadic movement is required to obtain various visual information from whole of the visual field. With this, many studies analyzed the eye saccade and fixation to unravel human cognitive process [24, 25]. In such studies, the response time of the eye movement to detect moving things (this time is described as η for later discussion), the gaze duration (which is named f), and the frequency/velocity/moving distance of saccade (μ, v, and d, resp.) have been utilized as indexes of eye movement characteristics. Considering Fact 1 and related findings, we summarized them into five hypotheses using these indexes in case of the beginner and expert, as shown in Table 2.
Hypotheses based on characteristic indexes of eye motion.
Hypothesis no.
Vehicle velocity
High
Low
Driving skill
Expert
Beginner
Expert
Beginner
H1
Response time, η [ms]
η1
<
η2
η3
<
η4
H2
Fixation time, f [ms]
f1
≒
f2
f3
<
f4
H3
Frequency of saccade, μ [/s]
μ1
≒
μ2
μ3
>
μ4
H4
Velocity of saccade, v [deg/s]
v1
<
v2
v3
<
v4
H5
Distance of saccade, d [deg]
d1
<
d2
d3
<
d4
In high-speed driving, Fact 2 implies that the expert can notice immediately a moving object since he/she can find it at the peripheral field. That is, it can be expected that the response time of the expert is smaller than that of the beginner. When the response times for expert and for beginner are denoted as η1 and η2, respectively, the hypothesis H1 is described as η1<η2, as shown in Table 2. In the following, numerical subscripts attached to variables are used to distinguish each other. Again based on Fact 1, in high-speed driving, both expert and beginner tend to concentrate the line-of-sight into narrow area to the direction of movement of the car; hence, the frequency of the saccade seems low in both cases (hypothesis H3: μ1≅μ2). Similarly, it is guessed that there is not actually difference between the expert and the beginner in terms of the fixation duration, and this guess was named the hypothesis H2(f1≅f2). On the other hand, in low-speed driving mainly in an urban area, the response of the beginner is slower than that of expert, since the beginner does not have sufficient ability to watch something important for safety driving (hypothesis H1: η3<η4). In contrast, the frequency of saccade of the expert seems larger than that of beginner because the expert tries to pay attention to the driving environment as far as possible (hypothesis H3: μ3>μ4). And then the fixation time becomes short (hypothesis H2: f3<f4).
Based on hypotheses of H1, H2, and H3, the present authors analyzed correlation between the task performance in the cc-task and the aforementioned indexes of eye movements. The high-speed and low-speed ranges were specified by 50 percent of maximum velocity of the virtual vehicle dynamics since absolute speed value concerning Fact 1 was not known generally. Although this threshold was changed and analysis was executed, we could not find significant difference between beginners and experts in terms of η, f, and μ. In short, there were large individual difference, and it was concluded that the indexes of η, f, and μ were inadequate to evaluate the operational skill level [26].
Considering Fact 2 again, this fact can be interpreted as a property of the expert who can use both central visual field and peripheral vision depending on the situation. In other words, the expert utilizes characteristics of peripheral vision that can detect motion sensitively in spite of its low resolution, and he/she can perceive something moving without changing of the gaze direction. Such perception does not require a motion of the eyeball, and it differs from an ocular exploration accompanied by the saccadic eye movement; hence, it appears that this processing does not depend on the difference of the vehicle speed. Therefore, the followings are inferred in case of the expert: the moving distance of saccade is small regardless of the vehicle velocity (hypothesis H5:d1<d2,d3<d4), and velocity of saccade is also small due to the short moving distance (hypothesis H4:v1<v2,v3<v4). Hypotheses H4 and H5 are verified through the following sections.
3.2. Extraction of the Eye Movement Feature
Using an image processing algorithm presented in [26], the coordinate value corresponding to the line-of-sight was computed after detecting the position of the pupil image which was recorded by the eye camera. From the time-sequence data of the line-of-sight, the eye movement feature, which will be explained later, was derived. Although the number of operators for the experiment was 33, the number of the valid data set was of 25 operators. That is because perfect data including eye movement for all ten trials was not obtained since the recorded eye images were contaminated by shadow of eyelash or insufficient by positional error of the eye camera setting. Next, computing velocities of eye movement during all ten trials for each of the 25 operators, outlier case was checked by Smirnov-Grubbs’ outlier test using a level of significance of 0.1. Further, data including strong individual difference was eliminated by statistical testing of normality using a Lilliefors test (P<0.1) for each operator’s data. As a result, outlier or disnormality was found in six-operator case. Among the remained 19(=25-6) operators, seven operators were measured in terms of the biosignal. In short, the number of operators whose eye movement and biosignal could be perfectly recorded was seven. This is a reason of selection of operators A–G which were shown in previous Section 2.1.
The velocity and distance of saccade from the seven operators were investigated to check hypotheses H4 and H5. Figure 7 shows a result of analysis for the saccade velocity for all operators. The graphs (a) and (b) show the transitions of the saccade velocity at high-speed driving (say vh) and the other at low-speed driving vs, respectively. Concerning the distance of saccade, which are dh of high-speed driving and ds of low-speed driving, the results are shown in Figure 8. In both cases, strong correlations (correlation coefficients are r=-0.76,-0.70 for the saccade velocity, and r=-0.72 for the saccade distance at the high-speed driving) were confirmed. These tendencies coincide with hypotheses H4 and H5 which were shown in Table 2. Therefore, it was decided that the estimating equation of operation skill was derived using vh, dh, and vs from data of the valid seven operators.
Transition of velocity of saccade in a cooperative-carrying task.
At high-speed driving
At low-speed driving
Transition of distance of saccade in a cooperative-carrying task.
At high-speed driving
At low-speed driving
For development of the estimating equation based on the eye movement data, it is necessary to find a relation between scalar variable S and indexes of eye movement. For this aim, a conversion from three variables (vh, dh, and vs) into one scalar variable, that is named the eye movement feature, is obtained through the principal component analysis (PCA). Specifically, the factor of eye movement on ith trial xi (i=1,…,10) is defined as follows:
(1)xi:=[vih,svi,hdi]T.
Using xi, the observation data matrix X is defined as
(2)X≔[x1,…,x10]=[v1h⋯v10hv1s⋯v10sd1h⋯d10h].
After a variance-covariance matrix V was computed from X, an eigenvector ωm corresponding to mth maximum eigen value λm is derived from V. Then mth principal component of ith trial, zm·i, is obtained as
(3)zm·i=ωmT·xi.
Defining Vmm as mth diagonal element in Vmm, a contributing rate of mth principal component δm is given by
(4)δm=λm∑m=13Vmm.
Figure 9 shows the contributing rate computed by (4) using data concerning operators A–G. Since each first principal component occupied more than 90 percent in case of any operator, only the first principal component was utilized as a value of the eye movement feature finally.
Cumulative contribution ratio of each principal component.
Although only common characteristics among all operators were decided in this state, difference of individual property was not investigated yet; hence, the weighting vectors concerning each operator were checked. Elements ω1,1, ω1,2, and ω1,3 in the weighting vector ωm=1T corresponding to the first principal component are summarized in Table 3. Describing the weighting vectors of the first principal component of jth operator as jω1, the simple similarity that is defined by
(5)θj,k=cos-1(ω1jω1k|ω1j|·|ω1k|)j,k=1,2,…,7,j≠k
was computed to all combination with j and k. Then data of the operator D was eliminated by an outlier test (P<0.1) against all simple similarities computed by (5). The normality of the remained data except the operator D case was confirmed using the Lilliefors test (P<0.1). Therefore, the mean vector ω- was defined as average of the weighting vectors without the operator D case. Values of ω- are described at the bottom row in Table 3. Finally the value of eye movement feature Li of ith trial was computed as
(6)Li=ω-T·xi(∈ℛ1).
Elements of weighting vectors of first principal component.
Participant
ω1,1
ω1,2
ω1,3
Similarity
Contribution ratio
A
0.64
0.64
0.43
0.00
0.94
B
0.59
0.55
0.60
0.20
0.97
C
0.63
0.49
0.61
0.24
0.96
D
−0.08
0.71
0.70
0.80
0.95
E
0.71
0.67
0.22
0.22
0.92
F
0.58
0.59
0.56
0.16
0.93
G
0.60
0.55
0.58
0.18
0.91
ω-
0.63
0.58
0.50
—
—
3.3. Estimating Equation of Skill
After computing Li for six operators using (6) from raw data of factors of eye movement (vih, vis, and dih), correlation analysis found the strong correlation between Li and the simple skill level which is the number of trial i(r=-0.84~-0.56). This correlation analysis, however, does not care the bias which differs depending on each operator’s Li. Additionally, it is natural to think that the value of the index increases as an operator gets skilled up. With this, an estimating equation of skill was determined as follows by considering bias which was computed from the first and second trial data:
(7)S^i=-(Li-(L1+L2)2)i=1,…,10.
Figure 10 shows the estimation of skill level computed using (6) and (7) from raw data of factors of eye movement (vih, vis, and dih) in case of operators A–C and F-G. It was confirmed that the estimated skill value increases roughly as the number of trials increases. The correlation factors between those estimated values and the simple skill levels are as high as 0.56–0.84, and statistically high correlation was confirmed. Therefore, (7) is used in the following as the estimating equation of skill level S^ of the cc-task operation.
Transition of estimated skill level S^.
4. Analysis of Fatigue and Stress
In this section, how to compute index values of fatigue and stress is mentioned. Medically, HR (heart rate) is affected by sympathetic function; the larger value means higher mental or physical load, and the smaller values means the relaxing status. HR is small when a person is bored status in menial jobs. HR is also small when the alertness level is low. It is, however, said that HR becomes large when a person chafes even if he/she is in menial jobs [27].
As another index using heartbeat, coefficient of variation of R-R interval (RRI), that is denoted as Cvrri, is known to be affected by parasympathetic function. RRI is the time elapsing between two consecutive R waves in the electrocardiogram. It is said that fatigue is low (high) when Cvrri shows large (small) value [28].
One of other methods to evaluate the stress level is utilization of chaotic property of the pulse wave. Specifically Lyapunov exponent of chaotic trajectory, which is computed from acceleration component of pulse wave, is used as an index of the stress and relax. The larger value means higher stress (or concentration) status [18]. In the following subsection, effectiveness of these three indexes to the cc-task operation is verified for latter analyses.
4.1. How to Compute HR, Cvrri, and Lyapunov Exponent4.1.1. HR
HR used in the present analysis was computed as follows by using mean of R-R interval per trial. Denoting this mean value by τ, HR is computed as
(8)HR=60τ.
Transitions of HR of operators A–G and the corresponding linear regression lines are shown in Figure 11. It is found that all operators except G show nearly monotonic decrease.
Transition of HR of all operators.
4.1.2. Cvrri
Using standard deviation of RR-interval per trial (say v), Cvrri is computed by
(9)Cvrri=vτ.
Transitions of Cvrri and the corresponding linear regression lines are shown in Figure 12. The figure shows that variance of Cvrri is larger than that of HR and there are individual differences of an increase or decrease tendency among operators.
Transition of Cvrri of all operators.
4.1.3. Chaotic Analysis of Mental Stress
Based on the Takens method [29], chaotic trajectory was computed from electrocardiographic waveform. Using Lyapunov exponents computed from the trajectory, level of stress of operators was investigated. The details are as follows.
In the phase of computation of the Lyapunov exponent, the Poincaré section was put in the hyperspace of the chaotic trajectory so as to intersect to the movement on the trajectory with largest speed. Next, after the intersection points of the trajectory against the Poincaré section are found, the positional vector toward the points, ρj, is computed, where j (=1,2,…) is the number of passages through the Poincaré section. Then Lyapunov exponent γj is computed as
(10)γj=1Tjln|ρj+1||ρj|,
where Tj is the time interval between jth passage to j+1 one. For analysis of the cc-task, γ which is a mean of γj was adopted as a Lyapunov exponent to evaluate level of stress and relaxation. Transitions of γ of all operators and the linear regression lines are shown in Figure 13. Variance of γ was larger than that of Cvrri that was shown in Figure 12. From this result alone, it is difficult to find distinguishing feature at a glance.
Transition of Lyapunov exponent γ of all operators.
4.2. Outlier Test and Correlation Analysis
When seeing distribution in Figures 11, 12, and 13, it is seemed that several outliers exist in those data. Therefore, outliers were eliminated by Smirnov-Grubbs’ outlier test (P<0.1) against Mahalanobis distances of the distribution of data points of HR, Cvrri, and γ for each operator. After the elimination, correlation analysis was performed using all combinations between {T-,P-,andC-indexes} and {HR,Cvrri,andγ}. The details of the correlation analysis are mentioned below.
4.2.1. Stress Analysis by HR
Computing correlation factor between the number of trials and HR (Trial/HR), those factors and slopes of the regression lines are summarized in Table 4. The table shows that factors of six operators except operator G have same sign, and their absolute values are large. From this result, the data of operator G was eliminated for later analysis due to its exceptional difference of the sign. Then, since HR of all the remained operators decreased as the trial increased, we can interpret this to mean that they were getting relaxed. However, HR of several operators still continues decreasing at ten trial; hence, they might be bored status in menial jobs at that time.
Slopes of the regression lines and correlation coefficients in relation concerning HR.
Operator
A
B
C
D
E
F
G
Trial/HR
Slope
−1.22
−0.43
−0.50
−1.20
−0.26
−0.29
0.42
c.c.
−0.95
−0.70
−0.57
−0.98
−0.44
−0.72
0.43
T-index/HR
Slope
0.010
0.002
0.004
0.031
0.009
0.003
−0.030
c.c.*
0.77
0.84
0.59
0.97
0.68
0.65
−0.39
c.c.: correlation coefficient.
*Mean: 0.75; S.D.: 0.140338 (except G).
4.2.2. Relation between Collision and Fatigue-Stress
Next we ascertained whether operational error such as collisions was increased due to fatigue or stress. Specifically, correlations of the following combination were investigated: C-index and HR, C-index and Cvrri, and C-index and γ. From these correlations analyses for operators A–F, the obtained correlation factors and slopes of the regression lines are summarized in Table 5. Sign of value of the operator F in terms of the C-index/γ differed from others; hence, the operator F was eliminated due to its strong individual difference. Seeing correlation factors of C-index/HR concerning the remained operators A–E, their signs vary widely, and the absolute values are as small as about 0.2. Relation between C-index and Cvrri has similar tendency. From these results, we cannot conclude that stress (from relation with HR) or fatigue (from relation with Cvrri) raised collision. This result suggests that C-index is inadequate to evaluate the operational skill in this cc-task, so we decided not to use C-index in later analyses.
Slopes of the regression lines and correlation coefficients in each relation with C-index.
Operator
A
B
C
D
E
F
C-index/HR
Slope
−2.07
−0.49
−0.68
-0.88
−0.56
0.05
c.c.
−0.31
0.27
−0.23
0.14
−0.46
0.05
C-index/Cvrri
Slope (×10−3)
−2.50
0.88
−4.50
2.10
−5.50
−2.80
c.c.
−0.16
0.14
−0.34
0.14
−0.48
−0.36
C-index/γ
Slope (×10−6)
−1.39
−0.30
−1.88
−5.60
−2.32
6.50
c.c.
−0.14
−0.01
−0.66
−0.13
−0.33
0.48
c.c.: correlation coefficient.
4.2.3. Stress Analysis Using Chaotic Property
Using Lyapunov exponents γ of all operators except F and G (who were eliminated as exceptional cases), correlation analyses against T-index and P-index were performed, respectively. The results are shown in Table 6. This table shows that sign of operator D differs from others; hence, this case can be thought as additional exception. Since all correlation coefficients of T-index/γ of the remained operators are plus, chaotic properties are small when the trial time is small; hence, it can be said that mental stress was also small. This is a similar tendency to the result which was previously shown in Table 4. Values of correlation coefficients of T-index/γ shown in Table 6 were, however, small (M=0.24, SD=0.25, N=4), although other values of correlation coefficients of T-index/HR shown in Table 4 were large (M=0.75, SD=0.14, N=6). Therefore, it can be said that HR is more adequate than Lyapunov exponent in order to evaluate mental stress in the cc-task operation.
Slopes of the regression lines and correlation coefficients in relation concerning Lyapunov exponent.
Operator
A
B
C
D
E
T-index/γ
Slope (×10−8)
0.06
0.80
2.62
−9.92
1.14
c.c.*
0.01
0.18
0.60
−0.58
0.18
P-index/γ
Slope (×10−5)
−0.24
−0.12
−0.03
1.05
−0.10
c.c.
−0.37
−0.15
−0.03
0.59
−0.23
c.c.: correlation coefficient.
*Mean: 0.24; S.D.: 0.249757 (except D).
With regard to P-index/γ, the tendency of operators except D is same, as shown in Table 6. Negative signs in those correlation of coefficients indicate that value in Lyapunov exponent was large when frequency of pause of vehicle manipulation was small. In other words, mental stress (or concentration) of the operator was high when the vehicle was controlled constantly. This interpretation can be accepted naturally; however, the absolute values of those coefficients are not so large (0.03–0.37); hence, the stress analysis using chaotic property was not reliable.
5. Relation between the Estimating Equation of Skill and Stress-Fatigue
As demonstrated in Section 4, it was found that HR and Cvrri were effective for evaluation of stress and fatigue, respectively. In this section, effects of these indexes to the estimating equation of skill S^, which was derived in Section 3, are mentioned. From factors of eye movement, the eye movement feature L was computed by (6), and the estimated level of skill S^ was obtained using (7). Correlation coefficients between S^ and the internal status in case of operator A, B, C, and E are shown in Table 7.
Correlation coefficients between the estimated skill level and indexes of internal status.
Operator
A
B
C
E
Against HR
−0.76
−0.74
−0.21
−0.14
Against Cvrri
−0.77
−0.36
−0.32
−0.59
With respect to HR, each HR in four operators decreases as each estimated skill level increases since signs of the correlation coefficients are all minus. That means that they were in a tense situation at first, but they got relaxed as they became skilled. In other words, the estimated skill level derived using measurement of eye movement is affected by both operational skill and condition of stress. On the other hand, correlation coefficients concerning Cvrri are all minus, and Cvrri decreases as they became skilled. That is, fatigue increases as they became skilled.
With this, it was confirmed that the presented equation to estimate the operational skill based on the eye-movement measurement reflects effect of stress and fatigue, although the equation can estimate the skill level adequately.
6. Conclusion
To observe learning process of the machine operation, a cooperative carrying task (cc-task) simulator system was designed, and the eye movement, electrocardiographic waveform, and log data of the machine operators were recorded using this simulator. From the recorded data, a relation between the operational skill, mental stress, and fatigue was analyzed. First, outliers data caused by nonstandard behavior and failure in the measurement were eliminated by several types of outlier tests. After extracting valid data which satisfy statistical normality, the standard estimating equation of the operational skill level based on the eye-movement data was derived. Specifically, factors of the eye movement having strong correlation with operational skill were selected, and those factors were converted into the eye movement feature value through a principal component analysis. Then the estimating equation was derived. At the decision phase for the factors of eye movement, it was confirmed that velocity and distance of saccade were adequate for them. And effectiveness of the derived equation could be confirmed since the correlation coefficients between the estimated skill value computed by the equation and actual simple skill level were sufficiently high as r=0.56–0.84.
Second, biological signal such as the heart rate (HR) and the coefficient of variation of R-R interval (Cvrri) of operators in the cc-task work were analyzed. As a result of correlation analysis against simple skill levels, it was found that HR and Cvrri were effective for evaluation of stress and fatigue, respectively. Effectiveness of other stress analysis using chaotic properties in electrocardiographic waveform was, however, not shown.
Finally, from the correlation analysis between the biological information (HR and Cvrri) and the estimating equation of the operational skill, it was confirmed that the equation reflects effect of stress and fatigue, although the equation can estimate the skill level adequately.
Acknowledgments
The present study was supported by a Grant-in-Aid for Scientific Research (A) of the Japanese Ministry of Education, Culture, Sports, Science, and Technology. Software development of the cooperative-carrying task was supported by Hiroshi Igarashi. The experiment was helped by many participants who embraced our requests kindly. The present authors appreciate their cooperation.
FurutaK.Control of pendulum: from super mechano-system to human adaptive mechatronicsProceedings of the 42nd IEEE Conference on Decision and ControlDecember 2003Hawaii, USA149815072-s2.0-154229905510.1109/CDC.2003.1272824HarashimaF.SuzukiS.Intelligent Mechatronics and Robotics keynote speechProceedings of the IEEE International Conference on Emerging Technologies and Factory Automation, in CD-ROM2008Hamburg, GermanySuzukiS.Human adaptive mechatronics: skill acquisition in machine manipulation20104228352-s2.0-7795385057710.1109/MIE.2010.936762SuzukiS.KuriharaK.FurutaK.HarashimaF.Assistance control on a haptic system for human adaptive mechatronics20062033233482-s2.0-3364597433110.1163/156855306776014349FurutaK.KadoY.ShiratoriS.SuzukiS.Assisting control for pendulum-like juggling in human adaptive mechatronics201122567097202-s2.0-8005497834610.1177/0959651811403373SuzukiS.FurutaK.Adaptive impedance control to enhance human skill on a haptic interface system201220121036506710.1155/2012/365067BaronS.KleinmanD. L.LevisonW. H.An optimal control model of human response part II: prediction of human performance in a complex task1970633713832-s2.0-0014792421McRuerD. T.KrendelE. S.Mathematical models of human pilot behavior1974188NATO Advisory Group for Aerospace Research and DevelopmentGrayW. D.JohnB. E.AtwoodM. E.Project ernestine: validating a GOMS analysis for predicting and explaining real-world task performance1993832373092-s2.0-0027840057OhnoT.OgasawaraH.Information acquisition model of highly interactive tasksProceedings of the International Conference on Cognitive Science (ICCS/JCSS '99)1999Tokyo, Japan283293LandM. F.LeeD. N.Where we look when we steer199436964837427442-s2.0-002822567610.1038/369742a0CardS. K.MoranT.NewellA.1983Hillsdale, NJ, USALawrence Erlbaum AssociatesHarashimaF.SuzukiS.Future of mechatronics and human machine operation skill and visual perception2008111825SuzukiS.HarashimaF.FurutaK.Human control law and brain activity of voluntary motion by utilizing a balancing task with an inverted pendulum20102010162-s2.0-7834926387110.1155/2010/215825215825BhuiyanM. S.Driver assistance systems to rate drowsiness: a preliminary study20091994154252-s2.0-6544914519910.1007/978-3-642-00909-9_40YagiA.Variations of the eye fixation related potentials in VDT tasks199935522523TakahashiK.InoueH.A quantitative analysis on stress and fatigue for VDT workers measured by Heart Rate Variability (HRV)20093017AokiH.FujimotoY.SuzukiS.Sato-ShimokawaraE.YamaguchiT.Physiological responses on greeting with robot under difference of culture20121723546510.4303/acpsf/235465SuzukiS.KuronumaK.HarashimaF.Development of gaze detection system for multiple users: toward realization of an intelligent service interfaceProceedings of the International Joint Conference (ICROS-SICE '09)August 2009Fukuoka, Japan324532502-s2.0-77951113352IgarashiH.SuzukiS.KobayashiH.HarashimaF.Role sharing analysis on multi-operator cooperative workProceedings of the 18th IEEE International Symposium on Robot and Human Interactive (RO-MAN '09)October 2009Toyama, Japan106010652-s2.0-7284914238810.1109/ROMAN.2009.5326163KolodkoJ.SuzukiS.HarashimaF.Eye-gaze tracking: an approach to pupil tracking targeted to FPGAsProceedings of the IEEE IRS/RSJ International Conference on Intelligent Robots and Systems (IROS '05)August 2005Alberta, Canada294229472-s2.0-3454812395610.1109/IROS.2005.1545423MiuraT.ShinoharaK.Safety problems of the car navigation from the viewpoint of psychology of attention20012643344OhnoT.OgasawaraH.Information Acquisition Model of Highly Interactive TasksProceedings of the International Conference on Cognitive Science/Annual Meeting of the Japanese Cognitive Science Society Joint Conference (ICCS/JCSS '99)1999Tokyo, Japan288293ItoT.MatsubaraH.GrimbergenR.Cognitive science approach to Shogi playing processes(1)-some results on memory experiments2002431029983011SakitaK.OgawaraK.MurakamiS.KawamuraK.IkeuchiK.Flexible cooperation between human and robot by interpreting human intention from gaze informationProceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS '04)October 2004Sendai, Japan8468512-s2.0-14044266955KuronumaK.SuzukiS.IgarashiH.Estimation method of awareness level using an iconic model for support robotsProceedings of the IFToMM Asian Conference on Mechanism and Machine Science, in CDROM2010Taipei, TaiwanOhsugaM.ShimonoF.GennoH.Assessment of phasic work stress using autonomic indices20014032112202-s2.0-003513189510.1016/S0167-8760(00)00189-6TakeishiM.MurataK.Evaluation of child neurophysiological function in the environmental epidemiology20052124243246TakensF.RandD. A.YoungL. -S.Detecting strange attractors in turbulence1981898New York, NY, USASpringer366381Lecture Notes in Mathematics