This paper proposes and evaluates a multiagents system called NORA that predicts emotional attributes from learners' brainwaves within an intelligent tutoring system. The measurements from the electrical brain activity of the learner are combined with information about the learner's emotional attributes. Electroencephalogram was used to measure brainwaves and self-reports to measure the three emotional dimensions: pleasure, arousal, and dominance, the eight emotions occurring during learning: anger, boredom, confusion, contempt curious, disgust, eureka, and frustration, and the emotional valence positive for learning and negative for learning. The system is evaluated on natural data, and it achieves an accuracy of over 63%, significantly outperforming classification using the individual modalities and several other combination schemes.
All around us are hand-compatible tools and machines and keyboards, designed to fit the hand. We are not apt to think of them in that light, because it does not happen to us that anyone would bring out some device to be used by human hands without being sure that the nature of hands was considered. A keyboard machine or musical instrument that called for eight fingers on each hand would draw instant ridicule. Yet, we force millions of children into schools who have never seriously studied the nature and shape of the human brain and who not surprisingly prove actively brain antagonistic [
An Increasing number of researches in the field of artificial intelligence recognize that cognition, motivation, and emotions are three fundamental components of learning [
These sensors provide physiological data that is streamed to the tutoring software for students in real educational settings. These physiological responses have been linked to various affective states [
Unfortunately, all the sensors presented above measure only the physical reactions of the learner. Most of them are based on external appearance of the learner, on his ability to be expansive, to be physically in good conditions to move, and to have an interactive character. It is not always the case. Sometimes learners are physically disabled. In some cultures, learners do not express their emotions in their face or by moving. They can adopt a nonresponsive attitude when they interact with the machine.
Our research investigates the use of a new sensor: the electroencephalogram (EEG). This system allows recording the data directly from the brain through the electrical brain activity of the learner during emotional tasks. Learners interact with an intelligent tutoring system by learning and feeling particular emotions. Measuring the electrical brain activity during learning has an important role in helping students to reach the best conditions for learning; in fact, many research works suggest that learning to consciously control our brainwave states could help us increase our ability to concentrate and decrease our stress levels [
We present here a framework called NORA to automatically capture process and model the electrical brain activity of the learner for recognizing some emotional attributes that occur during learning situations. NORA is a multiagent system that will communicate with an intelligent tutoring system to enrich the learner module and give more precise information about the learner state. It will also help in developing theoretical understanding of the learner's electrical brain activity behavior in learning situations.
This work tackles a number of challenging issues. While most of the prior work on detecting learners' emotions has used popular sensors like camera, mouse, and wrist, our research focus on exploring the electrical brain activity by using an electroencephalogram as it is crucial to take into consideration the non expansive and disabled learners who do not move or express their emotions on their face. Second, despite the advances in face analysis and gesture recognition, the real-time sensing of nonverbal behaviors is still a challenging problem. In this work we demonstrate a multiagent system that can automatically extract brainwaves and features from the interaction between the learner and the ITS, which can be used to detect some emotional attributes like the frequent emotions occurring during learning.
In the human brain, each individual neuron communicates with the other by sending tiny electrochemical signals. When millions of neurons are activated, each contributing its small electrical current, they generate a signal that is strong enough to be detected by an EEG device [
Commonly, brainwaves are categorized into 4 different frequency bands, or types, known as delta, theta, alpha, and beta waves. Each of these wave types often correlates with different mental states. Table
Brainwaves and mental states.
Wave | Frequency | Mental state |
---|---|---|
Delta | 0–4 Hz | Deep sleep; hypnosis; increasing immune functions; physical and mental restructuring. |
Theta | 4–8 Hz | Deep relaxation; state of meditation; increase in creativity |
Alpha | 8–12 Hz | Mental and muscular relaxation; positive thought; improved memory; assimilation and capacity for study. Improved performance in sport. |
Beta | >12 Hz | Relaxed focus; improved attention; conscious attention, tension anxiety-stress; confusion; irritability; psychosomatic problems |
Delta frequency band is associated with deep sleep. Theta is dominant during dream sleep, meditation, and creative inspiration. Alpha brainwave is associated with tranquility and relaxation.
Beta frequency band is associated with an alert state of mind, concentration, and mental activity [
Subbands in beta frequency band.
Wave | Frequency | Mental state |
---|---|---|
Beta1 | 12–15 Hz | Relaxed focus; improved attention. |
Beta2 | 15–20 Hz | Vigilance; logical reasoning; conscious attention. |
Beta3 | >20 Hz | Fully awakened state; vigilance; tension-anxiety-stress; confusion; irritability; psychosomatic problems. |
Thus, the electrical brain activity can be filtered into 6 frequency bands. Research has shown that, although one brainwave state may predominate at any given time, depending on the activity level of the individual, the remaining five brain states are present in the mix of brainwaves at all times. In other words, while somebody is in attentive state and exhibiting a beta2 brainwave pattern, there also exists in that person's brain a component of Beta3, Beta2, Alpha, Theta, and Delta, even though these may be present only at the trace level [
The goal of this study is to implement and evaluate a system that recognizes some emotional attributes from brainwaves during learning situation.
First, the study implements a framework called NORA, a
Second, the study explores the link between the emotions frequently occurring during learning and brainwaves. In a previous study conducted byD'Mello et al. [
Third, the study measures the correlation between brainwaves and the emotional dimensions
NORA addresses all the challenges mentioned above by extracting the electrical brain activity of a learner during their interaction with an ITS, classifying emotional attributes related to learning situation. The system uses real-time tracking of brainwaves and interaction features to extract variables related to the learner. Brain sensory information and other characteristics of the learner are combined using machine learning techniques.
To measure the learners’ brainwaves, we use Pendant EEG
Pendant EEG tool kit.
Electrode placement was determined according to the 10–20 International System of Electrode Placement [
The 10–20 International System of electrode placement.
Pendant EEG sends the electrical signals to the computer via an infrared connection. Being light and easy to carry, it is not cumbersome and can easily be forgotten within a few minutes. The learner wearing Pendant EEG is completely free of his movements as shown in Figure
Learner wearing Pendant EEG.
The participants used in this study consisted of seventeen (10 women) undergraduates with a mean age of
NORA is a multiagent system entirely implemented in Java. Our choice for multiagent architecture is motivated by the numerous advantages of this type of architectures: robust, scalable, flexible, and rapid for integration since NORA is willing to communicate with any intelligent tutoring system. Figure
NORA overall architecture.
NORA includes 6 main agents distributed into two groups. Group 1 is the Data Collection Group. It includes Agent A1: Wave Input (WI), Agent A2: Interact Input (II), and Agent A3: Data Base (CDB). Group 2 is the Prediction Group; it includes Agent A4: Brain profile (BP), Agent A5: Emo Wave, and Agent A6: Demo Wave (DW). The following explains the role of each agent. Agent WI receives brainwaves amplitudes while agent II receives information concerning the interaction between the learner and the ITS. Agent DB stores these two categories of input into a central database (CDB) after a filtration of some variables. For instance, variables coming from brainwave signals and learner’s interactions need to be prepared or calculated for further machine learning processes. Agent EW requests data from the central data base and predicts the emotional and cognitive state of the learner from his cerebral waves. EW is able to predict these states by using machine learning techniques applied on CDB data and send this information on request of BP agent. Agent DW requests data from the central data base about the emotional and cognitive state of the learner in order to select a method for inducing cerebral waves able to improve learning. DW sends to BP agent a list of hearing and visual stimuli to apply for inducing specific brainwaves useful for learning. Agent BP plays the role of a supervising agent; it requests information from EW and DW agents to determine the cerebral profile of the learner, update the learner module, and select a pedagogical neuro-strategy to apply in order to improve learning.
NORA agents present the following properties (according to the agent theory). They are autonomous and control their internal state and actions. They cooperate with other agents using the FIPA agent communication language [ They are reactive to the environment (other agents’ requests, ITS actions). They are proactive, detect brainwaves evolution during learning, and are able to induce emotional stimuli to change the emotional state of the learner.
Communication between two agents.
The following scenario is presented to understand how NORA communicates with the ITS and how the agents coordinate their tasks. Agents WI and II send their collected data in real time to Agent DB. Agent DB stores the data in the central database (CDB). Agent DB sends its notifications to Agents EW and DW about the availability of new data collected from Agents WI and II. Agents EW and DW send their requests to Agent DB to get the data in a certain format to reinforce their models of prediction. Agent BP receives a request for prediction from the Learner Module in the ITS. The request is about to predict some emotional attributes from learners’ brainwaves. Agent BP recruits one or both of Agents EW or DW to answer the request. If Agent EW and DW are both selected for a request, they negotiate to optimize their results. Agent BP sends the results of prediction to the ITS. It also evaluates the prediction and sends it to EW or DW so that they update their belief base (their models of prediction).
The International Affective Picture System (IAPS) is a large colored bank of pictures. It provides the ratings of emotions. It includes contents across a wide range of semantic categories. IAPS is developed and distributed by the NIMH Center for Emotion and Attention (CSEA) at the University of Florida in order to provide standardized database that are available to researchers in the study of emotion and attention. IAPS has been characterized primarily along the dimensions of valence, arousal, and dominance. Even though research has shown that the IAPS is useful in the study of discrete emotions, the categorical structure of the IAPS has not been characterized thoroughly. Mikels et al. [
This study provided categorical data that allows the IAPS to be used more generally in the study of emotion from a discrete categorical perspective. In accord with previous reports [
To assess the three dimensions of pleasure, arousal, or dominance, we use the Self-Assessment Manikin (SAM), an affective rating system devised by Lang et al. [
Self-Assessment Manikin system
For the pleasure dimension, SAM ranges from a smiling, happy figure to a frowning, unhappy figure. For the arousal dimension, SAM ranges from an excited, wide-eyed figure to a relaxed, sleepy figure. For the dominance dimension, SAM ranges from a small figure (dominated) to a large figure (in control). Ratings are scored such that 9 represents a high rating on each dimension and 1 represents a low rating on each dimension.
Agent WI samples, digitalizes and filters the original signal recorded by the EEG, and divides it into six frequency bands: delta, theta, alpha, beta1, beta2, and beta3 (Figure
A raw EEG signal is filtered into its component frequencies.
After filtering, Agent WI calculates the signals average amplitude or strength and analyze the average amplitude of the signal to determine in which mental state (among those presented in Tables
Description of the information collected by Agent II.
Channel | Subchannel | Description |
---|---|---|
Permanent information | Gender | Gender |
Birth date | NORA considers the year of birth | |
Periodic information | Personality trait (PT) | Using big five personality test [ |
Learning style (LS) | Using the VARK Test [ | |
Dynamic information | Emotion in learning (EL) | According to D'Mello's Study [ |
Emotional dimension (ED) | According to Lang's Study [ |
Agent II collects some data exchanged between learners and the ITS. During their interactions with the ITS, learners provide some useful information needed by NORA’s Agents to build their prediction’s models of some emotional attributes from brainwaves. Table
Agent DB collects real-time data from Agent WI and Agent II. Its main task is to update the central database CDB presented in Figure
CDB schema in Agent DB.
The CDB contains the following tables. Not all the tables are considered by this study but will serve in future studies. Learners table: contains permanent and periodic information about the learner (personality traits: PT and learning style: LS). Emotions table: contains dynamic information about the learner (emotions felt during learning: EL and emotional dimensions: ED). Lessons table: contains the knowledge content followed by the learner. Quiz table: contains the results of the learner to the questions about the lessons he followed. Stimuli table: contains sensorial stimuli that induce particular emotions. In this study, we used visual stimuli: hundred of photos from the IAPS. Table Brainwave’s Amplitudes Table: Contains the recording of the electrical brain activity (the brainwaves’ amplitudes) and the associated emotions (EL and ED), quiz, lessons, and emotional stimuli (images’ ID from the IAPS).
Agent BP supervises Agent EW and Agent DW. Once Agent BP assigns a task to one of the Agents or both, Agents EW and DW communicate to cooperate and select their local choice to perform the task or not. For example, an Agent can decide to do not perform a task due to its weak model of prediction; this Agent decides to get more data from Agent DB, to perform more training and increase its percentage of prediction.
Both Agent EW and DW follow a markov decision process (MDP) to select a task and maximize its gain. MDP provides for Agents EW and DW a mathematical framework for modeling decision making in situations where Agents EW and DW have to choose between: (1) to get data from Agent DB to built a new model of prediction, (2) to get data from Agent DB to strengthen an existing model of prediction and (3) to answer Agent BP’s request and predict emotional attributes from brainwaves.
The strategy of coordination is based on a sequential allocation of tasks during several cycles of negotiation. At each step, an Agent decides to perform a task or to ignore it. Decision is based on the available resources.
For classification purpose, Agents EW and DW use algorithms from the Weka Library [
Agent BP attributes tasks and sends feedback to Agent EW and Agent DW. It also communicates with the ITS to get some additional information to take its decisions. For its function, Agent BP adopts a markov decisional process (MDP) Behavior. The model maximizes the expected gain in a long-term perspective.
Consider
The global gain
The data considered for our study is located in the brainwave’s amplitudes table described in Figure
For Agent EW, we need to consider all the recordings that associate any of the eight emotions (anger, boredom, confusion, contempt, curious, disgust, eureka, and frustration) to any of the four brainwaves (Delta, Theta, Alpha and Beta).
Agent DW needs to build a model that predicts the value (1 to 9) of any of the emotional dimensions (pleasure, dominance, arousal) from any of the four Brainwaves Amplitudes (Delta, Theta, Alpha and Beta).
Thus, the view that we consider from the brainwaves' amplitude table is
The size of the sample we collected is 31599 records. The amplitudes
For each dimension, we want to obtain a finite number of classes. We rounded each value
Table
Percentage of occurrences for each emotion.
Emotion | Instances from 31599 | Percentage from 31599– |
---|---|---|
Anger | 1746 | 8.88% |
Boredom | 1294 | 6.58% |
Confusion | 1500 | 7.63% |
Contempt | 754 | 3.83% |
Curious | 4217 | 21.44% |
Disgust | 3937 | 20.01% |
Eureka | 5016 | 25.50% |
Frustration | 1208 | 6.14% |
Others, or undefined | — |
Percentages of emotions are calculated on the 19672 relevant instances. We do not consider the 11927 other or undefined emotions.
Table
Percentages of observations of the three major emotions as a function of rating classes.
Rating classes | Rating frequencies, percentage (on 31599) | |||||
Pleasure ( | Arousal ( | Dominance ( | ||||
1349 | 04% | 71 | 01% | — | — | |
4528 | 14% | 1825 | 06% | 778 | 02% | |
4393 | 15% | 5345 | 17% | 5519 | 17% | |
3546 | 11% | 8479 | 26% | 5779 | 19% | |
5458 | 17% | 9893 | 31% | 11124 | 35% | |
6292 | 20% | 5412 | 17% | 8018 | 26% | |
5523 | 17% | 574 | 02% | 381 | 01% | |
510 | 02% | — | — | — | — |
Due to a low frequency of observations, the classes
This resulted in a further reduction in the database from 31599 to 29740 for pleasure dimension with 4528 instances of
Our database contains noise due to the limitations of the material used and the learners’ response time. We took that into consideration in our choice of training algorithms. Among those that have the advantage of being robust to noisy training data and effective when that data is large, IBK gave us the best results. IBK is an implementation of
Results among k values.
Table
Accuracy by emotion.
Classified as | Precision | Recall | Precision | |
---|---|---|---|---|
Anger | 0.814 | 0.822 | 0.818 | 0.814 |
Boredom | 0.8 | 0.796 | 0.798 | 0.8 |
Confusion | 0.792 | 0.809 | 0.8 | 0.792 |
Contempt | 0.798 | 0.787 | 0.792 | 0.798 |
Curious | 0.814 | 0.817 | 0.816 | 0.814 |
Disgust | 0.829 | 0.836 | 0.833 | 0.829 |
Eureka | 0.84 | 0.831 | 0.835 | 0.84 |
Frustration | 0.825 | 0.807 | 0.816 | 0.825 |
Classification precision varies between 79.2% and 84%. Recall varies between 78.7% and 82.2%. The performance of the classifier, given by F-measure (Table
Confusion matrix.
Classified as | A | B | C | D | E | F | G | H |
---|---|---|---|---|---|---|---|---|
A: anger | 412 | 7 | 7 | 1 | 25 | 11 | 23 | 3 |
B: boredom | 8 | 318 | 5 | 0 | 18 | 12 | 19 | 7 |
C: confusion | 9 | 5 | 374 | 10 | 15 | 16 | 28 | 4 |
D: contempt | 8 | 0 | 5 | 166 | 11 | 12 | 9 | 0 |
E: curious | 13 | 25 | 23 | 10 | 1060 | 64 | 67 | 24 |
F: disgust | 18 | 12 | 18 | 6 | 52 | 976 | 72 | 13 |
G: eureka | 31 | 25 | 31 | 14 | 104 | 65 | 1262 | 9 |
H: frustration | 9 | 12 | 8 | 1 | 11 | 17 | 14 | 285 |
The performance of our classifier varies between 79.2% and 83.5%. The Kappa statistic measures the proportion of agreement between two raters with correction for chance. Kappa scores ranging from 0.4 to 0.6 are considered to be fair, 0.6–0.75 are good, and scores greater than 0.75 are excellent [
To give more weight to emotional states with minority instances, we decided to use Youden’s J-index [
Preliminary analyses revealed significant correlations for all the three dimensions. Spearman correlations in a 4 by 3 matrix are presented in Table
Correlations between brainwaves and emotional dimensions.
Brainwaves | Pleasure | Arousal | Dominance |
---|---|---|---|
Delta | −.026** | −.005 | −.024** |
Theta | −.006 | .035** | −.031** |
Alpha | .005 | −.037** | .015** |
Beta | −.041** | .026** | −.055** |
*Significant at
Knowing the high degree of freedom (
Agent DW performed multiple machine learning techniques, one for each of the three emotional dimensions, with the four brainwave features as predictors. Significant overall relationships were found for all three emotional dimensions: pleasure, arousal and dominance.
Multiple regression analysis results were interesting. The ANOVA table reports a significant
Coefficient of the regression line to predict pleasure from brainwaves.
Model | Unstandardized | Standard. coeff. | sig | ||
Std. error | beta | ||||
Constant | 4.99 | 0.026 | 193.967 | 0.000 | |
Delta | −0.041 | 0.017 | −0.014 | −2.416 | 0.016 |
Theta | −0.022 | 0.017 | −0.007 | −1.314 | 0.189 |
Alpha | 0.033 | 0.016 | 0.012 | 2.041 | 0.041 |
Beta | −0.122 | 0.014 | −0.048 | −8.424 | 0.000 |
Theta brainwave does not contribute much to the model (
Multiple regression analysis results were interesting. The ANOVA table reports a significant
Coefficient of the regression line to predict arousal from brainwaves.
Model | Unstandardized | Standard. coeff. | sig | ||
Std. error | beta | ||||
Constant | 4.63 | 0.016 | 284.401 | 0.000 | |
Delta | −0.08 | 0.011 | −0.009 | −1.657 | 0.097 |
Theta | 0.110 | 0.011 | 0.059 | 10.438 | 0.000 |
Alpha | −0.082 | 0.010 | −0.045 | −7.925 | 0.000 |
Beta | 0.033 | 0.009 | 0.020 | 3.601 | 0.000 |
The two largest absolute standardized coefficients are
Multiple regression analysis results were interesting. The ANOVA table reports a significant
Coefficient of the regression line to predict dominance from brainwaves.
Model | Unstandardized | Standard. coeff. | sig | ||
Std. error | beta | ||||
Constant | 5.01 | 0.015 | 326.529 | 0.000 | |
Delta | −0.020 | 0.010 | −0.012 | −2.033 | 0.042 |
Theta | −0.040 | 0.010 | −0.023 | −4.008 | 0.000 |
Alpha | 0.046 | 0.010 | 0.027 | 4.728 | 0.000 |
Beta | −0.086 | 0.009 | −0.057 | −10.036 | 0.000 |
Theta brainwave do not contribute much to the model (
The multiple regression analyses presented in the previous section produced significant models for all three emotional dimensions: pleasure, arousal and dominance. Agent DW tested many classification algorithms: a
Table
The various classification algorithms were successful in detecting pleasure, arousal, and dominance. Classification accuracy varies from 58.54% to 75.16%. Kappa statistic measures the proportion of agreement between two raters with correction for chance. It is fair for Classification via Regression algorithm (
Comparison of classification techniques results.
Algorithm | Classification accuracy % (kappa statistic) | ||
Pleasure | Arousal | Dominance | |
Nearest Neighbor | 73.55 (0.71) | 74.86 (0.72) | 75.16 (0.71) |
J48 Decision tree | 66.33 (0.64) | 68.51 (0.64) | 68.92 (0.64) |
Bagging | 74.66 (0.72) | 74.79 (0.71) | 75.29 (0.71) |
Classification via regression | 59.01 (0.55) | 58.54 (0.53) | 58.93 (0.52) |
Precision by emotional dimension.
Classes | Precision | Recall | |||||||
2 | 0.74 | 0.71 | 0.71 | 0.77 | 0.73 | 0.74 | 0.75 | 0.72 | 0.73 |
2.5 | 0.73 | 0.70 | 0.63 | 0.75 | 0.74 | 0.69 | 0.74 | 0.72 | 0.66 |
3 | 0.74 | 0.71 | 0.74 | 0.75 | 0.74 | 0.78 | 0.75 | 0.72 | 0.76 |
3.5 | 0.71 | 0.72 | 0.74 | 0.72 | 0.74 | 0.75 | 0.72 | 0.73 | 0.75 |
4 | 0.72 | 0.76 | 0.74 | 0.73 | 0.77 | 0.76 | 0.72 | 0.76 | 0.75 |
4.5 | 0.72 | 0.77 | 0.74 | 0.72 | 0.77 | 0.75 | 0.72 | 0.77 | 0.74 |
5 | 0.75 | 0.76 | 0.77 | 0.76 | 0.76 | 0.77 | 0.75 | 0.76 | 0.77 |
5.5 | 0.69 | 0.78 | 0.77 | 0.69 | 0.76 | 0.76 | 0.69 | 0.77 | 0.76 |
6 | 0.73 | 0.76 | 0.77 | 0.70 | 0.73 | 0.75 | 0.72 | 0.75 | 0.76 |
6.5 | 0.78 | 0.74 | 0.72 | 0.76 | 0.72 | 0.69 | 0.77 | 0.73 | 0.70 |
7 | 0.75 | 0.69 | 0.72 | 0.73 | 0.66 | 0.69 | 0.74 | 0.67 | 0.71 |
7.5 | 0.76 | — | 0.83 | 0.72 | — | 0.79 | 0.74 | — | 0.81 |
Excerpt from the IAPS database.
Picture | Description | Correlations |
---|---|---|
1052 Dangerous snake | Emotion: Fear | |
Valence: Negative | ||
Plaisir: 3.50/9 | ||
Excitation: 6.52/9 | ||
Dominance: 3.36/9 | ||
1463 Happy cats | Amusement | |
Valence: Positive | ||
Plaisir: 7.45/9 | ||
Excitation: 4.79/9 | ||
Dominance: 6.43/9 | ||
1900 Fish in the ocean | Neutral | |
Valence: Posititive | ||
Plaisir: 6.65/9 | ||
Excitation: 3.46/9 | ||
Dominance: 6.07/9 | ||
2058 Happy Baby | Satisfaction | |
Valence: Posititive | ||
Pleasure: 7.91/9 | ||
Excitation: 5.09/9 | ||
Dominance: 6.67/9 | ||
7360 Flies on a cake | Disgust | |
Valence: Négative | ||
Plaisir: 3.59/9 | ||
Excitation: 5.11/9 | ||
Dominance: 5.21/9 |
The nearest neighbor and bagging techniques provided the highest accuracy for each of the three emotional dimensions (
Table
Precision (specificity) and recall (sensitivity) are standard metrics for assessing the discriminability of a given class. The precision for class C is the proportion of samples that truly belong to class C among all the samples that were classified as class C. The recall score (sensitivity or true positive rate) provides a measure of the accuracy of the learning scheme in detecting a particular class. Finally, the F-measure provides a single metric of performance by combining the precision and recall.
Table
To give more weight to the rating classes with minority instances, we decided to use, for each of pleasure, arousal, and dominance the Youden’s
NORA is a multi agent system that analyzes the electrical brain activity occurring during learning. Agent WI, Agent II, and Agent DB capture, filter, and store learners' brainwaves and some other emotional attributes. Agent EW, Agent DW and Agent BP predict emotions and emotional dimensions from brainwaves and communicate the result to the ITS.
The evaluation of NORA demonstrates the effectiveness of using an EEG to measure learners' brainwaves and to predict some emotional attributes. The framework implemented for the study allowed us to record the electrical brain activity of the learners when they were exposed to emotional stimuli. These data were used to predict the eight emotions frequently occurring during learning and the three emotional dimensions. The system is evaluated on natural data and it achieves an accuracy of over 63%, significantly outperforming classification using the individual modalities and several other combination schemes.
We acknowledge that the use of EEG has some potential limitations. Any physical learner’s movement can cause noise that is detected by the electrodes and interpreted as brain activity by Pendant EEG. Nevertheless, we think that the instructions given to participants (to remain very still), the number of participants (seventeen), and the database size (31599 recordings) can considerably reduce this eventual noise.
Agent EW uses kNN algorithm to predict emotions from brainwaves. It yielded a classification prediction of 82.27% with a good kappa statistics value (78%), this shows an acceptable concordance between the predicted emotion and the real emotion.
The multiple regression analyses resulted in accurate predictions for pleasure, arousal, and dominance degrees. Delta brainwave showed a significant negative correlation with pleasure and dominance. Theta and Beta brainwaves showed a significant negative correlation with pleasure and dominance (strong negative for Beta) but a positive correlation with arousal (strong positive for Theta). Alpha brainwave had a significant negative correlation with arousal and dominance.
Agent DW predicts the emotional dimensions from brainwaves with a good accuracy: 73.55% for pleasure, 74.86% for arousal, and 75.16% for dominance. Kappa statistics values are good and, respectively, 71%, 72%, and 71%.
The major advantages of using brainwaves for emotional attributes detection lie in its effectiveness when used in the case of disabled, impassible or taciturn learners. If the grounding criterion hypothesis holds in future replication, then it would give indications on how to help those learners to control their emotions during learning only by using EEG.
If the method described above proves to be effective in identifying the learners emotional attributes, we can direct our focus to a second stage. An ITS would select an adequate pedagogical strategy that adapts to certain learner’s emotional dimensions in addition to cognitive states. In addition to the traditional available pedagogical strategies (learning by explanation, by example, by contradiction, increasing motivation, etc.), the ITS could use the information about brainwave state of the learner in order to select a strategy able to improve the emotional state and to conduct a better receptivity and attention. The pedagogical selection would be guided by a more precise learner model which would include the elements presented in this paper. This adaptation would increase the bandwidth of communication and allow an ITS to respond at a better level.
The authors acknowledge the support of the FQRSC (Fonds Québécois de la Recherche sur la Société et la Culture) and NSERC (National Science and Engineering Research Council) for this work.