We describe analyses of the features of emotions (neutral, joy, sadness, and anger) expressed by Finger Braille interpreters and subsequently examine the effectiveness of emotional expression and emotional communication between people unskilled in Finger Braille. The goal is to develop a Finger Braille system to teach emotional expression and a system to recognize emotion. The results indicate the following features of emotional expression by interpreters. The durations of the code of joy were significantly shorter than the durations of the other emotions, the durations of the code of sadness were significantly longer, and the finger loads of anger were significantly larger. The features of emotional expression by unskilled subjects were very similar to those of the interpreters, and the coincidence ratio of emotional communication was 75.1%. Therefore, it was confirmed that people unskilled in Finger Braille can express and communicate emotions using this communication medium.
The Deafblind Association of Japan estimates that there are nearly 22,000 deafblind people in Japan (2008). Deafblind people use many different communication media, depending on the age of onset of deafness and blindness and the available resources. “
Two-handed Finger Braille and one-handed Finger Braille.
Various Braille input devices have recently been developed [
The objective of this study is the development of a Finger Braille support device which employs the skin-contact communication of deafblind people, because skin contact is the only form of nonverbal communication for deafblind people. The concept of the Finger Braille support device is shown in Figure
Finger Braille support device.
Our support device consists of a Finger Braille teaching system and a Finger Braille recognition system. The teaching system recognizes the speech of a non-disabled person and displays the associated dot pattern of Finger Braille. The non-disabled person can then dot Finger Braille on the fingers of the deafblind person by observing the displayed dot pattern [
In the present study, emotional expression by Finger Braille interpreters (Experiment I) and emotional communication by people who are unskilled in Finger Braille (Experiment II) were examined. This paper presents analyses of the features of emotional expression by Finger Braille interpreters and the effectiveness of emotional expression and emotional communication between unskilled people.
To analyze the features of emotional expression using Finger Braille, emotional expression by interpreters of Finger Braille (Experiment I) was examined [
The subjects were two Finger Braille interpreters (Subjects A and B) with 20 and 11 years of experience, respectively. Both subjects were right-handed. Both subjects gave informed consent after hearing a description of the study. In the experiment, the subject became a sender and a tester became a receiver. The same tester, who was one of the authors, was used throughout the experiment.
Two dialogues were used. “Rain has fallen” or
Plutchik proposed a multidimensional model of emotions and defined joy, anticipation, anger, disgust, sadness, surprise, fear, and acceptance as the basic human emotions [
We decided to investigate the expressed emotions of neutral, joy, sadness, and anger, which are basic emotions developed in early infancy of human beings. These emotions were common to those in several previous studies of emotional expression and communication [
In the previous studies, actors or announcers were selected as the subjects, because of their experiences expressing emotion [
Scenes of “Rain has fallen”
Joy: In the morning of a sport activity, you do not want to participate. If it is raining, the activity would be cancelled. Now rain is falling.
Sadness: On a holiday morning, the weather is fine. You are changing your clothes to go outside. Now rain is falling.
Anger: The probability of rain is 0% today. You have gone out without an umbrella. Now rain is falling.
Scenes of “It is the time”
Joy: A TV program that you want to watch is going to start. You call your family.
Sadness: You have come to an airport to send off your friend who is going abroad to study. Now it is time to depart.
Anger: It is the closing time of your shop, but a customer will not go home.
In this experiment, we adopted two-handed Finger Braille (the Perkins Brailler type, which is the most popular type of Finger Braille) and one-handed (right hand) Finger Braille.
In the two-handed Finger Braille experiment, the variables compose a 4 × 5 within-subjects design of expressed emotion (neutral, joy, sadness, anger) and position of characters (inner clause, end of clause, end of sentence, voiced sound, double consonant). In the one-handed Finger Braille experiment, the variables compose a 4 × 6 within-subjects design of expressed emotion (neutral, joy, sadness, anger) and position of characters (inner character, inner clause, end of clause, end of sentence, voiced sound, double consonant). The dependent variables are the duration of code, duration of pause, and finger load. The inner character is the dotting of DIP joints if the character includes the dotting of both DIP and PIP joints.
In our previous work [
The experimental flow is shown in Figure
Flow of Experiment I.
In one experimental session, the subjects dotted Finger Braille on the fingers of the tester eight times in the order of “Rain” with neutral, “Time” with neutral, “Rain” with joy, “Time” with joy, “Rain” with sadness, “Time” with sadness, “Rain” with anger, and “Time” with anger. The dialogue and emotion associated with the dot was displayed on the LCD of a notebook PC that was placed in front of the subject. The subject did not hear and transfer the emotional speech of the dialogues but expressed the impressions of emotions using Finger Braille. The subject did not alter the dialogue in any way and only expressed emotions by changing the dotting speed and applied pressure. To eliminate the influence of the previous dotted dialogue with emotion, the subject dotted at intervals (1-2 minutes) and confirmed the next scene of dialogue with emotion.
As is customary in communication using Finger Braille, the subject and tester sat side by side during the two-handed Finger Braille experiment and face to face during the one-handed Finger Braille experiment. The hands of the subject and tester were in constant contact during the two-handed Finger Braille experiment.
The tester put his fingers on a pressure sensor sheet (Tactile Sensor System, Nitta) which measured the change of pressure as a result of dotting. The sampling frequency was 137 Hz, the measurement range was 15–150 kPa, the sensibility was 527 Pa, the size of the sensor sheet was 84 × 84 mm, and the sensor matrix was 44 × 44. The sensor sheet was segmented into three blocks (width: 28 mm) and the tester put his index, middle, and ring fingers into the blocks.
The load of each finger was determined by the change in pressure applied to the three blocks, and the start, end, and maximum load of dotting were measured. The duration and finger load for each dotting were also calculated. Sample data from the one-handed Finger Braille experiment are shown in Figure
Calculation of durations and finger loads (one-handed Finger Braille).
The duration of the code was defined as the time from the start of dotting to the end of dotting. The duration of the pause was defined as the duration from the completion of dotting to the start of the next dotting sequence. The finger load was defined as the difference between the maximum load and the mean load between the start and end of dotting. If multiple fingers were dotted at once, the finger load was defined as the mean of the finger loads for the dotted fingers.
The mean of the duration of the code as a function of the emotions and position of characters is shown in Figure
A 4 × 5 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,580) = 119.3,
A test of the simple main effect of emotion revealed a significant effect of emotion in all the positions of characters (
A test of the simple main effect of the position of characters revealed a significant effect of the position of characters in all emotions (
Mean of duration of code as a function of emotions and position of characters for two-handed Finger Braille (error bars represent standard deviations).
The mean of the duration of the pause as a function of the emotions and position of characters is shown in Figure
A 4 × 4 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,504) = 5.615,
Mean of duration of pause as a function of emotions and position of characters for two-handed Finger Braille (error bars represent standard deviations).
The mean of the finger load as a function of the emotions and position of characters is shown in Figure
A 4 × 5 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,580) = 344.6,
A test of the simple main effect of emotion revealed a significant effect of emotion in all positions of characters (
A test of the simple main effect of position of characters revealed a significant effect of position of characters in all emotions (
Mean of finger load as a function of emotions and position of characters for two-handed Finger Braille (error bars represent standard deviations).
The mean of the duration of the code as a function of the emotions and position of characters is shown in Figure
A 4 × 6 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,1013) = 180.3,
A test of the simple main effect of emotion revealed a significant effect of emotion in all positions of characters (
A test of the simple main effect of the position of characters revealed a significant effect of the position of characters in all emotions (
Mean of duration of code as a function of emotions and position of characters for one-handed Finger Braille (error bars represent standard deviations).
The mean of the duration of the pause as a function of the emotions and position of characters is shown in Figure
A 4 × 5 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,935) = 4.009,
Mean of duration of pause as a function of emotions and position of characters for one-handed Finger Braille (error bars represent standard deviations).
The mean of the finger load as a function of the emotions and position of characters is shown in Figure
A 4 × 6 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,1013) = 261.4,
A test of the simple main effect of emotion revealed a significant effect of emotion in all positions of characters (
A test of the simple main effect of the position of characters revealed a significant effect of the position of characters in all emotions (
Mean of finger load as a function of emotions and position of characters for one-handed Finger Braille (error bars represent standard deviations).
After the experiment, the subjects were asked how they expressed the emotions, and their responses are presented in Table
Responses of interpreters to questionnaire on expressing emotions.
Emotion | Subject | Answer |
---|---|---|
Neutral | A | Conveying the meaning exactly to someone in view |
B | Imaging synthesized speech | |
Joy | A | Rhythmically |
B | Airily | |
Sadness | A | Uninterrupted dotting, like wrapping someone’s hand |
B | Negatively speaking | |
Anger | A | Like shoving someone away |
B | Like dashing words over someone |
Subjects also responded that it was more difficult to express emotions using one-handed Finger Braille with people sitting face to face than using two-handed Finger Braille sitting side by side, because the skin contact of the hands and the distance of each person were considered important for emotional communication.
To analyze the contribution of the variables to emotional expression, a discriminant analysis was conducted on data from both the two-handed and one-handed Finger Braille experiments (
Standardized coefficients of canonical discriminant functions and contribution ratios were calculated for both the two-handed and one-handed Finger Braille experiments. The results are presented in Tables
Standardized coefficients of canonical discriminant functions and contribution ratios (two-handed Finger Braille).
Discriminant variable | Function 1 | Function 2 | Function 3 |
---|---|---|---|
Duration of code | −0.005 | 0.990 | −0.272 |
Duration of pause | 0.005 | 0.100 | 1.015 |
Finger load | 1.000 | 0.164 | 0.059 |
Contribution ratio | 84.3% | 15.6% | 0.1% |
Significance probability |
Standardized coefficients of canonical discriminant functions and contribution ratios (one-handed Finger Braille).
Discriminant variable | Function 1 | Function 2 | Function 3 |
---|---|---|---|
Duration of code | −0.560 | 0.820 | −0.161 |
Duration of pause | 0.053 | 0.248 | 0.993 |
Finger load | 0.903 | 0.494 | 0.058 |
Contribution ratio | 76.0% | 23.9% | 0.1% |
Significance probability |
According to the standardized coefficients, discriminant function 1 represented the finger load, discriminant function 2 represented the duration of code, and discriminant function 3 represented the duration of pause. According to the contribution ratios, the finger load was the variable contributing the most to emotional expression, followed by the duration of code. The duration of pause did not contribute to emotional expression.
The discriminant ratios of emotion for two-handed and one-handed Finger Braille are presented in Tables
Discriminant ratios of emotion (two-handed Finger Braille).
Expressed emotion | Discriminated emotion | Discriminant ratio | |||
---|---|---|---|---|---|
Neutral | Joy | Sadness | Anger | ||
Neutral | 62 | 25 | 43 | 0 | 47.7% |
Joy | 14 | 113 | 3 | 0 | 86.9% |
Sadness | 38 | 16 | 76 | 0 | 58.5% |
Anger | 14 | 9 | 2 | 105 | 80.8% |
Total | 128 | 163 | 124 | 105 | 68.5% |
Discriminant ratios of emotion (one-handed Finger Braille).
Expressed emotion | Discriminated emotion | Discriminant ratio | |||
---|---|---|---|---|---|
Neutral | Joy | Sadness | Anger | ||
Neutral | 159 | 38 | 40 | 3 | 66.3% |
Joy | 20 | 201 | 2 | 13 | 85.2% |
Sadness | 83 | 5 | 152 | 0 | 66.3% |
Anger | 13 | 55 | 4 | 167 | 69.9% |
Total | 275 | 299 | 198 | 183 | 71.1% |
As indicated by the results of ANOVA (2.2.2–2.2.3), the features of emotional expression were as follows: (1) the duration of the code of joy was significantly shorter than that of the other emotions; (2) the duration of the code of sadness was significantly longer than that of the other emotions; (3) the finger load of anger was significantly larger than that of the other emotions; (4) the finger load of joy was significantly larger than that of sadness and neutral; (5) the duration of the code of anger was significantly shorter than that of sadness and neutral.
Next we discuss the features of emotional expression by the other communication media. In our previous work [
Laukka analyzed the emotional expression of speech [
Bresin and Friberg analyzed the emotional expression of music [
Dahl and Friberg analyzed the emotional expression of a musician’s body movement in performances on the marimba [
Clynes and Panksepp proposed a “sentic form” to express the fundamental human emotions. Sentic form can be measured by the patterns of finger pressure [
The duration of code in the present study (Experiment I) was similar to the speech rate, tempo of music, speed of movement, and duration of finger pressure. The finger load in Experiment I was similar to the voice intensity, sound level of music, largeness of movement, and strength of finger pressure. Therefore, the features of emotional expression using Finger Braille are similar to the features of the emotional expressions of speech, music, body movement, and finger pressure.
As indicated by the results of ANOVA (2.2.2–2.2.3), the subjects dotted dialogues with prosody (intonation) in addition to emotional expression. The followings were the features of prosody: (1) the duration of the code of the end of the clause was significantly longer; (2) the duration of the pause of the end of the clause was significantly longer; (3) the duration of the code of the end of the sentence was significantly longer; (4) the finger load of the end of the sentence was significantly larger; (5) the duration of the code of the double consonant was significantly shorter; (6) the duration of the pause before the double consonant was significantly shorter; (7) the duration of the code of the voiced sound was significantly shorter; (8) the finger load of the voiced sound was significantly larger.
Dousaka et al. and Fujimori et al. analyzed the features of prosody of Finger Braille [
Comparison of durations of code, pause, and dotting between the four emotions in our study and in previous studies [
Miyagi et al. analyzed the changes of prosody in emphasized clauses [
To confirm the features of emotional expression using Finger Braille, an additional experiment of emotional expression by the Finger Braille interpreter was conducted.
The subject was Subject A in Experiment I. The additional experiment was conducted one year and eight months after Experiment I. The subject gave informed consent after hearing a description of the study.
Two new dialogues were used. “Let’s hurry up” or
The expressed emotions were neutral, joy, sadness, and anger, which are identical to those used in Experiment I. We set six scenes (two dialogues with joy, sadness, and anger) for the subject. The scenes were as follows.
Scenes of “Let’s hurry up”
Joy: You are planning to go out with your close friend.
Sadness: Your family member has to leave home because he has a new job. Now you help pack his bag.
Anger: Your work is late but your colleague has not started to work.
Scenes of “Is it really?”
Joy: You have received news that you passed a difficult examination.
Sadness: You have received news that your close friend is in the hospital.
Anger: Suddenly, you have been wrongly accused.
In this experiment, we adopted two-handed Finger Braille (the Perkins Brailler type) and one-handed (right hand) Finger Braille, as in Experiment I.
In the two-handed Finger Braille experiment, the variables compose a 4 × 4 within-subjects design: expressed emotion (neutral, joy, sadness, anger) and position of characters (inner clause, end of clause, end of sentence, voiced sound). In the one-handed Finger Braille experiment, the variables compose a 4 × 5 within-subjects design: expressed emotion (neutral, joy, sadness, anger) and position of characters (inner character, inner clause, end of clause, end of sentence, voiced sound). The diphthong was categorized as a voiced sound. The dependent variables were duration of code, duration of pause, and finger load.
The experimental flow is shown in Figure
Flow of addition to Experiment I.
In the experimental session, the subject dotted Finger Braille on the fingers of the tester eight times in the order of “Hurry” with neutral, “Really” with neutral, “Hurry” with joy, “Really” with joy, “Hurry” with sadness, “Really” with sadness, “Hurry” with anger, and “Really” with anger.
The other procedure and conditions were identical to those used in Experiment I.
We calculated the duration of code, duration of pause, and finger load for each dotting by the change of pressure, as in Experiment I (see Figure
The mean of the duration of the code as a function of the emotions and position of characters is shown in Figure
A 4 × 4 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,283) = 152.8,
A test of the simple main effect of emotion revealed a significant effect of emotion in all positions of characters (
A test of the simple main effect of the position of characters revealed a significant effect of the position of characters in all emotions (
Mean of duration of code as a function of emotions and position of characters for two-handed Finger Braille (error bars represent standard deviations).
The mean of the duration of the pause as a function of the emotions and position of characters is shown in Figure
A 4 × 3 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,247) = 5.353,
A test of the simple main effect of emotion revealed a significant effect of emotion in the end of the clause and voiced sound (
A test of the simple main effect of the position of characters revealed a significant effect of the position of characters in all emotions (
Mean of duration of pause as a function of emotions and position of characters for two-handed Finger Braille (error bars represent standard deviations).
The mean of the finger load as a function of the emotions and position of characters is shown in Figure
A 4 × 4 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,283) = 229.5,
A test of the simple main effect of emotion revealed a significant effect of emotion in all positions of characters (
A test of the simple main effect of the position of characters revealed a significant effect of the position of characters in all emotions (
Mean of finger load as a function of emotions and position of characters for two-handed Finger Braille (error bars represent standard deviations).
The mean of the duration of the code as a function of the emotions and position of characters is shown in Figure
A 4 × 5 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,579) = 109.9,
A test of the simple main effect of emotion revealed a significant effect of emotion in all positions of characters (
A test of the simple main effect of the position of characters revealed a significant effect of position of characters in all emotions (
Mean of duration of code as a function of emotions and position of characters for one-handed Finger Braille (error bars represent standard deviations).
The mean of the duration of the pause as a function of the emotions and position of characters is shown in Figure
A 4 × 4 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,541) = 4.390,
Mean of duration of pause as a function of emotions and position of characters for one-handed Finger Braille (error bars represent standard deviations).
The mean of the finger load as a function of the emotions and position of characters is shown in Figure
A 4 × 5 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,580) = 106.5,
A test of the simple main effect of emotion revealed a significant effect of emotion in all positions of characters (
A test of the simple main effect of the position of characters revealed a significant effect of the position of characters in all emotions (
Mean of finger load as a function of emotions and position of characters for one-handed Finger Braille (error bars represent standard deviations).
As indicated by the results of ANOVA (3.2.1–3.2.2), the features of emotional expression were as follows: (1) the duration of the code of joy was significantly shorter than that of the other emotions; (2) the duration of the code of sadness was significantly longer than that of the other emotions; (3) the finger load of anger was significantly larger than that of the other emotions; (4) the finger load of joy was significantly larger than that of sadness and neutral. These features were very similar to the results of Experiment I.
As indicated by the results of ANOVA (3.2.1–3.2.2), the followings were the features of prosody: (1) the duration of the code of the end of the clause was significantly longer; (2) the duration of the pause of the end of the clause was significantly longer; (3) the duration of the code of the end of the sentence was significantly longer; (4) the finger load of the end of the sentence was significantly larger; (5) the duration of the code of the double consonant was significantly shorter; (6) the duration of the code of the voiced sound was significantly shorter; (7) the finger load of the voiced sound was significantly larger. These features were also very similar to the results of Experiment I.
In addition, there was a tendency for the finger load of the additional experiment to be larger than the finger load of Experiment I. This tendency could be a daily variance of the dotting strength of Finger Braille.
Therefore, we can confirm that the features of emotional expression and prosody are independent of the dotted dialogues.
To analyze the effectiveness of emotional expression and emotional communication between people who are unskilled in Finger Braille, we conducted another experiment of emotional communication using Finger Braille (Experiment II) [
The subjects were twelve non-disabled college students (ages: 21–26 years old) who were not unskilled in Finger Braille. All subjects were right-handed. All subjects gave informed consent after hearing a description of the study.
The dialogues were “Rain has fallen (
Expressed emotions were neutral, joy, sadness, and anger, which were identical to those used in Experiment I. We set six scenes (two dialogues with joy, sadness, and anger) for subjects, which were also identical to the ones used in Experiment I.
In this experiment, we adopted one-handed Finger Braille (right hand), because one-handed Finger Braille is easier than two-handed Finger Braille for unskilled individuals.
The variables compose a 4 × 6 within-subjects design: expressed emotion (neutral, joy, sadness, anger) and position of characters (inner character, inner clause, end of clause, end of sentence, voiced sound, double consonant). The dependent variables were duration of code, duration of pause, and finger load.
The experimental flow is shown in Figure
Flow of Experiment II.
The sender and receiver sat side by side during the experiment to promote the emotional expression by the sender. To simulate deafblindness, the receiver wore earplugs, headphones playing white noise, and a night shade. The sender dotted one-handed Finger Braille (right-hand) on the fingers of the receiver eight times per session in a randomized order of dialogues with emotions. To eliminate the influence of the previous dotted dialogue with emotion, the subject dotted at intervals (1-2 minutes) and confirmed the next scene of dialogue with emotion. The sender observed a sheet of dot patterns of the experimental dialogues, and the receiver responded to the dialogue and emotion after the sender dotted one time. Unlike the interpreters in Experiment I, the subjects were not taught the features of emotional expression and prosody. The subjects expressed emotions freely by changing the strength and speed of dotting.
The receiver put his fingers on the pressure sensor sheet (Grip Force Sensor System, Nitta), which measured the change of pressure as a result of dotting. The sampling frequency was 150 Hz, the measurement range was 20–200 kPa, and the sensibility was 781 Pa. The sensor sheet was divided into 20 blocks, and the size of each block was 16 × 16 mm. The sensor matrix of the block was 4 × 4. The receiver put his index, middle, and ring fingers on the three blocks.
We calculated the duration of code, duration of pause, and finger load for each dotting by the change of pressure, as in Experiment I (see Figure
The mean of the duration of the code as a function of the emotions and position of characters is shown in Figure
A 4 × 6 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,13689) = 675.6,
A test of the simple main effect of emotion revealed a significant effect of emotion in all positions of characters (
A test of the simple main effect of the position of characters revealed a significant effect of the position of characters in all emotions (
Mean of duration of code as a function of emotions and position of characters (error bars represent standard deviations).
The mean of the duration of the pause as a function of the emotions and position of characters is shown in Figure
A 4 × 6 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,12607) = 43.926,
Mean of duration of pause as a function of emotions and position of characters (error bars represent standard deviations).
The mean of the finger load as a function of the emotions and the position of characters is shown in Figure
A 4 × 6 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,13689) = 728.9,
A test of the simple main effect of emotion revealed a significant effect of emotion in all positions of characters (
A test of the simple main effect of the position of characters revealed a significant effect of the position of characters in all emotions (
Mean of finger load as a function of emotions and position of characters (error bars represent standard deviations).
In this experiment, the receiver answered the recognized dialogue and emotion dotted by the sender after each dotting. As a result, the accuracy of recognition of the dialogues was a total of 96.5%. The relationship between expressed emotions by the senders, recognized emotions by the receivers, and their coincidence ratios is presented in Table
Coincidence ratios of emotion.
Expressed emotion | Recognized emotion | Coincidence ratio | |||
---|---|---|---|---|---|
Neutral | Joy | Sadness | Anger | ||
Neutral | 172 | 31 | 44 | 17 | 65.2% |
Joy | 38 | 191 | 8 | 27 | 72.3% |
Sadness | 43 | 11 | 207 | 3 | 78.4% |
Anger | 21 | 14 | 6 | 223 | 84.5% |
Total | 274 | 247 | 265 | 270 | 75.1% |
Coincidence ratio of emotion as a function of emotions and subject (sender).
The change of coincidence ratios as a function of the emotions and order of dotted dialogue in a session are shown in Figure
Change of coincidence ratio as a function of emotions and order of dotted dialogue in session.
After the experiment, the subjects were asked how they expressed these emotions, and their responses are listed in Table
Responses of unskilled senders to questionnaire on expressing emotions.
Emotion | Subject | Answer |
---|---|---|
Neutral | 1 | As the standard of the other emotions |
2 | Constant tempo, medium duration | |
3 | Mechanically, medium strength, medium speed | |
4 | Constant rhythm | |
5 | Constant speed, medium strength | |
6 | Somewhat weaker than angry | |
7 | Normally | |
8 | Not one of the other emotions | |
9 | Characterless, only considered dotting a sentence | |
10 | Flat rhythm, medium strength, somewhat long | |
11 | Somewhat slowly, medium strength | |
12 | Carefully, slowly | |
Joy | 1 | Quick tempo, airily |
2 | Rhythmically, short duration, airily | |
3 | Rhythmically, medium strength | |
4 | Rhythmically, quick tempo | |
5 | Quickly, airily | |
6 | Quickly | |
7 | Rhythmically | |
8 | Weakly, rhythmically | |
9 | Short duration, somewhat quickly | |
10 | Quick rhythm, medium strength | |
11 | Quicker tempo than neutral, like flicking fingers | |
12 | Airily, quickly | |
Sadness | 1 | Considered to avoid recognition as neutral |
2 | Slow tempo, long duration, softly | |
3 | Weakly, slowly | |
4 | Slowly, airily | |
5 | Slowly, weakly | |
6 | Weakly, slowly | |
7 | Slowly, airily | |
8 | Weakly, slowly | |
9 | Long duration, weakly, slowly | |
10 | Slow rhythm, quite weakly | |
11 | Quite weakly, airily | |
12 | Softly | |
Anger | 1 | Strongly, heavily |
2 | Somewhat quickly, medium duration, strongly | |
3 | Strongly, quickly | |
4 | Strongly, slowly | |
5 | Strongly | |
6 | Strongly | |
7 | Strongly, somewhat slowly | |
8 | Strongly, like pushing fingers | |
9 | Same duration and speed as neutral, strongly | |
10 | Rhythm between joy and sadness, quite strongly | |
11 | Quite strongly, like pushing fingers | |
12 | Claw-like fingers |
Six subjects answered that they dotted Finger Braille while sometimes glancing at the sheet of dot patterns. The other six subjects answered that they dotted Finger Braille while continually observing the sheet of dot patterns. As a degree of emotional expression, one subject answered “well expressed”; six subjects answered “almost expressed”; five subjects answered “somewhat expressed.”
All subjects recognized the dotted dialogues by the first two characters or the last two characters. Some subjects also answered that it was hard to recognize the expressed emotions if the first emotion in the session was neutral, and two or three dialogues in the session were necessary to recognize the neutral level of the sender.
There were similar features of the emotional expression between the interpreters in Experiment I and the unskilled subjects in Experiment II: the duration of joy was significantly shorter than that of the other emotions; the finger load of anger was significantly larger than that of the other emotions. As different features, the duration of the code of anger by the interpreters was slightly shorter, but the duration of the code of anger by the unskilled subjects was significantly longer. In a previous study, we found that the unskilled subjects faced difficulties in dotting strongly in a short time [
Since the unskilled subjects were not taught the prosody of Finger Braille, they could not express its metrical structure. The unskilled subjects dotted Finger Braille by glancing at or continually observing the sheet of dot patterns, and then many of them paused at the end of a clause and checked the patterns.
According to the questionnaire results, the impressions of emotional expression by the unskilled subjects were similar to those by the interpreters.
To evaluate emotional expression by the subjects in Experiment II, twelve discriminant analyses on the data of each subject were conducted (
The total discriminant ratios of emotion are listed in Table
Total discriminant ratios of emotion.
Expressed emotion | Discriminated emotion | Discriminant ratio | |||
---|---|---|---|---|---|
Neutral | Joy | Sadness | Anger | ||
Neutral | 1,356 | 601 | 763 | 436 | 43.0% |
Joy | 174 | 2,551 | 220 | 207 | 80.9% |
Sadness | 678 | 571 | 1,727 | 182 | 54.7% |
Anger | 500 | 368 | 278 | 2,015 | 63.7% |
Total | 2,708 | 4,091 | 2,988 | 2,840 | 60.6% |
Discriminant ratio as a function of emotions and subject (sender).
The represented variables for discriminant functions, contribution ratios, and significance probability are listed in Table
Represented variables for discriminant functions, contribution ratios, and significance probability (
Group | Subject | Discriminant function 1 | Discriminant function 2 | Discriminant function 3 |
---|---|---|---|---|
Duration of code | Finger load | Duration of pause | ||
CLP | 2 | 53.9% | 44.6% | 1.5% |
4 | 84.3% | 14.7% | 1.0% | |
8 | 62.1% | 37.9% | 0% | |
9 | 75.4% | 21.2% | 3.3% | |
11 | 90.8% | 9.1% | 0.1% | |
Finger load | Duration of code | Duration of pause | ||
LCP | 1 | 78.2% | 21.8% | 0% |
3 | 75.7% | 24.1% | 0.1% | |
5 | 64.2% | 35.6% | 0.2% | |
7 | 78.3% | 21.5% | 0.1% | |
10 | 81.8% | 18.2% | 0% | |
12 | 65.3% | 29.2% | 5.5% | |
Duration of code | Duration of pause | Finger load | ||
CPL | 6 | 83.3% | 16.4% | 0.3% |
The mean of the discriminant ratio of group CLP was 66.5% (S.D. = 6.0), and the mean of the discriminant ratio of group LCP and group CPL was 56.4% (S.D. = 7.3). A
The relationship between the discriminant ratio and coincidence ratio of each subject (sender) is shown in Figure
Relationship between discriminant ratio and coincidence ratio of each subject.
As shown in Figure
Now, we discuss the features of emotional communication by other communication media.
Shirasawa et al., Iida et al., and Kinjou et al. analyzed emotional communication by speech [
In Shirasawa’s experiment [
In Iida’s experiment [
In Kinjou’s experiment [
Yoshikawa and Morisaki analyzed emotional communication by facial expression [
Ohgushi and Hattori analyzed emotional communication by vocal performance [
The results of the present study showed that the coincidence ratios of Experiment II were equal to or more than the coincidence ratios of these previous studies. Thus, it was considered that the unskilled people can express and communicate emotions using Finger Braille, which is a new tactile communication medium.
However, some subjects expressed emotions poorly, and it was difficult for the receivers to recognize the dotted emotions. Thus, some assistance in emotional communication was needed for the non-disabled people. One of the methods to teach emotional expression to unskilled people is to teach the impression of emotional expression by applying the similarities of the impression of emotional expression between the interpreters and the unskilled people (e.g., “rhythmically” for joy, “weakly and slowly” for sadness, “strongly and quickly” for anger). Another method of teaching emotional expression is to introduce a dot pattern with some symbols directly representing dotting speed and strength (e.g., a dot pattern with a long or short array for dotting speed, and a dot pattern with a colored background for dotting strength). Some example patterns for teaching emotional expression to people unskilled in Finger Braille are shown in Figure
Examples of patterns for teaching unskilled people how to express emotions using Finger Braille.
In the present study, emotional expression by interpreters using Finger Braille (Experiment I) was first examined, and the features of emotional expression using Finger Braille were analyzed. The resulting features were as follows: (1) the duration of the code of joy was significantly shorter than that of the other emotions (neutral, anger, sadness); (2) the duration of the code of sadness was significantly longer than that of the other emotions; (3) the finger load of anger was significantly larger than that of the other emotions; (4) the finger load of joy was significantly larger than that of sadness and neutral; (5) the duration of the code of anger was significantly shorter than that of sadness and neutral. As shown by the results of the additional experiment, these features of emotional expression are independent of the dotted dialogue.
Next, emotional communication by people unskilled in Finger Braille (Experiment II) was examined, and the effectiveness of emotional expression and emotional communication between these individuals was analyzed. The results indicate that the features and the impression of emotional expression by the unskilled people were very similar to those by the interpreters, and the coincidence ratio of emotional communication was 75.1%. Therefore, it was confirmed that unskilled people can express and communicate emotions using Finger Braille.
The followings are plans for future studies: (1) expansion of the emotions expressed and the number of subjects used in Finger Braille experiments, (2) development of methods to teach non-disabled people emotional expression using Finger Braille, (3) analysis of the relationship between emotional expression using Finger Braille and speech, and (4) development of an emotion recognition algorithm.
The authors greatly thank Ms. Satoko Mishina and Ms. Megumi Fukuma (interpreters of Finger Braille) for their support. This study was supported by the Japan Society for the Promotion of Science under a Grant-in-Aid for Scientific Research (no. 21500522) and the Ministry of Education, Culture, Sports, Science and Technology of Japan under a Grant-in-Aid for Scientific Research (no. 16700430). This study was partly supported by Kanagawa Academy of Science and Technology (KAST) under a research grant.