International Journal of Computer Games Technology
Volume 2019 (2019), Article ID 6981698, 12 pages
https://doi.org/10.1155/2019/6981698

Research Article

Hall of Heroes: A Digital Game for Social Skills Training with Young Adolescents

Melissa E. DeRosier and James M. Thomas

3C Institute, 4364 South Alston Avenue, Suite 300, Durham NC 27713, USA

Correspondence should be addressed to Melissa E. DeRosier; derosier@3cisd.com

Received 12 July 2018; Revised 26 November 2018; Accepted 4 December 2018; Published 1 January 2019

Academic Editor: Michael J. Katchabaw

Copyright © 2019 Melissa E. DeRosier and James M. Thomas. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

Traditional social skills training (SST) programs are delivered in person and suffer from significant time, financial, and opportunity barriers that limit their reach and potential benefits for youth. This paper describes the design and preliminary evaluation of Hall of Heroes, a digital game that presents SST through an engaging superhero-themed virtual story world. Participants were randomly assigned to complete the digital game (n = 15) or to a waitlist control condition (n = 14) and were compared on parent-report measures of social emotional functioning. Youth who completed Hall of Heroes significantly improved in their abilities to relate to others (both peers and family members) as well as to accept affection and express emotions with others, compared to youth who did not complete the SST intervention. Further, youth in the treatment condition showed a significantly greater decline in feelings of anxiety, depression, and hopelessness than did youth in the control condition. Both parents and youth reported high levels of engagement in and acceptability of the Hall of Heroes. This study adds to the research literature, supporting the potential of a game-based SST platform for effectively helping youth develop prosocial social problem-solving skills.

1. Introduction

1.1. Social Relationships in Early Adolescence

The early adolescent period is a turbulent and stressful one for youth who must adapt and adjust to a myriad of concurrent life changes. At the same time as significant physical and cognitive growth is occurring, youth move from the smaller, more protective elementary school setting, with smaller classrooms and strong teacher support, to the larger, more impersonal middle school environment. In late elementary and middle school, academic demands increase considerably with more difficult assignments, higher expectations for independence, and more stringent grading [1, 2]. Concomitant with escalating academic demands, the social environment shifts from smaller, more protective settings to much larger, varied, and competitive social groups [3–6].

The turbulence of these academic, social, and developmental shifts, combined with a decline in adult supervision, leads to a period of heightened vulnerability. For many youth, the early adolescent period is associated with loss of self-esteem, increased anxiety and depression, and drops in academic performance [3, 7]. And, adolescents with lower social skills (e.g., poor emotion regulation, difficulty with communication) and poorer peer relations are at greater risk of academic failure, behavioral problems, and emotional difficulties [8–11]. In contrast, positive social support and prosocial behaviors serve as protective factors, helping youth adjust more quickly to changing demands and fostering greater school engagement and academic success [2, 3, 8].

1.2. Social Skills Training

Social skills training (SST) interventions have repeatedly been shown to significantly improve social skills, behavior, and interpersonal relationships [12–15]. Effective SST can serve a protective function, as youth learn and practice those social skills that increase prosocial and/or inhibit maladaptive behaviors [16, 17]. For example, cooperation fosters companionship—a key function of peer relations—while impulsive and domineering behaviors undermine the development and maintenance of positive social relationships. As Peterson et al. [7] explain, “Improvements in social skills and problem-solving abilities allow the preadolescent to build a supportive network and engage in appropriate social activities that may ultimately contribute to reduced school drop-out rates as well as other indices of improved adaptive functioning” (pp. 289). Effective SST programs that improve social problem solving and social skills are essential for preparing youth for the transition to middle school [8, 10].

Unfortunately, SST is traditionally delivered in person by trained clinicians or school staff which presents significant logistical barriers, including a lack of trained providers, scheduling difficulties, and significant time commitments, which too often prevent youth from participating in and benefiting from SST [14, 15]. Poor implementation fidelity by providers (e.g., nonadherence to the in-person treatment protocol) has also been shown to undermine outcomes for youth [7, 18]. And, lastly, SST is typically delivered in a group format with one provider to many youths. Research suggests tailoring interventions to the individual may be necessary to achieve intended gains in social and academic outcomes [19, 20].

1.3. Games for Impact

To help overcome the limitations of in-person SST, a growing number of intervention developers are employing emerging technologies [21]. Game-based platforms offer a cost-effective means to deliver program content in an accessible and engaging way, eliminate travel and provider staffing and training requirements, and standardize program delivery on a broad scale [19] (Merry et al., 2012). When digital games are dynamic and individualized, the user benefits from increased practice opportunities as well as a more tailored and personalized learning experience [19, 22–24]. Further, the engaging and interactive nature of games, and the relative anonymity of digital learning environments, means youth are often motivated to more fully and openly participate in game-based SST compared to traditional in-person SST [25, 26].

There is growing support for the effectiveness of digital games for social, emotional, and behavioral development [21, 27–30]. Digital platforms have been shown to increase comprehension and recall of presented material over conventional instructional methods and produce greater generalization of acquired skills to real-life behavior (Clark et al., 2016) [31, 32]. Further, there is increasing evidence that digital environments are especially effective at targeting higher-order thinking skills of analysis, synthesis, and evaluation, which form the basis of social problem solving [28, 33].

1.4. The Development of Hall of Heroes

Drawing on the foundational principals of SST and digital game-based learning, 3C Institute developed Hall of Heroes, an online single-player point-and-click adventure game wherein the youth navigate a virtual school-like world, engaging with other characters to solve tailored social problem-solving tasks. As illustrated in Figures 1 and 2, the full Hall of Heroes software includes a personalized avatar builder, a gameplay training scene, six baseline assessment scenes, seven tutorial scenes, a final skill mastery application scene, six cinematic cut scenes, and seven knowledge monitoring quizzes. Each of the first six scenes serves as an introduction to and baseline measure of one of Hall of Heroes’ six core social skills: impulse control, communication, cooperation, social initiation, empathy, and emotion regulation. Each of the seven tutorial scenes provides dynamic instructional content focusing on two or three particular social problem-solving strategies for these core skills (see Table 1 for scene overviews and learning objectives). The skill concepts covered in these scenes are reinforced in the knowledge assessments where players are asked a series of application questions relevant to the skill nuances explored in the preceding scenes and provided with dynamic feedback based on their answers.

Table 1: Overview of Hall of Heroes six social emotional tutoring scenes with skill areas and learning objectives.

Scene
Overview
Skill Areas
Learning Objectives

The Replica
Players work with two other students to put together a time machine replica during science class.
Social Initiation
Cooperation
Impulse Control
Identify the correct group to initiate with and do so appropriately
Cooperate effectively, including negotiating about how to divide work
Utilize available resources and organizational skills to effectively manage the assigned task

PowerFlip
Players spend their free period attempting to join in with two different pairs of students playing a card game.
Social Initiation
Emotion Regulation
Determine where to stand when joining in
Identify the appropriate person to talk to without interrupting
Identify their own emotions after being rejected
Choose which group to re-join post rejection

Hero Simulation
Players work with a group of fellow graders to produce a hero training video.
Cooperation
Communication
Negotiate and compromise with another student to determine project roles
Coach other students in expressive nonverbal and verbal communication, focusing on tone of voice, facial expressions, and body language

Fire and Ice
Players hone key interpersonal skills during their Hero-Civilians Relations class by playing three mini-games in a virtual reality simulator.
Impulse Control
Empathy
Emotion Regulation
Demonstrate impulse control in a response inhibition simulation
Appropriately label dialog options based on their anticipated impact on another student
Navigate 3 “choose your own adventure” style activities which require stopping and thinking in highly emotional situations

The Escape
Students have just witnessed the evil villain Dr. Klepto steal the teachers’ superpowers. Players must manage the fallout and rally other students to save the school.
Emotion Regulation
Communication
Empathy
Regulate their own emotions with an activity (e.g., deep breathing and positive self-talk)
Use receptive communication skills to determine the emotional states of other students
Help those students to regulate their emotions using appropriate activities

Surveillance
Players watch security footage to figure out what is going on at Hall of Heroes, and then work with other students to clarify details and form a theory and timeline of what happened.
Empathy
Impulse Control
Observe and interpret other characters’ emotions in context
Balance attention to detail and time constraints in a high-stakes situation
Distinguish facts from assumptions

The Mission
Players and other students work together to formulate a plan to travel back in time to stop Dr. Klepto and save the school.
Impulse Control
Cooperation
Communication
Work with other students to make and implement an action plan, which includes identifying the best team members for each step
Cooperate with difficult others, using I statements and reflective listening to diffuse conflict

Figure 1: Hall of Heroes scene flow diagram.

(a) Personalized avatar builder

(b) Scene selector

(c) Knowledge monitoring assessment

(d) Cinematic cut scenes

Figure 2: Examples screenshots of Hall of Heroes game components.

Hall of Heroes was developed by a multidisciplinary team comprising clinical and developmental psychologists, SST intervention developers, computer programmers, artists and animators, and experts in computer games for learning. The development process followed best practices in educational game design, with thoughtful consideration for the unique social and developmental challenges of early adolescence [23, 24, 28, 32, 34]. These considerations led us to employ multiple gameplay styles, varying the settings and backgrounds for each scene, and to incorporate a compelling storyline through both playable scenes and cinematic cut scenes, resulting in a motivating and engaging game environment where all types of players can find something to attract them [26, 35–37]. Figure 3 presents a sample of the range of gameplay experiences in Hall of Heroes, which include simulations, drill and practice mini-games, opportunities for open exploration, dialog driven content, and action-based challenges. By incorporating direct instruction, varied practice exercises, dynamic feedback, and self-directed elements, the game is able to effectively target multiple modes of learning [23, 26, 34, 37].

(a) Simulation: players act as the director during a video shoot

(b) Dialog menu: players discuss where to sit during lunch

(c) Skills application: players demonstrate receptive communication in a notes-taking exercise

(d) Mini-game: players practice deep breathing techniques

(e) Practice exercise: players practice using a combination lock to open their locker

(f) Practice exercise: players choose where to stand when joining in

Figure 3: Example screenshots of Hall of Heroes gameplay.

Situating the game in a school-like setting allows for the inclusion of tasks that directly address the routine aspects of middle school, such as changing classes and using a locker, which repeatedly come up as areas of pretransition concern and anxiety for youth [7, 38]. In addition, the school setting allowed us to simulate challenging social situations unique to a school environment, such as choosing a table for lunch and understanding when and how to engage a teacher for help in a bullying situation. Incorporating these common transition experiences into a superhero world serves to balance fantasy and realism, maintaining engagement and allowing for deeper learning and skills transfer [32].

To further develop this rich narrative world, Hall of Heroes is populated with a large and diverse cast of nonplayable characters (NPCs) which serve narrative and pedagogical roles that mirror traditional instructional roles: peer, teacher, mentor, learning companion, etc. Experts in the field of educational game design agree that the inclusion of interactive embodied social pedagogical agents enhances student motivation and engagement and improves learning outcomes including retention and skills transfer (e.g., [39–41]). NPCs as embodied pedagogical agents support the social cognitive aspects of learning, serving as an interactive bridge to help players understand complex concepts [39, 42]. Rather than simply presenting information, the NPCs in Hall of Heroes engage and motivate players through direct experiences with the game world to actively participate in the construction of social problem-solving knowledge, for example, by guiding players to look for clues in school security footage in order to check their assumptions about a situation. NPCs also serve to model and guide players through difficult cognitive processes, for example, by facilitating positive self-talk through conversation and branching dialog menus.

In the current study, we conducted an initial investigation of the efficacy and acceptability of the Hall of Heroes game for SST using a randomized-control design. We expected youth who interacted with Hall of Heroes would demonstrate significantly greater improvements in social, emotional, and behavioral outcomes compared to youth in the waitlist control condition who did not complete the game.

2. Methods

2.1. Participants

Participants were recruited via postings on national parenting, game research, and school listservs and social media sites. Interested parents completed an online survey to determine eligibility. To be eligible to participate, youth must have (1) been attending the 5th or 6th grades; (2) received instruction in a regular education classroom for at least 40% of the school day; (3) been English language proficient; and (4) had access to a computer with Internet use at home.

Eligible participants (N = 32) were randomly assigned to either the treatment (TX) or waitlist control (WLC). Nine percent of participants were attrited over the course of the study (i.e., failed to complete post-data collection tasks), resulting in a final sample of 29 youths. Chi-squared tests revealed no significant differences between participants who completed the study and those who did not based on age, race, gender, or study condition. The final sample of youth ranged in age from nine to thirteen years (M = 11.35, SD = 0.88), with 76% male and 31% representing a racial or ethnic minority group. There were no significant demographic differences across the two conditions.

2.2. Measures

Positive Behavioral and Emotional Skills. Youth behavioral and emotional strengths were measured using The Behavioral and Emotional Rating Scale, Second Edition (BERS-2 [43]), a 58-item strength-based assessment that measures youth positive emotions, behaviors, and interpersonal skills. Parents reported how many statements, such as “acknowledges painful feelings,” “considers consequences of own behavior,” and “completes homework regularly,” were like their child over the past 30 days on a 4-point scale ranging from 0 (not at all) to 3 (very much). The BERS-2 composite Strength Index was calculated by summing across all items and converting this raw score to a standard score (M=100, SD=15). Further, the composite score was broken down into its five subscales: interpersonal strength (strengths in how the child relates to others), affective strength (strengths in how the accepts affection and expresses emotions), family involvement (strengths in the child’s relationships with family members), intrapersonal strength (child’s internal emotional strengths, outlook on one’s own competencies and accomplishments), and school functioning (strengths in completing school-related tasks). Raw scores for each subscale were computed by summing scores for all relevant items and converting them to standard scores (M=10, SD=3). Higher scores reflect greater behavioral and emotional strength in that area. The BERS-2 Strength Index demonstrated good reliability (Cronbach’s α = .97) and prior factor analyses have established the measure’s content, criterion, and construct validity [43].

Psychosocial Distress. Youth psychosocial distress was measured using the 64-item parent-report Youth Outcomes Questionnaire, Second Edition (Y-OQ [44]). Parents were asked to think about their child over the past 7 days and report how true statements were for their child on a 5-point scale from 0 (never or almost never) to 4 (almost always or always). The Y-OQ Total Distress score was computed by summing across all items and respective items were combined for six subscales: intrapersonal distress (18 items covering anxiety, depression, and hopelessness), somatic (8 items including headaches, dizziness, and stomachaches), interpersonal relations (10 items covering arguing, defiance, and communication problems), social problems (8 delinquent or aggressive behavior items), behavioral dysfunction (11 items covering organization, concentration, handling frustration, and ADHD-related symptoms), and critical items (9 items consisting of symptoms often found in youth receiving inpatient services, such as paranoid ideation, hallucinations, mania, and suicidal feelings). Higher scores on each scale indicate higher levels of psychosocial distress. The Y-OQ showed high reliability (Cronbach’s α = .97) and its publishers detail several studies establishing content, criterion, and construct validity with both clinical and nonclinical (community) samples [44].

Game Evaluation. Following completion, youth and parents were asked to provide feedback on the Hall of Heroes program via evaluation surveys. Youth were asked to rate how much they agreed with statements regarding their engagement and liking of Hall of Heroes. Parents rated the extent to which they agreed with statements such as “I would recommend Hall of Heroes to other parents of or grade youth” and “Hall of Heroes was easy for my child to understand.” Youth and parent evaluations were made using a 5-point scale with 1 = strongly disagree and 5 = strongly agree. Open-ended responses were also collected, to collect more detailed feedback on youth and parent impressions of the SEL intervention.

2.3. Procedures

Precautions were taken to ensure study ethics and protection of human subjects. The study protocol was approved by 3C Institute’s institutional review board. Parent consent and youth assent were obtained from all participants prior to participation. All parts of the study were completed online through a secure project website.

All parents in the TX and WLC groups completed the BERS-2 and Y-OQ at pretest. The week following pretest data collection, youth in the TX group began interacting with Hall of Heroes. The game was released in four segments, with one segment released per week for four weeks; youth had one week to complete each segment. Within one week following the TX group’s four-week interaction with the program, parents in both the TX and WLC groups completed the same set of outcome measures again, as posttest measures, and youth and parents in the TX group completed product evaluation surveys.

3. Results and Discussion

Preliminary analyses were conducted using a Multivariate Analysis of Variance (MANOVA) with condition as the between-subjects factor to determine whether scores on the BERS-2 and the Y-OQ differed at baseline (preintervention) across the two conditions. These analyses revealed no significant difference between groups at baseline on any subscale of the BERS-2 or the Y-OQ. However, given the small sample size, we elected to use an Analysis of Covariance (ANCOVA) approach in order to control for baseline scores when analyzing intervention effects [45, 46].

3.1. Positive Behavioral and Emotional Skills

As can be seen in Table 2, results of the ANCOVA revealed significant improvements in positive behavioral and emotional skills for youth in the treatment condition, as measured by the BERS-2. As expected, parents reported that youth who completed Hall of Heroes showed significantly greater growth in their total Strength Index (a measure of youth overall behavioral and emotional strengths) compared to WLC youth who did not complete the intervention. Further, when this index was broken down into its composite subscales, youth in the treatment condition showed particular gains in three areas:  interpersonal strength, affective strength, and family involvement. Thus, youth who completed Hall of Heroes significantly improved in their abilities to relate to others (both peers and family members) as well as to accept affection and express emotions with others, whereas youth who did not complete Hall of Heroes showed essentially no change in these areas. These results were expected given that Hall of Heroes directly targets social skills that are critical for positive interpersonal relationships, including communication, cooperation, and social initiation as well as emotional awareness and emotion regulation skills which would directly impact the youth’s affective abilities.

Table 2: Summary of relevant descriptive information, ANCOVA statistics, and effect sizes for parent-reported outcomes on the BERS-2.

Mean (SE)
ANCOVA

Pre
Post
F
p-value
Effect Size

()

Total Strength Index

    Treatment
92.73 (3.73)
110.33 (4.39)
5.96
.022
.19

    Wait-list Control
94.50 (6.01)
95.43 (6.21)

Interpersonal Strength

    Treatment
6.73 (.67)
9.07 (.67)
19.25
<.001
.43

    Wait-list Control
7.07 (.74)
6.43 (.75)

Affective Strength

    Treatment
8.13 (.77)
10.20 (.73)
6.29
.019
.20

    Wait-list Control
7.43 (.80)
7.43 (.98)

Family Involvement

    Treatment
7.07 (.69)
9.60 (.77)
14.32
<.001
.36

    Wait-list Control
7.14 (.58)
7.29 (.62)

Intrapersonal Strength

    Treatment
6.40 (.47)
9.20 (.84)
2.67
.115
.09

    Wait-list Control
7.00 (.89)
7.57 (.83)

School Functioning

    Treatment
7.80 (.55)
9.60 (.62)
.91
.349
.03

    Wait-list Control
8.57 (1.07)
8.64 (1.01)

No significant difference by treatment condition was found for the BERS-2 intrapersonal strength subscale. It may be that the duration of the study was too brief (4 weeks) to impact youth’s internal emotional strengths and opinions of their own competencies and accomplishments, which may take more time and real-world experience to alter than possible with this study design. No significant difference was found for the BERS-2 school functioning subscale either. However, Hall of Heroes does not directly address academics or school-related tasks, so lack of a treatment effect for this subscale may be expected.

While intrapersonal and school academics were not directly targeted by the Hall of Heroes intervention, improvements for these subscales may emerge with greater exposure to the intervention and/or more time for these more distal outcomes to be impacted by social skill improvements for the youth. As the youth practice and implement the social skills taught in Hall of Heroes, effects on self-efficacy and academic competence may emerge. Future research with a longer intervention period could directly test this hypothesis.

3.2. Psychosocial Distress

As can be seen in Table 3, ANCOVA results demonstrated TX youth experienced significantly lower Overall Psychosocial Distress on the Y-OQ at postintervention compared to WLC youth, after controlling for preintervention scores. This result suggests that Hall of Heroes’ bolstering of social emotional skills may be key to helping youth cope with stressful situations. Results further indicated that this overall significant effect was largely due to significant improvement in intrapersonal distress for youth in the TX condition. Thus, youth who completed Hall of Heroes showed a significantly greater decline in feelings of anxiety, depression, and hopelessness than did youth who did not participate in the intervention.

Table 3: Summary of relevant descriptive information, ANCOVA statistics, and effect sizes for parent-reported outcomes on the Y-OQ.

Mean (SE)
ANCOVA

Pre
Post
F
p-value
Effect Size

()

Overall Distress

    Treatment
82.07 (6.46)
54.20 (7.06)
4.57
.042
.15

    Wait-list Control
73.79 (9.76)
65.57 (8.89)

Intrapersonal Distress

    Treatment
29.47 (2.27)
18.73 (2.72)
8.23
.008
.24

    Wait-list Control
24.29 (2.75)
21.71 (2.81)

Somatic

    Treatment
9.73 (1.35)
6.07 (1.34)
3.58
.070
.12

    Wait-list Control
8.29 (1.58)
7.79 (1.52)

Critical Items

    Treatment
9.53 (1.75)
5.33 (.92)
.86
.103
.10

    Wait-list Control
8.43 (1.92)
6.64 (1.30)

Interpersonal Relationships

    Treatment
8.07 (1.36)
5.00 (1.83)
.99
.329
.04

    Wait-list Control
9.43 (1.61)
7.93 (1.62)

Social Problems

    Treatment
3.80 (1.23)
2.60 (1.20)
.98
.332
.04

    Wait-list Control
5.14 (1.42)
4.71 (1.49)

Behavioral Dysfunction

    Treatment
21.47 (1.93)
16.47 (1.54)
.24
.632
.01

    Wait-list Control
18.21 (2.19)
16.79 (1.73)

While not statistically significant, two other Y-OQ subscales showed trends suggesting youth who completed Hall of Heroes appeared to experience fewer symptoms of somatic distress (e.g., headache, stomach, and dizziness) and fewer critical psychological symptoms (e.g., thoughts of suicide, paranoia, and mania) compared to youth who did not complete the intervention. We know, from the literature, youth who experience social problems often develop somatic complaints, either as a function of the anxiety associated with interpersonal problems or as a coping mechanism to avoid socially stressful situations, such as school [9]. The trend towards fewer somatic symptoms may be an indirect effect from the overall lessening of psychosocial distress in the area of anxiety for youth in the treatment condition. Similarly, lower psychosocial distress in the areas of depression and hopelessness would be expected to decrease the level of serious psychological (critical) symptoms experienced by a youth.

No statistically significant between-group differences were found on the Y-OQ for the interpersonal relationships or social problems subscales. This finding is somewhat surprising given the significant findings on similar interpersonal strength subscale on the BERS-2, reported above. However, for the Y-OQ, interpersonal relations focuses on arguing, defiance, and negative communication problems and social problems focuses on delinquent and aggressive behavior problems. While greater (but nonsignificant) declines in these problem areas were seen for youth in the TX condition compared to those in the WLC condition, these areas represent antisocial behavior problems that are not the primary focus of Hall of Heroes. Hall of Heroes focuses on teaching prosocial skills and behaviors and positive social problem solving. We know that antisocial behavior problems are more resistant to change [13] and that positive impact on antisocial behavior problems may take longer than the 4-week intervention period of this study. It may be that declines in these antisocial behavior problems would emerge with greater exposure to Hall of Heroes as well as greater real-world experience using prosocial skills that are antithetical to antisocial behaviors.

With regard to the behavioral dysfunction of the Y-OQ, this subscale focuses on ADHD-related symptoms, such as disorganization, poor concentration, and trouble handling frustration, which are not directly addressed by the Hall of Heroes intervention. Thus, it is not too surprising that there was no significant group difference for this particular subscale. However, while Hall of Heroes is not designed as a treatment tool for any specific disorder (e.g., Attention Deficit Hyperactivity Disorder, Conduct Disorder), it is possible that improvements in these types of behavioral problem areas may arise over time as youth practice new social emotional skills in the real-world. Future longitudinal research would be needed to test whether group differences in antisocial and ADHD-related behavior problems emerge over time for youth.

3.3. Hall of Heroes Evaluation

Because Hall of Heroes was designed to be used independently by youth and because greater engagement in intervention is associated with greater skill gains (Clark et al., 2016) [47], it was important to examine how youth evaluated the game. In general, as can be seen in Table 4, both youth and parents rated Hall of Heroes favorably across all areas. Parents supported Hall of Heroes as engaging, and youth reported they found the game fun and enjoyed both the characters and story. Also of note were youths’ high marks for interest in playing more games like Hall of Heroes and parents’ belief that Hall of Heroes is an effective tool for teaching social skills with youth.

Table 4: Means (SD) of youth and parent evaluations of Hall of Heroes.

Youth Ratings
Mean (SD)

    Overall, I liked Hall of Heroes.
4.23 (.83)

    I would like to play more games like Hall of Heroes.
4.38 (.65)

    I thought Hall of Heroes was fun to play.
4.31 (.85)

    I liked the characters in Hall of Heroes.
4.23 (.92)

    I liked the pictures/graphics in Hall of Heroes.
3.92 (1.38)

    I liked the story in Hall of Heroes.
4.15 (1.14)

Parent Ratings
Mean (SD)

    Hall of Heroes is of high overall quality.
4.00 (.65)

    Hall of Heroes was fun and engaging for my child.
4.27 (.80)

    Hall of Heroes was easy for my child to understand.
4.00 (.85)

    My child’s overall experience with Hall of Heroes was positive.
4.13 (.83)

    Hall of Heroes is an effective tool for teaching students social skills.
4.20 (.68)

    Hall of Heroes is an effective tool for preparing students for middle school.
4.27 (.59)

    I would recommend Hall of Heroes to other parents of or grade students.
4.33 (.82)

Note. All ratings were made on a 5-point scale from 1 (strongly disagree) to 5 (strongly agree).

Follow-up requests for open-ended feedback helped to put these ratings into context. When the youth were asked what they learned in Hall of Heroes, nearly all noted at least one of the program goals, including “cooperation with others,” “respect for others,” “how to deal with feelings,” and “communication and working together,” to name a few. Representative comments from parents were like “my child learned about working together for a common goal,” “my child had fun enough to keep her wanting to play, yet helped her with some new ideas about how to interact with others,” and “there were more than few scenarios that my child encountered that forced him to reevaluate and change directions,” among other comments. Further, parents reported they were likely to recommend Hall of Heroes to other parents of adolescents.

4. Conclusion

The results of this study provide support for the usability and acceptability of Hall of Heroes as well as initial evidence of its potential as a game-based social skills training (SST) intervention for adolescents. High ratings from both youth and parents, and the positive open-ended feedback, support the idea that the digital game Hall of Heroes is an acceptable and engaging way to implement SST. Also encouraging was the fact that we found all participants were able to complete the intervention in the allotted time, and there were no reports of technical or user errors with the software. These findings indicate youth are able to complete Hall of Heroes independently and are sufficiently engaged in the program to maintain interest and motivation throughout SST.

Findings also provided initial support for the efficacy of participation in Hall of Heroes for increased interpersonal and emotional strength and decreased psychosocial distress, key ingredients to preparing youth for the transition to middle school. Despite the small sample size and brief intervention period, youth who completed the game-based SST showed improvements in multiple areas. In fact, those areas found to change significantly as a function of intervention (e.g., lower intrapersonal distress, greater interpersonal and affective strengths) showed moderate to high effect sizes, a treatment benefit commiserate with more traditional in-person SST programs (see [13]).

This work supports the growing body of evidence showing that digital games are potentially effective for delivering a wide array of psychotherapeutic interventions, including SST. As evidence mounts confirming the importance of social skills as a protective force in early adolescence, particularly in helping youth cope with the stressful transition to middle school, the need for effective intervention methods that can overcome the barriers to traditional SST programs becomes increasingly urgent. Hall of Heroes and programs like it which deliver dynamic, personalized, and engaging social skills instruction offer a low-cost, accessible, and easy to scale alternative to reach youth with effective SST on a broader scale than ever before.

4.1. Limitations and Future Directions

This initial study of the Hall of Heroes game-based SST intervention adds to the growing literature indicating digital games can effectively address a wide array of social, emotional, and behavioral concerns (see review in [21]). This pilot test represents the first step in evaluating the efficacy of Hall of Heroes specifically and as such a number of limitations should be considered. First, our sample size was quite small which limits statistical power and our ability to investigate potentially informative subgroup differences (e.g., grade or gender differences) and also can potentially limit generalizability of findings. It is important to note that while the sample was recruited from the broader community with no requirement that youth have a preexisting social skills problem, it appears that parents who signed up were at least concerned about their child’s social skills or social functioning. Examination of baseline levels of each outcome measure showed elevated distress in all areas of the Y-OQ and below average strengths on the BERS-2 for both TX and WLC youth (no group differences were present at baseline). Given what appears to be a self-selection bias by parents for participation in this intervention study, results should be considered generalizable to adolescents in need of social skills intervention, rather than youth more generally. Future studies employing a larger sample size are needed to confirm treatment effects and investigate potential differential treatment response by subgroup, including youth with varying degrees of baseline social difficulties.

Second, the current study relied on parent report for outcome measures. Given our design, it was not possible to include independent measures where the reporter was not aware of the intervention condition of youth. Because parents were not blind to treatment condition, biases may have influenced their reported behavioral changes. However, it is important to note that the pattern of results was not equivalent across all subscales. If biases were solely responsible for parental ratings, we would expect all subscales to be equally impacted. However, those subscales that were most closely related to the content taught through Hall of Heroes (prosocial skills, positive social problem solving) were those that were found to be significantly improved by participation in the intervention. Outcome areas which are not directly targeted by the intervention (e.g., school-related performance, antisocial behavior problems, and attentional problems) or which may require a longer time period than 4 weeks to see change were not found to be significant in this study. Future investigations with multiple informants and informants who are blind to treatment condition, such as teachers, as well as a longer longitudinal design, would be important for clarifying the observed changes by parents in the current study.

Third, the brief intervention period meant data collection and intervention delivery occurred during the spring semester for all participants. Further research is needed to examine whether time of year impacts treatment effects. Studies which compare timing relative to the transition to middle school (e.g., late grade, early/late grade) and include measurement beyond immediate postintervention would be useful for determining the optimal conditions for delivering the intervention.

Data Availability

Individuals interested in the data generated through this study or gaining access to the Hall of Heroes game should contact the first author.

Conflicts of Interest

The authors acknowledge possible conflicts of interest related to this study. While the authors do not directly benefit financially from the results of this study, both authors are affiliated with the company that sells Hall of Heroes (https://www.Centervention.com) and as such may indirectly benefit in the future from the results of the study. They have administered this study in accordance with ethical research guidelines and do not believe this potential conflict has impacted the reported results.

Acknowledgments

This research was supported by the US Department of Education, Institute of Education Sciences, through contract ED-IES-13-C-0041. The authors would like to thank the parents and youth who participated in this research as well as the Hall of Heroes development team for their contributions to this program. In addition, the authors would like to thank Dr. Ashley Craig and Emily Brown for their contributions to earlier drafts of this paper as well as their contributions to the development and testing of Hall of Heroes.

References

A. Bellmore, “Peer Rejection and Unpopularity: Associations With GPAs Across the Transition to Middle School,” Journal of Educational Psychology, vol. 103, no. 2, pp. 282–295, 2011.

K. R. Wentzel and K. Caldwell, “Friendships, Peer Acceptance, and Group Membership: Relations to Academic Achievement in Middle School,” Child Development, vol. 68, no. 6, pp. 1198–1209, 1997.

J. W. Aikins, K. L. Bierman, and J. G. Parker, “Navigating the transition to junior high school: The influence of pre-transition friendship and self-system characteristics,” Social Development, vol. 14, no. 1, pp. 42–60, 2005.

M. E. Gifford-Smith and C. A. Brownell, “Childhood peer relationships: Social acceptance, friendships, and peer networks,” Journal of School Psychology, vol. 41, no. 4, pp. 235–284, 2003.

K. H. Rubin, W. M. Bukowski, and J. G. Parker, “Peer interaction, relationships, and groups,” in Handbook of child psychology: Vol. 3. Social, emotional, and personality development, N. Eisenberg, R. W. Damon, and R. M. Lerner, Eds., vol. 3, pp. 571–645, Wiley, New York, NY, USA, 6th edition, 2006.

L. Zarbatany, D. P. Hartmann, and D. B. Rankin, “The Psychological Functions of Preadolescent Peer Activities,” Child Development, vol. 61, no. 4, pp. 1067–1080, 1990.

M. A. Peterson, E. B. Hamilton, and A. D. Russell, “Starting well: Facilitating the middle school transition,” Journal of Applied School Psychology, vol. 25, no. 3, pp. 286–304, 2009.

J. N. Kingery, C. A. Erdley, and K. C. Marshall, “Peer acceptance and friendship as predictors of early adolescents' adjustment across the middle school transition,” Merrill-Palmer Quarterly, vol. 57, no. 3, pp. 215–243, 2011.

J. Kupersmidt and M. E. DeRosier, “How peer problems lead to negative outcomes: An integrative mediational model,” in Children's peer relations: From development to intervention, J. B. Kupersmidt and K. A. Dodge, Eds., pp. 119–138, American, Washington, DC, USA, 2004.

J. Parker, K. Rubin, S. Erath, J. Wojslawowicz, and A. Buskirk, “Peer relationships, child development, and adjustment: Developmental psychopathology perspective,” in Developmental Psychopathology, Cicchetti and Cohen, Eds., pp. 419–493, Wiley, New York, NY, USA, 2006.

K. R. Wentzel, “Sociometric status and adjustment in middle school: A longitudinal study,” Journal of Early Adolescence, vol. 23, no. 1, pp. 5–28, 2003.

M. E. DeRosier, “Building Relationships and Combating Bullying: Effectiveness of a School-Based Social Skills Group Intervention,” Journal of Clinical Child & Adolescent Psychology, vol. 33, no. 1, pp. 196–201, 2004.

J. A. Durlak, R. P. Weissberg, A. B. Dymnicki, R. D. Taylor, and K. B. Schellinger, “The impact of enhancing students’ social and emotional learning: a meta-analysis of school-based universal interventions,” Child Development, vol. 82, no. 1, pp. 405–432, 2011.

S. Foster and J. Bussman, “Evidence-based approaches to social skills training with children and adolescents,” in Handbook of evidence-based therapies for children and adolescents: Bridging science and practice, pp. 409–428, Springer Science & Business Media, New York, NY, USA, 2007.

K. T. Mueser and A. S. Bellack, “Social skills training: Alive and well?” Journal of Mental Health, vol. 16, no. 5, pp. 549–552, 2007.

M. Greenberg, C. Domitrovich, and B. Bumbarger, “The prevention of mental disorders in school-aged children: current state of the field,” Prevention & Treatment, vol. 4, pp. 1–67, 2001.

R. Raimundo, A. Marques-Pinto, and M. L. Lima, “The Effects Of A Social-Emotional Learning Program On Elementary School Children: The Role Of Pupils' CharacterisTICS,” Psychology in the Schools, vol. 50, no. 2, pp. 165–180, 2013.

F. M. Gresham, “Current status and future directions of school-based behavioral interventions,” School Psychology Review, vol. 33, no. 3, pp. 326–343, 2004.

K. Bosworth, D. Espelage, T. DuBay, G. Daytner, and K. Karageorge, “Preliminary evaluation of a multimedia violence prevention program for adolescents,” American Journal of Health Behavior, vol. 24, no. 4, pp. 268–280, 2000.

Social and Character Development Research Consortium (SCDRC), “Efficacy of Schoolwide Programs to Promote Social and Character Development and Reduce Problem Behavior in Elementary School Children,” Tech. Rep. NCER 2011-2001, NCER, IES, Washington, DC, USA, 2010.

M. E. DeRosier and J. M. Thomas, “Video games and their impact on teens mental health,” in Proceedings of the Technology and adolescent mental health, M. Moreno and A. Radovic, Eds., pp. 237–253, Springer International Publishing, 2018.

J. J. Horan, “Effects of Computer-Based Cognitive Restructuring on Rationally Mediated Self-Esteem,” Journal of Counseling Psychology, vol. 43, no. 4, pp. 371–375, 1996.

D. H. Goh, R. P. Ang, and H. C. Tan, “Strategies for designing effective psychotherapeutic gaming interventions for children and adolescents,” Computers in Human Behavior, vol. 24, no. 5, pp. 2217–2235, 2008.

M. S. Khanna and P. C. Kendall, “Computer-assisted cognitive behavioral therapy for child anxiety: Results of a randomized clinical trial,” Journal of Consulting and Clinical Psychology, vol. 78, no. 5, pp. 737–745, 2010.

K. Bosworth, D. Espelage, and T. DuBay, “A computer-based violence prevention intervention for young adolescents: Pilot study,” Adolescence, vol. 33, no. 132, pp. 785–795, 1998.

D. S. McNamara, G. T. Jackson, and A. Graesser, “Intelligent tutoring and games (ITaG),” in Gaming for classroom-based learning: Digital role playin as a motivator of study, Y. Baek, Ed., pp. 44–65, formation Science Reference, Hershey, Pa, USA, 2010.

Á. E. Carrasco, “Acceptability of an adventure video game in the treatment of female adolescents with symptoms of depression,” Research in Psychotherapy: Psychopathology, Process and Outcome, vol. 19, no. 1, pp. 10–18, 2016.

K. Fenstermacher, D. Olympia, and S. M. Sheridan, “Effectiveness of a computer-facilitated, interactive social skills training program for boys with attention deficit hyperactivity disorder,” School Psychology Quarterly, vol. 21, no. 2, pp. 197–224, 2006.

A. Rubin-Vaughan, D. Pepler, S. Brown, and W. Craig, “Quest for the Golden Rule: An effective social skills promotion and bullying prevention program,” Computers & Education, vol. 56, no. 1, pp. 166–175, 2011.

R. P. Sanchez, C. M. Bartel, E. Brown, and M. E. DeRosier, “The acceptability and efficacy of an intelligent social tutoring system for children with social skills challenges,” Computers and Education, vol. 78, pp. 321–332, 2014 (Bulgarian).

M. Moisala, V. Salmela, L. Hietajärvi et al., “Gaming is related to enhanced working memory performance and task-related cortical activity,” Brain Research, vol. 1655, pp. 204–215, 2017.

J. L. Tan, D. H.-L. Goh, R. P. Ang, and V. S. Huan, “Participatory evaluation of an educational game for social skills acquisition,” Computers & Education, vol. 64, pp. 70–80, 2013.

A. Corbett, K. Koedinger, and W. Hadley, “Cognitive tutors: From research to classrooms,” in Technology enhanced learning: Opportunities for change, P. S. Goodman, Ed., pp. 235–263, Erlbaum, Mahway, NJ, USA, 2001.

J. P. Gee, “What video games have to teach us about learning and literacy,” Computers in Entertainment, vol. 1, no. 1, p. 20, 2003.

R. Bartle, “Hearts, clubs, diamonds, spades: Players who suit MUDs,” Journal of MUD Research, vol. 1, no. 1, p. 19, 1996.

C. Girard, J. Ecalle, and A. Magnan, “Serious games as new educational tools: How effective are they? A meta-analysis of recent studies,” Journal of Computer Assisted Learning, vol. 29, no. 3, pp. 207–219, 2013.

K. Kiili, “Digital game-based learning: towards an experiential gaming model,” The Internet and Higher Education, vol. 8, no. 1, pp. 13–24, 2005.

P. Akos and J. P. Galassi, “Middle and high school transitions as viewed by youth, parents, and teachers,” Professional School Counseling, vol. 7, no. 4, pp. 212–221, 2004.

C.-H. Chen and M.-H. Chou, “Enhancing middle school students' scientific learning and motivation through agent-based learning,” Journal of Computer Assisted Learning, vol. 31, no. 5, pp. 481–492, 2015.

Y. Kim and A. L. Baylor, “A social-cognitive framework for pedagogical agents as learning companions,” Educational Technology Research and Development, vol. 54, no. 6, pp. 569–596, 2006.

R. Moreno, R. E. Mayer, H. A. Spires, and J. C. Lester, “The case for social agency in computer-based teaching: do students learn more deeply when they interact with animated pedagogical agents?” Cognition and Instruction, vol. 19, no. 2, pp. 177–213, 2001.

Y. Kim, J. Thayne, and Q. Wei, “An embodied agent helps anxious students in mathematics learning,” Educational Technology Research and Development, vol. 65, no. 1, pp. 219–235, 2017.

M. H. Epstein, Behavioral and Emotional Rating Scale, Pro-ed, Austin, Tex, USA, 2nd edition, 2004.

G. M. Burlingame, M. G. Wells, J. C. Cox, M. J. Lambert, M. Iatkowski, and D. Justice, Youth Outcome Questionnaire, OQ Measures, LLC, Salt Lake City, Utah, USA, 2005.

S. E. Maxwell, D. A. Cole, R. D. Arvey, and E. Salas, “A Comparison of Methods for Increasing Power in Randomized Between-Subjects Designs,” Psychological Bulletin, vol. 110, no. 2, pp. 328–337, 1991.

A. J. Vickers and D. G. Altman, “Analysing controlled trials with baseline and follow up measurements,” British Medical Journal, vol. 323, no. 7321, pp. 1123-1124, 2001.

J. J. Vogel, D. S. Vogel, J. Cannon-Bowers, G. A. Bowers, K. Muse, and M. Wright, “Computer gaming and interactive simulations for learning: a meta-analysis,” Journal of Educational Computing Research, vol. 34, no. 3, pp. 229–243, 2006.