Autism Research and Treatment

Volume 2014 (2014), Article ID 935686, 12 pages

http://dx.doi.org/10.1155/2014/935686

Research Article

Computer Vision Tools for Low-Cost and Noninvasive Measurement of Autism-Related Behaviors in Infants

Jordan Hashemi,¹ Mariano Tepper,¹ Thiago Vallin Spina,² Amy Esler,³ Vassilios Morellas,⁴ Nikolaos Papanikolopoulos,⁴ Helen Egger,⁵ Geraldine Dawson,⁶ and Guillermo Sapiro⁷

¹Department of Electrical and Computer Engineering, Duke University, Durham, NC 27708, USA

²Institute of Computing, University of Campinas, 13083 Campinas, SP, Brazil

³Department of Pediatrics, University of Minnesota, Minneapolis, MN 55455, USA

⁴Department of Computer Science and Engineering, University of Minnesota, Minneapolis, MN 55455, USA

⁵Department of Psychiatry and Behavioral Sciences, Duke University, Durham, NC 22708, USA

⁶Department of Psychiatry and Behavioral Sciences and School of Medicine, Duke University, Durham, NC 27708, USA

⁷Department of Electrical and Computer Engineering, Department of Computer Science, and Department of Biomedical Engineering, Duke University, Durham, NC 27708, USA

Received 19 November 2013; Revised 30 April 2014; Accepted 13 May 2014; Published 22 June 2014

Academic Editor: Herbert Roeyers

Copyright © 2014 Jordan Hashemi et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

The early detection of developmental disorders is key to child outcome, allowing interventions to be initiated which promote development and improve prognosis. Research on autism spectrum disorder (ASD) suggests that behavioral signs can be observed late in the first year of life. Many of these studies involve extensive frame-by-frame video observation and analysis of a child's natural behavior. Although nonintrusive, these methods are extremely time-intensive and require a high level of observer training; thus, they are burdensome for clinical and large population research purposes. This work is a first milestone in a long-term project on non-invasive early observation of children in order to aid in risk detection and research of neurodevelopmental disorders. We focus on providing low-cost computer vision tools to measure and identify ASD behavioral signs based on components of the Autism Observation Scale for Infants (AOSI). In particular, we develop algorithms to measure responses to general ASD risk assessment tasks and activities outlined by the AOSI which assess visual attention by tracking facial features. We show results, including comparisons with expert and nonexpert clinicians, which demonstrate that the proposed computer vision tools can capture critical behavioral observations and potentially augment the clinician's behavioral observations obtained from real in-clinic assessments.

1. Introduction

The analysis of children’s natural behavior is of key importance for the early detection of developmental disorders such as autism spectrum disorder (ASD). For example, several studies have revealed behaviors indicative of ASD in early home videos of children that were later diagnosed with ASD [1–5]. These studies involved video recording infant behavior and then coding and analyzing the data a posteriori, using frame-by-frame viewing by an observer who typically trains for several weeks to achieve interrater reliability. Hours of labor are required, thereby making such analyses burdensome for clinical settings as well as for big data studies aiming at the discovery or improvement of behavioral markers. While clinical tools for early screening of ASD are available, they require administration and interpretation by specialists. Many families in low resource communities lack easy access to specialists in ASD. This work examines the potential benefits that computer vision can provide for research in early detection of ASD risk behaviors. It is a first milestone in a long-term project aimed at developing low-cost, automatic, and quantitative analysis tools that can be used by general practitioners in child development and in general environments to identify children at risk for ASD and other developmental disorders.

Although much is unknown about the underlying causes of ASD, some neuropathological studies indicate that ASD may have its origins in abnormal brain development early in prenatal life [6]. Moreover, Zwaigenbaum et al. [7] argue that many children with ASD exhibit several specific behavioral markers as early as in the first year of life. In high-risk siblings of children who later developed ASD, some of these symptoms can be observed during activities involving visual attention and are often expressed as difficulties in disengagement and shifting of attention [8, 9]. In addition, high-risk infant siblings have been shown to demonstrate impaired visual attention development between the ages of and months [10]. With this said, there is still much research needed to be done in determining potential risk indices. For example, a study performed by Nadig et al. [11] found that there is no significant difference in response-to-name disengagement between at-risk and low-risk infants.

Despite the fact that autism symptoms often emerge early and the syndrome can be diagnosed in toddlers, the average age of ASD diagnosis in the USA is close to 5 years [12]. Recently, research has demonstrated the benefit of early detection and diagnosis to allow for early intensive intervention. Early intervention, initiated in preschool and sustained for at least 2 years, can substantially improve child outcomes [13]. Detecting ASD risk and starting interventions before the full set of behavioral symptoms appears to may ultimately have an even greater impact, preventing difficult behaviors and delayed developmental trajectories from taking hold [14]. Although the diagnosis of ASD involves much more than the detection of symptoms, improving availability of cost-effective and accessible methods for identifying which children might be at risk and in need of further evaluation would potentially be of value. Towards this end, we have focused on developing semiautomatic computer vision video analysis techniques to aid in measuring ASD-related behaviors which can be used in early detection research.

More specifically, the main objectives in this paper are to use and validate computer vision tools to capture reliably two critical visual attention behaviors, Disengagement of Attention and Visual Tracking, belonging to the Autism Observation Scale for Infants (AOSI) [15], a behavioral observation tool for gathering information on early ASD risk signs [16]. (In this paper we refer to the AOSI and the scoring based on the DSM-IV. Research is needed to assess how measures of early signs of autism derived from automated video coding, such as those from the AOSI, relate to later diagnosis of ASD based on the newly established DSM-5 criteria. This is the subject of on-going efforts in our team.) Thus, the aim of the study is to examine the correspondence between the measures derived from the computer vision tools and the clinical assessment given by one trained expert. To demonstrate the validity of our tools, we compare our computer vision methods’ results to those of the AOSI trained expert who performed the assessments and three nonexperts. Towards this end, other objectives of our work are to demonstrate the accurate and objective measurements provided by our low-cost methods and their potential to be used in the research of ASD risk marker identification. The work with such a specific population of infants and toddlers is unique in the computer vision community, making this a novel application for the psychology community. While the data is obtained from actual clinical assessments, the tasks pulled from the assessment are easy to administer and/or involve recordings of the child’s natural behavior, thereby opening the door to broad behavioral studies, considering that the actual analysis is automatically done as introduced here.

These tools could potentially aid the practitioner and researcher in the risk marker identification task by providing accurate and objective measurements. These measurements can further provide means for improving the shareability of clinical records without compromising anonymity. In addition and particularly for research, automatic analysis will permit researchers to analyze vast amounts of naturally recorded videos, opening the door for data mining towards the improvement of current assessment protocols and the discovery of new behavioral features. This project is being developed by a multidisciplinary group bringing together professionals from psychology, computer vision, and machine learning. As an alternative to other research strategies [17–19], which require laboratory assessments, one of our main goals is to provide nonintrusive capturing systems that do not necessarily induce behavioral modification in the children. In other words, hardware must not constrain the testing environment; for example, the children are not asked to wear any type of sensors [20, 21].

2. Methods

2.1. Procedures for AOSI Tasks

The AOSI consists of a set of tabulated tasks that are designed for assessing specific behaviors, where each task consists of a certain number of presses and the child’s responses receive scores. According to the AOSI, Disengagement of Attention is characterized as the “ability to disengage and move eyes/attention from one of two competing visual stimuli” [15, 22], while Visual Tracking is characterized as the “ability to visually follow a moving object laterally across the midline” [15, 22]. During the AOSI assessment, the clinician performs three trials for the Disengagement of Attention task and two trials for the Visual Tracking task, per participant. Every trial receives an AOSI-tabulated score, according to the following guidelines.

Disengagement of Attention. This activity consists of (1) shaking a noisy toy to one side of the infant until his/her attention is engaged and then (2) shaking a second noisy toy on the opposite side, while continuing to shake the first object. A delayed response in high-risk infants has been shown to be associated with a later ASD diagnosis [7, 15]. A trial is considered “passed” if the child looks to the second object in less than 1s, considered “delayed” if the child looks after a 1-2 s delay, and considered “stuck” if the child looks after more than 2 s.

Visual Tracking. To evaluate this activity, the following is performed: (1) a rattle or other noisy toy is used to engage the infant’s attention, (2) the rattle is positioned to one side of the infant, and (3) the rattle is then moved silently at eye level across the midline to the other side. In high-risk infants, an interrupted, delayed, or partial gaze tracking has been shown to be associated with a later ASD diagnosis [15]. Depending on how continuously and smoothly the participant is able to track the object, the trial is considered “passed,” “delayed or interrupted,” or “partial or no tracking.”

The clinician makes a “live” judgment about these time frames or may look at videos of this task if available. Finally, an overall score for each task is computed by merging the individual trials. We followed the protocol of comparing the assessments done by (1) an expert psychologist examiner who has been trained in the AOSI as well as ASD diagnosis in children, (2) a child/adolescent’s psychiatrist, (3) two psychology students with no particular autism training, and (4) the results of our new computational tools. The child/adolescent’s psychiatrist and two psychology students assigned their scores by following the AOSI guidelines, without prior training, while watching the same videos used by the automatic method. This setup allows us to contrast the automatic method’s findings with human assessments across the full range of expertise.

2.2. Computer Vision Algorithms for Assessing Visual Attention

To analyze the child’s reactions in the Visual Attention activities, we automatically estimate the changes of two head pose motions: yaw (left and right motion) and pitch (up and down motion). For the Visual Tracking and Disengagement of Attention tasks, which involve lateral motions, we focus on the yaw motion. We develop computer vision algorithms for estimating these head motions from low-cost cameras. The algorithms track specific facial features: the left ear, left eye, and nose (see right image of Figure 1). From their positions we estimate the participant’s yaw and pitch motions. The only user input in our algorithm is during initialization. On the first frame, the user places a bounding box around the left ear, left eye, and nose (Figure 7). This could potentially be avoided by standard feature detection techniques. We marked the play objects by hand, although this also can be done automatically from prior knowledge of their visual and sound features (e.g., color or squeaking noise). Additional technical details are available in Appendices A and B (Figure 6 presents the data flow of our visual attention analysis system). We should note that we exploit and extend computer vision techniques that form the basis of virtual all automatic face analysis systems, and therefore have been extensively validated in the literature.

(a)

(b)

Figure 1: (a) General scene capturing the AOSI evaluation session. (b) Example of our algorithm automatically tracking three facial features: the left eye, left ear, and nose. In this paper, all figures have been blurred to protect the participants’ privacy.

Scoring for our automatic method is based on automatic visual inspection of the estimated head motion measurements. After marking when the second object is presented in the Disengagement of Attention task, our method is able to automatically determine when the participants start and complete disengagement from the first object to the second. We assign the disengagement delay based on how many frames/seconds it takes the participant’s head motion to completely transition to the second object (note that we are recording the video at 30 frames per second). We incorporate of a second margin for each delay to accommodate human error of making a live judgment. The scoring for the Visual Tracking task is determined by visual inspection of the head motion measurements as the object is moving laterally in front of the participants. More specifically, the scores are assigned based on whether or not the measurements exhibit full lateral head motion and also depend on the rate of change of the measurements. A “pass” is assigned if the head motion measurements exhibit full lateral head motion and a smooth rate of change. If there is an instance where the measurements exhibit a plateau or the rate of change changes direction for a short period of time but the measurements still display full lateral head motion, an “interrupted” score is assigned. For trials where the measurements do not exhibit full lateral head motion, a “partial” or “no tracking” score is assigned. Examples of our method’s measurements for a “pass,” “interrupted,” and “partial” or “no tracking” tracking scores are shown in Figure 3. The developed automatic technique operates at a much higher resolution and accuracy than the standard -second intervals used by the expert clinician during live testing.

2.3. Participants

The purpose of the study was not to examine the correspondence between early assessments and outcome but rather the ability of our tools to accurately capture individual differences in behavior. We sought to include a sample in which a diversity of responses to the AOSI would be expected; thus the sampled population of this study involves 12 at-risk participants being examined in a clinic, including both males and females ranging in age from 5 to 18 months. Approval for this study was obtained from the Institutional Review Board at the University of Minnesota, and we have gathered our data from a series of ASD evaluation sessions of an ongoing concurrent study performed on a group of at-risk infants, at the Department of Pediatrics of the University of Minnesota.

All at-risk participants were infant siblings of a child diagnosed with ASD, a premature infant, or as a participant showing developmental delays. Table 1 presents a summary of this information. Note that, the participants are not clinically diagnosed until they are months of age and only participant number 3 has presented conclusive signs of ASD.

Table 1: Information on participants involved in this study. Each participant was chosen for a different reason: being a baby sibling of someone with ASD, a premature infant, or showing developmental delays.

Part number
Age (months)
Gender
Risk degree

Number 1
14
F
Showing delays

Number 2
5
F
Baby sibling

Number 3
16
M
Showing delays

Number 4
15
M
Showing delays

Number 5
8
M
Premature infant

Number 6
9
F
Premature infant

Number 7
10
F
Premature infant

Number 8
9
M
Premature infant

Number 9
7
M
Premature infant

Number 10
6
M
Baby sibling

Number 11
9
M
Premature infant

Number 12
18
M
Showing delays

2.4. Hardware

In our clinical setup, we use a low-cost GoPro Hero HD color camera (with a resolution of 1080 p at 30 fps), placed freely by the clinician in the center of the table between 2 and 4 feet away from the participant to ensure that it remains still throughout each trial and captures both the clinician and the participant (e.g., left image of Figure 1). The displayed images here are downsampled, blurred, and/or partially blocked to preserve anonymity (processing was done on the original videos).

3. Results

3.1. Disengagement of Attention

Table 2 summarizes the results of our method, the clinical assessments, and the ratings by a child/adolescent’s psychiatrist and two psychology students for the Disengagement of Attention task. (See all video results in supplementary video files available online at http://dx.doi.org/10.1155/2014/935686.) Since the current set up for the visual attention tasks only involves a single camera placed nonintrusively, there are trials that our current method cannot handle (this could be easily solved in the future with a second low-cost camera). These trials include instances when the participant left the camera’s field of view or when a toy or object obstructed it. (Standard face detection algorithms, such as the ones used in digital cameras, can be used to automatically alert the clinician of such cases for repositioning of the camera if needed.) For Table 2, the trials with blank spaces and a horizontal line correspond to such cases. Out of the 24 trials that the clinician assigned a “pass” score, our method agreed on 23 of them and scored a “delayed” for the other trial. And out of the 3 trials the clinician scored “delayed” our method agreed on 2 trials, scoring one as a “pass.” The clinician did not assign a score for number 2, stating that it was a “spoiled” trial due to the participant being afraid of the toys. However, we show our method’s results to exemplify a possible Disengagement of Attention score.

Table 2: Results of Disengagement of Attention task. Scores from the clinician (Clin.), automatic method (Automatic), a psychiatrist (Psy.), and two students (St. 1 and St. 2) for each trial. A trial is considered either as “passed” (Pass), “delayed” (Del), or “stuck” (Stck) depending on whether the child disengages from the first object in less than 1 s, between 1 and 2 s, or more than 2 s, respectively. We also present the automatically computed delay that the child takes to disengage. Note that we consider of a second margin for each delay to accommodate human error of making a live judgment.

Part number
Clin.
Automatic
Psy.
St. 1
St. 2

Score
Delay (s)

First trial score

Number 1
Pass
Pass
0.9
Pass
Del
Del

Number 2
N/A
Del
1.87
N/A
N/A
N/A

Number 3
Pass
Pass
0.5
Pass
Del
Del

Number 4
Pass
Pass
0.23
Pass
Pass
Pass

Number 5
Del
Pass
1.07
Del
Del
Del

Number 6
Pass
Pass
1.03
Pass
Del
Pass

Number 7
Pass
Pass
0.83
Pass
Del
Del

Number 8
—
—
—
—
—
—

Number 9
—
—
—
—
—
—

Number 10
Pass
Pass
0.87
Del
Del
Del

Number 11
Pass
Pass
0.83
Pass
Del
Pass

Number 12
Pass
Pass
0.93
Pass
Pass
Pass

Second trial score

Number 1
Pass
Pass
0.7
Pass
Pass
Pass

Number 2
—
—
—
—
—
—

Number 3
—
—
—
—
—
—

Number 4
Pass
Pass
1.1
Pass
Pass
Pass

Number 5
Del
Del
1.77
Del
Stck
Del

Number 6
Pass
Del
1.43
Pass
Pass
Pass

Number 7
Pass
Pass
0.97
Del
Del
Del

Number 8
Pass
Pass
1.33
Pass
Del
Del

Number 9
—
—
—
—
—
—

Number 10
Pass
Pass
1.3
Pass
Pass
Pass

Number 11
Pass
Pass
0.63
Pass
Pass
Pass

Number 12
Pass
Pass
0.9
Pass
Del
Del

Third trial score

Number 1
Pass
Pass
0.37
Pass
Pass
Pass

Number 2
—
—
—
—
—
—

Number 3
—
—
—
—
—
—

Number 4
Pass
Pass
0.3
Pass
Pass
Pass

Number 5
Pass
Pass
0.5
Pass
Del
Pass

Number 6
Pass
Pass
0.7
Del
Del
Del

Number 7
Pass
Pass
1.13
Pass
Del
Del

Number 8
—
—
—
—
—
—

Number 9
Del
Del
1.37
Pass
Del
Stck

Number 10
Pass
Pass
1.33
Del
Del
Del

Number 11
Pass
Pass
0.87
Pass
Pass
Pass

Number 12
Pass
Pass
0.87
Pass
Pass
Pass

To further clarify our results, Figure 2 displays examples of our method’s results and important cases for the Disengagement of Attention task. In Figure 2(a), the participant is able to disengage from the first object and look at the second within 0.7 s (21 frames) of the second object being presented. This would be scored as “passed” on the AOSI test. The participant in Figure 2(b) disengages to the second object within 1.3 s (40 frames), which would be scored as “delayed” on the AOSI test.

(a)

(b)

(c)

Figure 2: (a) First example of the Disengagement of Attention task. Top: when the clinician is holding one object, when the second object appears, and when the child recognizes the second object. Middle: changes in the yaw motion ( values in the -axis; see Appendices A and B) for every frame (-axis). The dotted line represents when the second object is presented, followed by boxes representing 1 and 2 seconds after the object is presented. Bottom: 6 examples of the infant’s face during the task. All facial features are automatically detected and tracked (as indicated by the colored boxes around the nose, eyes, and ear). Colors and roman numerals identify corresponding images and spikes in the graph. (b, c) Two examples of the Disengagement of Attention task.

(a)

(b)

(c)

Figure 3: (a) First example of the Visual Tracking task. Top: when the clinician is holding the object, when the object is at one extreme side (right or left), and when the object is at the other extreme side. Middle: changes in the yaw motion ( values in the -axis; see Appendices A and B) for every frame (-axis). The boxes labeled “R” and “L” represent when the object is to the right and left of the participant, respectively. The gray shaded areas represent when the object is not moving and at an extreme side (either right or left). Bottom: 6 examples of the infant’s face during the task. Colors and roman numerals identify corresponding images and spikes in the graph. (b, c) Two examples of the Visual Tracking task.

3.2. Visual Tracking

Table 3 summarizes the results of our method, the clinical assessments, and the ratings by a child/adolescent’s psychiatrist and two psychology students for the Visual Tracking task. As in Table 2, the trials with blank spaces and a horizontal line could not be used by our automatic method. Out of the 14 trials that the clinician assessed as “pass”, our method agreed with 13 of them and scored an “interrupted” for 1 of the trials. For all the 4 trials the clinician assessed as “interrupted,” our automatic method was in agreement. The clinician scored two trials as “partial,” our method scored one of them as “partial” and the other as “interrupted.”

Table 3: Results of Visual Tracking task. Scores from the clinician (Clin.), automatic method (Automatic), a psychiatrist (Psy.), and two students (St. 1 and St. 2) for each trial. A trial is considered “passed” (Pass), “delayed” (Del), “interrupted” (Int), “partial” (Prt), or “no tracking” depending on how smoothly the child visually tracks the object.

Part number
First trial score
Second trial score

Clin.
Automatic
Psy.
St. 1
St. 2
Clin.
Automatic
Psy.
St. 1
St. 2

Number 1
Pass
Pass
Pass
Pass
Pass
Pass
Int
Int
Int
Int

Number 2
Int
Int
Pass
Int
Pass
Int
Int
Pass
Pass
Pass

Number 3
Del
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass

Number 4
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass

Number 5
Prt
Int
Pass
Int
Pass
Prt
Prt
Del
Prt
Prt

Number 6
Pass
Pass
Pass
Pass
Pass
—
—
—
—
—

Number 7
—
—
—
—
—
Int
Int
Int
Prt
Prt

Number 8
Pass
Pass
Pass
Del
Pass
Pass
Pass
Pass
Pass
Pass

Number 9
Pass
Pass
Pass
Del
Pass
Pass
Pass
Pass
Pass
Pass

Number 10
Pass
Pass
Pass
Pass
Pass
Int
Int
Int
Prt
Prt

Number 11
Pass
Pass
Pass
Pass
Pass
Int
Int
Int
Prt
Prt

Number 12
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass

Figure 3 shows important examples of our results for the Visual Tracking task. Figure 3(a) demonstrates a participant that received a “passed” on the AOSI’s Visual Tracking task, since the participant was able to smoothly track the object with minimal delay as the object approached the participant’s right. In Figure 3(b), the participant exhibited “interrupted” tracking motion. The participant’s tracking of the object was interrupted as the object moved across the clinician’s face. Instead of tracking the object as it moved across the clinician’s face, the participant stopped tracking the object and looked at the clinician for 0.46 s (14 frames) before continuing to track the object as it moved to the participant’s left. Such short behaviors can be detected by an automatic system. In Figure 3(c), the participant displays a “partial” tracking score on the AOSI. As the object crosses the clinician’s face, the participant completely stops tracking the object and instead looks straight at the clinician.

3.3. Comparisons between Ratings by Automatic Computer Vision Method, Nonexpert Clinical Raters, and Expert Clinician

We next compared ratings made by nonexpert clinical raters (child/adolescent’s psychiatrist and two psychology students) and by the computer vision methods with ratings made by an expert clinician. The results obtained by the child/adolescent’s psychiatrist and two psychology students are presented in Tables 2 and 3. Out of the 27 Visual Disengagement trials (Table 2), the two psychology students agreed with the clinician on 13 and 16 of the trials, respectively, while the child/adolescent’s psychiatrist agreed on 22 trials. The computer vision system agreed with the expert clinician in 25 out of the 27 cases. Similarly for the 22 Visual Tracking trials (Table 3), the two psychology students agreed with the expert clinician on 13 and 14 of the trials, respectively, while the child/adolescent’s psychiatrist agreed on 16 trials. The computer vision system agreed on 19 of the 22 cases. Table 4 shows the interrater reliability value for each individual compared with the expert clinician, based on weighted Cohen’s kappa with a range of 0-1, where 1 means complete agreement.

Table 4: Number of agreements with the autism expert for each participant in the two visual attention tasks and overall interrater reliability using weighted Cohen’s kappa. See Tables 2 and 3.

Task
Trials
Automatic
Psychiatrist
Student 1
Student 2

Disengagement
27
25
22
13
16

tracking
22
19
16
13
14

Total
49
44
38
26
30

Interrater score
—
0.75
0.37
0.27
0.27

4. Discussion

In addition to providing a broadly deployable low-cost tool for ASD risk assessment, if validated in future research, the potential benefits of an automated method for head motion estimation are threefold. First, it would provide accurate quantitative measurements for tasks assessing infant visual attention, such as the AOSI tasks, improving the shareability of clinical records while not compromising anonymity. Second, it could also prove beneficial in the discovery of new behavioral patterns by easily collecting large amounts of data and mining it. Third, it could increase the granularity of the analysis by providing data at a finer scale. As the results demonstrate, the computer vision method performed very well when compared to the expert clinician and outperformed the three other nonexperts. Using unweighted kappa, Bryson et al. [15] reported a combined average interrater reliability score of for both the Disengagement of Attention and Visual Tracking. Although our combined average weighted score of is not as high as reported by the original authors, it is still considered excellent (greater than ) according to them.

4.1. Disengagement of Attention

Compared to the expert clinician’s results, the computer vision method achieved high accuracy. In Table 2, the method obtained one false positive by scoring one trial “delayed” that the clinician scored as “pass” (participant number 6) and missed one “delayed” trial (participant number 11). In the current project, the temporal resolution is 30 frames per second, allowing for discovery of possible latent head motion patterns. Figure 2(c) provides an interesting pattern in the participant’s head movement. Not only does it take the third participant over 1 s to look at the second object (which is “delayed” on the AOSI), but the participant displays piece-wise constant lateral head movements compared to the other two examples (which presented a much smoother motion), a pattern virtually impossible to detect with the naked eye. Again, such automatic and quantitative measurements could potentially add critical information that could aid in risk detection, such as new ways of scoring the AOSI. With a study on a larger population, new time intervals (and their variability) for scoring may be discovered, and these false positives could be analyzed not as a strict “pass” or “delayed” but as something in between.

4.2. Visual Tracking

Again, compared to the expert clinician’s results, the automated method achieved high accuracy. As shown in Table 3, the clinician scored one trial belonging to the only participant that has been positively diagnosed as “delayed”; however, based on our nonintrusive camera placement (as selected by the practitioner), it was not possible to continuously extract the object’s location accurately enough to assign “delayed” scores. In future studies, it would be possible to extend the setup to include an overhead camera to accurately record the playing objects’ positions. Another aspect of the method is that it provides accurate and quantitative measurements of the participant’s head motions; thus, one is able to automatically determine the delays between when the participant looks at the object or how long the participant stops his/her tracking.

5. Concluding Remarks

This work is the first milestone in a long-term project focused on the development of noninvasive early observation of children in order to aid in risk detection and research of neurodevelopmental disorders. With the goal of aiding and augmenting the visual analysis capabilities in evaluation and developmental monitoring of ASD, we proposed (semi-) automatic computer vision tools to observe specific important behaviors related to ASD elicited during AOSI, providing both new challenges and opportunities in video analysis. The proposed tools, if validated in future research, could significantly reduce the effort by only requiring interactive initialization in a single frame. The eventual goal is to minimize the need for extensive training and add both accuracy of quantitative measurements and objectivity. We focused on two visual attention activities performed during the AOSI. We developed specific computer vision algorithms for these activities, obtaining encouraging results that correlated with an expert’s judgment. The automated method proved to be more consistent with the expert clinician’s rating that those produced by less expert human observers; it should be noted that these human observers still have higher levels of expertise than normally available in most school settings.

The improvement and extension of the proposed methods is an ongoing work and we plan to provide code for the method. Current limitations of our methods include requiring the participant’s face to be present throughout the video, estimating the yaw and pitch motions independently of one another, and we have only tested on relatively high-definition video sequences (from low-cost GoPro Hero cameras). Although the present study included at-risk infants and toddlers, some limitations of our study include the use of only a single expert examiner as well as the small and relatively unimpaired sample size (although as mentioned before, the exploited tools have been extensively validated in the literature). For the visual attention tasks, we plan to complement the estimation of the child’s motions with estimating the red examiner’s behaviors. The work presented demonstrates the validity of the tools with a specific examiner’s assessments. To expand these tools to broader clinical and naturalistic settings that involve different examiners and examiners with different levels of training, the quality of interaction, engagement, and variability between the examiners must also be considered. These examiner-related behaviors include how far away from the participant the examiner positions the toy(s), the displacement velocity of the toy(s) throughout the assessment, audio cues from the examiner, and head position of the examiner in the sense of whether he/she is looking at the participant or away. A study on the variability of examiner’s performance in both clinical and naturalistic settings, such as at the participant’s home, is an important topic for future research and should be considered before any data mining analyses are carried out. A method that examines the examiner’s behaviors would also allow the examiner to receive immediate feedback on whether a trial or press needs to be redone. Notice that this could also lead to an automatic method for training examiners. Based on the set-up of the AOSI tasks discussed in this work, we assume that the participant’s head motion is directly correlated to his/her gaze direction. This assumption is known as “center bias” and has been well established in gaze estimation and saliency detection literature [23, 24]. To further research and broaden the scope of this method, we plan to validate the assumption of the direct relationship between the head motion and eye movements in a specific population of infants and children at risk for ASD or with ASD; see also [25] for some early results in this direction.

5.1. Extensions of Computer Vision Tools for ASD Behavioral Observations

There are additional potential behavioral risk indices for ASD, both included in and beyond the scope of AOSI, such as facial expression in first birthday home videos [26] and mounting a camera near the examiner’s face to estimate the participant’s gaze [27], which are not addressed by the current method but we aim to address in the future, both in terms of the technical methods and the assessment of their validity and reliability. Computer vision tools hold promise as a novel and cost-efficient method for capturing behavior that could be used in both the clinical and research settings. Using the methods in this work, we also provide initial data suggesting that these methods might be useful in less structured situations, such as tracking a participant’s head motion during a ball playing activity (Figure 4), providing information regarding the participant’s interaction with the examiner such as the time it takes for a participant to look up after receiving a ball.

Figure 4: Two examples of the ball playing activity. Top: when the ball contacts the child, when the child looks down at the ball, and when the child looks up at the clinician. Middle: changes in the pitch motion (-axis) for each frame (-axis). The dotted line represents when the ball contacts the participant. Bottom: 6 examples of the infant’s face during the administration. All facial features are automatically detected and tracked. Colors and roman numerals identify corresponding images and spikes in the graph.

In addition, computer vision tools are not restricted to only aiding visual attention related markers. Our group has also been developing nonintrusive tools for aiding the assessment of motor patterns [28]. Through a meticulous process of hand fitting stickman to the participants in every frame, Esposito et al. [29] have found that toddlers with ASD often presented asymmetric arm positions in early life. Using computer vision, we were able to automatically estimate the D body pose of the toddlers in the video segments (Figure 5) as well as estimate arm angles relative to the ground with little user intervention (see mentioned reference for technical details and evaluations). The only required user intervention consisted of interactively creating a segmentation mask in the initial frame of the video segment. By creating nonintrusive and semiautomatic computer vision tools, such as these, large datasets can be analyzed, potentially leading to the discovery of new and/or improved behavioral markers.

Figure 5: Comparison of arm-asymmetry scores between automatic method and ground truth in a video segment containing participant number 1. The cyan line represents normalized results from our method, while the red line represents the ground truth (GT) of the normalized arm differences. See [28] for more information on arm-asymmetry calculations and analysis. The normalized color scale visually displays the angle difference between the right and left forearms, where symmetric arm positions have similar overlaying colors.

Figure 6: Data flow of the system for estimating yaw and pitch motions. We only require the user to place a bounding box around the left ear, left eye, and nose in the first frame of the video segment (black ellipse at time ). All the subsequent steps occur in a fully automatic fashion (blue blocks).

Figure 7: The triangle created by the left ear, left eye, and nose. The leftmost and rightmost images depict the triangle when the infant is looking right and more towards the camera, respectively. The middle image shows the points used for calculating .

Appendices

A. Tracking and Validating Facial Features

This section provides technical details about the algorithm for tracking facial features and computing head motions from them. The large variability of the data and the lack of control about the camera positioning call for using very simple and robust features and algorithms.

We assume that, in the first frame, we have bounding boxes of three facial features: the left ear, left eye, and nose (see, e.g., Figure 2). These bounding boxes are in practice selected by hand on the first frame. It is possible to achieve a fully automatic initialization, but this was not the objective of the present work. The user intervention is nonetheless minimal.

We aim at tracking these three facial features. Following a scheme loosely based on the TLD tracker [30], we use dense motion estimation coupled with a validation step that employs an offline-trained facial feature detector. The dense motion estimator [31] tracks the features with high accuracy in most cases, but when the child’s head moves quickly, illumination changes can sometimes cause the tracker to lag behind the features. Thus, we validate the output of the tracker using facial feature detectors in every frame (Figure 6).

To validate the features we train left eye, right eye, left ear, and nose detectors. For this, we adapt the widely used method by Dalal and Triggs [32], proposed for pedestrian detection, to our particular setting (see also [33, 34]). Our method employs the popular multiscale Histograms of Orientated Gradients (HOG) using pixel blocks and 9 orientation bins as descriptors to represent each facial feature and then classifies these descriptors using a support vector machine (http://www.csie.ntu.edu.tw/~cjlin/libsvm/) with a radial basis function kernel (see [32] for further technical details). As positive training samples, we use hand labeled facial patches from children in our experimental environment. As negative training samples, we extract random patches from around multiple children’s faces. Our classifier was trained a single time before any experiment was carried out. Then, we used it for all experiments with no need for retraining or parameter setting.

For each frame, search areas for the facial feature detectors are defined around the bounding boxes given by the tracker. The left eye, left ear, and nose are present in every frame for the given camera position and their final detected positions are determined by the locations that exhibit a maximal response from the classifier (i.e., extrema of each feature’s classifier output). The tracker’s bounding boxes are validated if their centers are within the bounding boxes returned by the detectors; however, if the tracker’s centers are outside of the detector’s bounding boxes for two consecutive frames, then the corresponding bounding box for the tracker is reset to a new location within the detector’s bounding box. Determining the presence of the right eye aids in the estimation of the yaw motion. The rectangular search area for the right eye, which is not tracked since it appears and disappears constantly due to the camera position, is based on the location of the detected nose and the horizontal and vertical distances between the detected left eye and nose. More specifically, the search area is between the detected nose’s location plus/minus the horizontal and vertical distances between the detected left eye and nose. Also employed as a right eye search area restriction is that the nose must be between the left and right eyes.

Thus, using our method, we are able to track the facial features via a dense motion estimator, and validate their positions via the facial feature detectors. To estimate head motion, see the following, we use the facial feature locations given by the dense motion estimators. The dense motion estimator provides smoother and more precise locations than the detector.

B. Yaw and Pitch Motion Estimation from Facial Features

As a way to provide an accurate motion estimation of the pitch angle, we cumulatively sum the vertical coordinate changes of the left eye and nose with respect to the left ear every frame. We expect a positive sum when the child is looking up and a negative sum when the child is looking down, with the magnitude representing how much the child is looking up or down.

For estimating the yaw motion, we calculate two ratios based on the triangle created by the left ear, left eye, and nose (Figure 7); we also use information about the presence of the right eye. Let , , and denote the locations of the nose, left eye, and left ear, respectively. For the first ratio , we project into the line defined by , thus defining the point ; we then define , where is the Euclidian distance. For the second ratio, we project into the line defined by , defining .

The two ratios and are inversely proportional. Looking at Figure 7, we can observe that when the face is looking in profile view, will be large and will be small, conversely when the face is in frontal view (looking more towards the camera). To combine these two ratios into one value, we calculate the normalized difference between them, . Thus, as the child is looking to his/her left, goes to −1; and, as the child is looking to his/her right, goes to 1. The presence of the right eye further verifies that the infant is looking left.

We incorporate whether the right eye is present or not to verify that the infant is looking left or right at the maximum and minimum values.

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Authors’ Contribution

J. Hashemi, M. Tepper, and T. V. Spina equally contributed to this work.

Acknowledgments

The authors would like to thank the participants and their families for their participation. This work was partially supported by NSF Grants 1039741 and 1028076, CAPES (BEX 1018/11-6) and FAPESP (2011/01434-9) Ph.D scholarships from Brazil, ONR, NGA, ARO, and AFOSR.

References

J. L. Adrien, M. Faure, A. Perrot et al., “Autism and family home movies: preliminary findings,” Journal of Autism and Developmental Disorders, vol. 21, no. 1, pp. 43–49, 1991.

J. L. Adrien, A. Perrot, D. Sauvage et al., “Early symptoms in autism from family home movies. Evaluation and comparison between 1st and 2nd year of life using I.B.S.E. scale,” Acta Paedopsychiatrica, vol. 55, no. 2, pp. 71–75, 1992.

G. Losche, “Sensorimotor and action development in autistic children from infancy to early childhood,” Journal of Child Psychology and Psychiatry and Allied Disciplines, vol. 31, no. 5, pp. 749–761, 1990.

G. T. Baranek, “Autism during infancy: a retrospective video analysis of sensory-motor and social behaviors at 9-12 months of age,” Journal of Autism and Developmental Disorders, vol. 29, no. 3, pp. 213–224, 1999.

F. Muratori and S. Maestro, “Early signs of autism in the first year of life,” in Signs of Autism in Infants: Recognition and Treatment, pp. 46–62, Karnac, London, UK, 2007.

P. M. Rodier, “Converging evidence for brain stem injury in autism,” Development and Psychopathology, vol. 14, no. 3, pp. 537–557, 2002.

L. Zwaigenbaum, S. Bryson, T. Rogers, W. Roberts, J. Brian, and P. Szatmari, “Behavioral manifestations of autism in the first year of life,” International Journal of Developmental Neuroscience, vol. 23, no. 2-3, pp. 143–152, 2005.

R. Landry and S. E. Bryson, “Impaired disengagement of attention in young children with austism,” Journal of Child Psychology and Psychiatry and Allied Disciplines, vol. 45, no. 6, pp. 1115–1122, 2004.

M. Elsabbagh, A. Volein, K. Holmboe et al., “Visual orienting in the early broader autism phenotype: disengagement and facilitation,” Journal of Child Psychology and Psychiatry and Allied Disciplines, vol. 50, no. 5, pp. 637–642, 2009.

M. Elsabbagh, J. Fernandes, S. Jane Webb, G. Dawson, T. Charman, and M. H. Johnson, “Disengagement of visual attention in infancy is associated with emerging autism in toddlerhood,” Biological Psychiatry, vol. 74, no. 3, pp. 189–194, 2013.

A. S. Nadig, S. Ozonoff, G. S. Young, A. Rozga, M. Sigman, and S. J. Rogers, “A prospective study of response to name in infants at risk for autism,” Archives of Pediatrics and Adolescent Medicine, vol. 161, no. 4, pp. 378–383, 2007.

P. T. Shattuck, M. Durkin, M. Maenner et al., “Timing of identification among children with an autism spectrum disorder: findings from a population-based surveillance study,” Journal of the American Academy of Child and Adolescent Psychiatry, vol. 48, no. 5, pp. 474–483, 2009.

J. S. Howard, C. R. Sparkman, H. G. Cohen, G. Green, and H. Stanislaw, “A comparison of intensive behavior analytic and eclectic treatments for young children with autism,” Research in Developmental Disabilities, vol. 26, no. 4, pp. 359–383, 2005.

G. Dawson, “Early behavioral intervention, brain plasticity, and the prevention of autism spectrum disorder,” Development and Psychopathology, vol. 20, no. 3, pp. 775–803, 2008.

S. E. Bryson, L. Zwaigenbaum, C. McDermott, V. Rombough, and J. Brian, “The autism observation scale for infants: scale development and reliability data,” Journal of Autism and Developmental Disorders, vol. 38, no. 4, pp. 731–738, 2008.

K. Downing, K. Concannon, V. Vogel-Farley, C. A. Nelson, and H. Tager-Flusberg, “Can the AOSI at nine months discriminate between infants at high or low risk for ASD?” in International Meeting for Autism Research (IMFAR '11), San Diego, Calif, USA, 2011.

W. Jones, K. Carr, and A. Klin, “Absence of preferential looking to the eyes of approaching adults predicts level of social disability in 2-year-old toddlers with autism spectrum disorder,” Archives of General Psychiatry, vol. 65, no. 8, pp. 946–954, 2008.

M. Freeth, T. Foulsham, and P. Chapman, “The influence of visual saliency on fixation patterns in individuals with autism spectrum disorders,” Neuropsychologia, vol. 49, no. 1, pp. 156–160, 2011.

A. Klin, W. Jones, R. Schultz, F. Volkmar, and D. Cohen, “Visual fixation patterns during viewing of naturalistic social situations as predictors of social competence in individuals with autism,” Archives of General Psychiatry, vol. 59, no. 9, pp. 809–816, 2002.

M. S. Goodwin, S. S. Intille, F. Albinali, and W. F. Velicer, “Automated detection of stereotypical motor movements,” Journal of Autism and Developmental Disorders, vol. 41, no. 6, pp. 770–782, 2011.

F. A. Boujarwah, S. Sadler, A. Mogus, G. D. Abowd, and R. I. Arriaga, “Understanding the challenges and opportunities for richer descriptions of stereotypical behaviors of children with ASD: a concept exploration and validation,” in Proceedings of the 12th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '10), pp. 67–74, Orlando, Fla, USA, October 2010.

S. E. Bryson, L. Zwaigenbaum, J. Brian et al., “A prospective case series of high-risk infants who developed autism,” Journal of Autism and Developmental Disorders, vol. 37, no. 1, pp. 12–24, 2007.

S. Mannan, K. H. Ruddock, and D. S. Wooding, “Automatic control of saccadic eye movements made in visual inspection of briefly presented 2-D images,” Spatial vision, vol. 9, no. 3, pp. 363–386, 1995.

Y. Li, A. Fathi, and J. Rehg, “Learning to predict gaze in egocentric video,” in Proceedings of the International Conference on Computer Vision (ICCV '13), Sydney, Australia, 2013.

H. Bloch and I. Carchon, “On the onset of eye-head coordination in infants,” Behavioural Brain Research, vol. 49, no. 1, pp. 85–90, 1992.

J. Osterling and G. Dawson, “Early recognition of children with autism: a study of first birthday home videotapes,” Journal of Autism and Developmental Disorders, vol. 24, no. 3, pp. 247–257, 1994.

Z. Ye, Y. Li, A. Fathi et al., “Detecting eye contact using wearable eye-tracking glasses,” in Proceedings of the 14th International Conference on Ubiquitous Computing (UbiComp '12), pp. 699–704, Pittsburgh, Pa, USA, September 2012.

T. V. Spina, J. Hashemi, M. Tepper et al., “Automatically aiding the assessment of atypical motor patterns for autism spectrum disorder diagnosis,” in Proceedings of the 59th AACAP Meeting, San Fransisco, Calif, USA, 2012.

G. Esposito, P. Venuti, F. Apicella, and F. Muratori, “Analysis of unsupported gait in toddlers with autism,” Brain and Development, vol. 33, no. 5, pp. 367–373, 2011.

Z. Kalal, K. Mikolajczyk, and J. Matas, “Face-TLD: tracking-learning-detection applied to faces,” in Proceedings of the 17th IEEE International Conference on Image Processing (ICIP '10), pp. 3789–3792, Hong Kong, September 2010.

M. Tepper and G. Sapiro, “Decoupled coarse-to-fine matching and nonlinear regularization for efficient motion estimation,” in Proceedings of the 19th IEEE International Conference on Image Processing (ICIP '12), pp. 1517–1520, Orlando, Fla, USA, October 2012.

N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '05), pp. 886–893, San Diego, USA, June 2005.

M. Everingham, J. Sivic, and A. Zisserman, ““Hello! My name is... Buffy”—automatic naming of characters in TV video,” in Proceedings of the British Machine Vision Conference (BMVC '06), Edinburgh, UK, 2006.

T. Vatahska, M. Bennewitz, and S. Behnke, “Feature-based head pose estimation from images,” in Proceedings of the 7th IEEE-RAS International Conference on Humanoid Robots (HUMANOIDS '07), pp. 330–335, Pittsburgh, Pa, USA, December 2007.