Measuring progress of children with autism spectrum disorder (ASD) during intervention programs is a challenge faced by researchers and clinicians. Typically, standardized assessments of child development are used within research settings to measure the effects of early intervention programs. However, the use of standardized assessments is not without limitations, including lack of sensitivity of some assessments to measure small or slow progress, testing constraints that may affect the child’s performance, and the lack of information provided by the assessments that can be used to guide treatment planning. The utility of a curriculum-based assessment is discussed in comparison to the use of standardized assessments to measure child functioning and progress throughout an early intervention program for toddlers with risk for ASD. Scores derived from the curriculum-based assessment were positively correlated with standardized assessments, captured progress masked by standardized assessments, and early scores were predictive of later outcomes. These results support the use of a curriculum-based assessment as an additional and appropriate method for measuring child progress in an early intervention program. Further benefits of the use of curriculum-based measures for use within community settings are discussed.
Advancements in the identification, diagnosis, and treatment of very young children with autism spectrum disorder (ASD) have challenged researchers and clinicians to examine alternative assessments of child progress and outcome in early intervention programs. The most common assessments evaluating change across developmental domains (i.e., cognition, communication, social skills, adaptive behavior, and behavior challenges) are standardized assessments [
A recurrent theme throughout the intervention literature is the lack of sensitivity of standardized measures for children with ASD. For example, a child may show maintenance or decrease in standardized scores over time while simultaneously increasing in raw scores [
Another concern related to use of standardized measures with children with ASD is the influence of constraints during test administration. Administration of standardized assessments is usually highly structured, with the intention of keeping the administration consistent across many participants and true to established norms. Due to these restrictions, it is likely that a child’s behavior may be very different when assessed by an unfamiliar clinician in an unfamiliar clinic than during normal routines [
Ideally, child testing would provide valuable information that may be incorporated into an individual child’s ongoing intervention program. However, standardized assessments were not designed to provide detailed information relating to treatment goal development. In fact, developing intervention goals based on a child’s performance on standardized assessments is discouraged in order to avoid “teaching to the test.” Clinicians are advised to let at least six months pass before readministering the same standardized assessment to the same child as practice effects often are seen with multiple presentations of the same material within a recent time frame, making the results an invalid representation of the individual’s abilities [
In order to address the issues discussed above, we propose that standardized testing be supplemented with the use of curriculum-based assessments to provide finer detail on child progress and to assist with treatment individualization and planning. This paper presents the results of an evaluation of the utility of the adapted student learning profile (aSLP) to measure progress of children in an early intervention program specific to the aSLP curriculum. The aSLP is a curriculum-based measure that assesses mastery of targeted skills to measure a child’s progress during, and outcome after, an intervention program. The aSLP has the potential to measure child progress throughout ongoing intervention in a systematic way that better allows comparison of child progress and rate of learning in intervention within and across programs. We posit that using a combination of curriculum-based and standardized assessments to measure child functioning will provide a greater understanding of child progress and outcomes in early intervention.
The toddlers and families in this investigation were participants in a larger multidisciplinary research project examining early neurobiological features and development of ASD at the University of California, San Diego. Toddlers at risk for an ASD were obtained from one of two sources: general community referral (e.g., website or outside agency) and a population-based screening method called the 1-Year Well-Baby Check-Up Approach [
Participant demographics.
Gender | |
Male | 36 |
Female | 9 |
Age (months) | |
Intake | 22.67 (4.13) |
Exit | 34.06 (3.25) |
Length of treatment program | |
Months in intervention | 11.53 (4.04) |
Ethnicity | |
Hispanic or Latino | 12 |
Not-Hispanic or Latino | 21 |
Not reported | 12 |
Race | |
American Indian or Alaska Native | 0 |
Asian | 3 |
Black or African American | 4 |
Native Hawaiian or other Pacific Islander | 1 |
White | 25 |
Not reported | 12 |
Note: averages listed with standard deviations within parentheses.
The Strategies for Teaching Based on Autism Research (STAR) curriculum [
Each child received approximately 6–12 hours per week of direct one-on-one intervention with a trained ECI at home until 36 months of age (
The aSLP is a curriculum-based assessment for determining student learning goals and was adapted from the STAR curriculum to include additional goals from the TSC curriculum (see [
aSLP assessment example.
Lesson | Concept | Example instruction cue | Target skill | Student response | ||
---|---|---|---|---|---|---|
Preverbal communication | Goal directed reach to request | Teacher holds up object | Child reaches toward desired object | Rarely | Sometimes | Usually |
Eye contact to request | Teacher blocks access/withholds desired object | Child makes eye contact to obtain desired object | Rarely | Sometimes | Usually | |
Proximal point to request | Teacher holds up object | Child points to desired object | Rarely | Sometimes | Usually | |
Teacher holds up two objects | Child points to indicate a choice between two objects | Rarely | Sometimes | Usually | ||
|
||||||
Expanded learning to play | Multiple-step imitation | Verbal cue: “Do this.” Nonverbal play model: teacher puts man in toy car and then pushes the car | Student models teacher’s action (e.g., student puts man in a toy car and then student pushes car) | Never | If prompted | Independently |
|
||||||
Expanded playing with toys | Following two- or three-step play commands | Teacher gives verbal cue (e.g., “Put man in car and push car”) | Student responds to teacher cue without need for initiation (e.g., student puts man in the toy car and pushes it) | Never | If prompted | Independently |
Independent construction or functional play | No specific cue is needed | Child plays appropriately with toy during reinforcement phase of PRT lessons | Never | If prompted | Independently |
The MSEL assesses developmental functioning of children between birth and 68 months [
Three children were administered the WPPSI rather than the MSEL at exit as these children were performing at ceiling levels on the MSEL at the close of their early intervention program. The WPPSI assesses cognitive development in children between two years, six months and seven years, three months [
The VABS provides a measure of adaptive skills used to cope with challenges of daily living [
Means with standard deviations were calculated for all assessments at intake into the program, exit, and change scores from intake to exit. In order to analyze whether the aSLP measured progress in intervention similarly to the MSEL and VABS, partial correlations were conducted controlling for the amount of time the child was in treatment. Lastly, a linear regression was conducted to analyze if aSLP scores at three months into intervention were predictive of aSLP scores at exit. This analysis was included to examine if early treatment progress on the aSLP is indicative of later outcomes, which would give practitioners information about the importance of early performance, and insight into the type of outcome that could be expected. Analyses were conducted using the IBM SPSS Statistics 22 statistical analysis package.
Results for all assessments are summarized in Table
Summary of assessment scores.
MSEL ELC | |
Intake | 75.44 (14.87) |
Exit | 83.51 (21.76)* |
Change scores | 8.07 (16.46) |
VABS ABC | |
Intake | 84.38 (11.46) |
Exit | 83.13 (10.85) |
Change scores | −1.24 (11.58) |
aSLP score | |
Intake | 23.56 (24.27) |
Exit | 120.58 (61.87)* |
Change scores | 97 (50.96) |
Note: average scores listed with standard deviations within parentheses.
*Indicates statistically significant change from intake to exit.
aSLP scores for each participant over time. Each participant is depicted by one line. The aSLP was administered to each child at intake into the early intervention program and every 3 months thereafter. The aSLP score depicts the number of skills mastered at each assessment. Significant changes were seen in aSLP scores from intake to exit from the early intervention program.
Partial correlations between the MSEL, VABS, and aSLP scores, controlling for the number of months the children were in the treatment program, were conducted. Since child age or time passing is not controlled for on the aSLP as it is not a standardized measure, the effect of the amount of time the child spent in treatment on aSLP scores was investigated. The amount of time the child was in treatment had a small negative correlation with the aSLP scores at intake (
Correlations between the aSLP scores and MSEL ELC and VABS ABC at intake and exit of the early intervention program. Correlation coefficients are reported for Pearson correlations and partial correlations controlling for the amount of time in intervention. Significant positive correlations were found between all measures.
Correlation between MSEL ELC change scores and aSLP change scores (i.e., change in scores from intake to exit). Correlation coefficients are reported for the Pearson correlation and partial correlation controlling for the amount of time each child was in treatment. A significant positive correlation was found. Children exhibited an average of 97 aSLP skills learned on average and an average change score of 8 on the MSEL ELC.
Correlation (controlling for the amount of time each child was in treatment) between VABS ABC change scores and aSLP change scores (i.e., change in scores from intake to exit). Correlation coefficients are reported for the Pearson correlation and partial correlation controlling for the amount of time each child was in treatment. A significant positive correlation was found. Children exhibited an average of 97 aSLP skills learned on average and an average change score of −1 on the VABS ABC.
A linear regression between aSLP scores at 3 months and exit. aSLP scores at 3 months into treatment significantly predicted aSLP scores when participants exited the program.
Two of the children displayed very high scores across all three assessments at intake. These children were the only participants to score two standard deviations above the mean score on at least two of the three assessments. Therefore, the following analyses also were conducted excluding these two participants. Partial correlations controlling for the number of months the children were in treatment were conducted. aSLP scores at intake were positively correlated with the MSEL ELC at intake (
The results of this study provide support for the supplemental use of a curriculum-based assessment, the aSLP, for determining the benefits of an early intervention program for children with ASD. While standardized assessments such as the MSEL and the VABS are valuable tools in evaluating child outcome, they have some limitations that suggest the need for the addition of a curriculum-based measure such as the aSLP. High variability in scores among participants was seen across all assessments. Assessments were highly correlated, likely indicating that children who had higher overall cognitive and adaptive functioning were more likely to have mastered more skills, which is not surprising but may support the validity of a curriculum-based assessment. The VABS standard scores were not sensitive to change in the children’s skills during intervention as evidenced by nonsignificant changes in VABS standard scores over time. The MSEL and aSLP showed significant changes in scores over time. This may indicate that the aSLP and MSEL are sensitive to change in skill acquisition and development but do not capture change in adaptive functioning. One explanation may be that the STAR program is more heavily focused on behaviors related to cognitive gains for these very young children. Alternatively, the children may be showing specific changes in skill acquisition during treatment and testing sessions, but these skills may not generalize to daily functioning and use across multiple environments. The assessments may simply be measuring very different skills, or the age of the children may be a factor given the limited change in adaptive behavior expected on the VABS at this young age.
Given that the children’s scores on the aSLP were positively correlated with the MSEL and VABS at all time points and over time, it is likely that the aSLP is a comparable measure of child progress despite its lack of standardized norms. Two participants displayed high scores across all three assessments at intake. Correlations between the aSLP and MSEL and VABS were stronger with these participants included; however, intake, exit, and change scores were still moderately correlated when these participants were removed from the analyses. The evidence of significant positive correlations with or without these two participants suggests that the extreme scores of these two participants were not driving the correlations between the assessments and rather supports the use of the aSLP across a wide range of children with varying levels of functioning.
The addition of the aSLP in this study was very informative, in terms of ongoing child response to treatment, treatment trajectory, and overall outcome at the conclusion of the study. For example, over 1/3 of the children showed a decrease or no gain in MSEL or VABS standardized scores. However, all children showed an increase in skills mastered as measured by the aSLP. Tracking progress of children can become complicated when utilizing standardized measures as standardized scores are determined by comparing raw scores on the assessment to norms of children of the same chronological age. Thus, when evaluating children who may be progressing slower than typically expected, those children may actually exhibit no gain or a decrease in standardized scores since the children are not keeping up with the expected progress. Thus, interpreting the treatment success for this particular group of children based on the MSEL or VABS alone makes it difficult to determine rate of progress. It is important to have a measure of treatment progress in children who are developing more slowly to make appropriate changes to intervention strategies, determine rate of learning, and make predictions regarding service needs.
Additionally, the aSLP allowed the children’s in-home treatment coordinators to quickly and easily determine the children’s intervention progress throughout the intervention period. The results of the aSLP provided detailed information about specific behaviors, rather than general information about the child’s ability level, which is provided by standardized assessments. This allowed for easy analysis of child functioning at the skill by skill level, which is particularly beneficial for treatment planning. Very young children, like those in the current study, have very little experience with testing and may not do well on standardized tests, which makes using them to guide early treatment even more problematic. In contrast, the aSLP may be useful for early goal development and allow for a systematic yet individualized process for the child’s treatment program to follow. Additionally, the aSLP provided some information to hypothesize child’s trajectory of treatment. aSLP scores after 3 months of treatment predicted aSLP scores at exit. The aSLP scores provide information early on regarding the child’s predicted treatment trajectory. Early information on responsivity to treatment may provide valuable information for individualization of treatment. The utility of the aSLP for assessing and detecting treatment trajectory information so early on gives it a unique advantage for practical use within community settings, beyond what a standardized assessment might provide. Information about child progression through a curriculum such as the aSLP can be a very useful complement to standardized assessments as it gives a researcher or practitioner the opportunity to observe the real-world effects of treatment.
The main limitation of the current study is that the in-home coordinator for each child administered the aSLP, rather than employing a blind rater to assess the children. It is possible that the in-home coordinators had an expectancy bias when assessing the children, which may have influenced the scores derived from the aSLP. However, the in-home coordinators were not providing direct service to the children. Children were familiar with them but did not receive intervention directly from the person doing the testing. Blind raters were not employed due to the amount of additional resources that would be needed to carry out a blind rating system and because the purpose of the aSLP was to provide clinical data. Another limitation was the lack of formalized procedures to ensure interrater reliability of the aSLP ratings between in-home coordinators. Although the scoring procedure is straightforward and in-home coordinators were trained in the same manner, there is a potential for differences to occur between aSLP raters. Another concern with the use of the aSLP over time is the possibility of practice effects. Specific steps were taken to try to limit practice effects. The aSLP was given only every three months and tested items not specifically targeted during treatment. Feedback was not provided to the children regarding appropriate or correct responses. The task format was similar to that of intervention and familiar procedures tend to lead to fewer practice effects. However, it remains possible that practice effects may account for some of the increase in skills measures on the aSLP.
Additional limitations include the use of a small, young sample. Forty-five children under the age of three were evaluated throughout participation in an early intervention program. Future research should evaluate the usefulness of a curriculum-based measure such as the aSLP in a larger and older population. Likely, the benefits of a curriculum-based measure would generalize across populations, as the same benefits of ease of implementation, frequency of use, and direct translation to informing programming would be applicable. Additionally, as the aSLP is not a normed, standardized measure, we do not have a good understanding of the validity and reliability of the measure beyond what was explored in the current study. As such, the aSLP provides limited information in regard to the functioning level of the child with respect to same-aged peers and rather focuses on the skills learned in the treatment program. Therefore we are suggesting the use of a curriculum based assessment in conjunction with, not as a replacement for, standardized assessments. Also, although the items on the aSLP were not directly taught, the aSLP was used to develop treatment goals which may have inflated progress. However, skill gain was the important variable for these analyses. Additionally, these limitations are generalizable to how the assessment would be used in community practice; therefore this study represents an examination of the utility of the measure in community care.
The aSLP shows promise as a useful tool for measuring intervention progress and assisting with intervention development; however future research is needed to fully determine accuracy and limitations of the measure. Future studies should compare progress across groups of children in various intervention programs, including those not using the aSLP to develop curriculum items, to examine sensitivity to specific treatment changes. This will also help reduce issues of children learning items that are directly related to the assessment. Studies of the psychometric properties of the aSLP are imperative. Future research should establish interrater reliability with blind raters to further evaluate the rigor of this type of measure. Examination of the generalization of skills assessed by the aSLP is needed, including the relationship between the specific items on the aSLP and those on standardized measures of adaptive and cognitive functioning.
One of the greatest challenges facing early intervention researchers and community providers today is finding accurate and useful methods for assessing child response to treatment and overall outcome during and after a course of early intervention [
See
The authors declare that there is no conflict of interests regarding the publication of this paper.
This research was supported by the National Institutes of Health P50-MH081755 awarded to Eric Courchesne and R01-MH080134 awarded to Karen Pierce. The authors would like to thank Kevin Smith and Cory Rieth for assistance with statistical analyses as well as the families and children who participated in the study.