Questionnaire for Assessing Preschoolers ’ Organizational Abilities in Their Natural Environments : Development and Establishment of Validity and Reliability

Despite the consensus in the literature regarding the importance of organizational abilities in performing daily tasks, currently there is no assessment that focuses exclusively on such abilities among young children. The study aims to develop a Questionnaire for Assessing Preschoolers’ Organizational Abilities (QAPOA), Parents’ and Teachers’ versions, and to examine their reliability and validity. QAPOA was distributed to preschool teachers and parents of 215 typically developing 4–5.6-year-old children. The teachers’ and parents’ versions demonstrated good internal consistency. Factor analysis performed to examine the tool’s content validity yielded two factors: motor-based and language-based OA. Furthermore, both versions of the questionnaire demonstrated significant differences between OA among boys and girls. Concurrent validity was demonstrated between the QAPOA total scores and the equivalent subscale of the BRIEF-P. Given these findings, different cut-off scores were established for identifying boys and girls with either motor-based and/or language-based OA. The results indicate that both the teachers’ and parents’ versions of the QAPOA are reliable and valid measures of children’s organizational abilities. The questionnaires can assess and identify risk for organizational disabilities as early as preschool age. Thus, it can contribute to the planning of appropriate intervention programs and the prevention of difficulties in the future.

Children require organizational abilities to participate in age-appropriate daily occupations [7][8][9][10].For example, so as not to disrupt the progression of activities performed at home and in school, they must ensure that the necessary objects are readily accessible, do not become dispersed, or get lost.Spatial organization (school, playground, home) enables children to orient themselves and perform activities independently in familiar environments.Activities of daily living (such as brushing teeth, dressing, eating) should be temporally organized so that the child can meet his/her schedule demands (getting to school/friends/after-school activities) in a timely fashion.Organization of speech is also necessary to communicate one's experiences in the proper sequence so that they can be understood by the listener.Environmental demands for performing school tasks are easier to manage for children with efficient organizational ability [2,[11][12][13][14].The relationship between organizational ability and client factors such as perceptual, motor, cognitive, and language abilities is articulated in both the cognitive and neurodevelopmental frames of reference.Fried [15] emphasizes that organizational ability functions as a "conductor," orchestrating the client functions mentioned above.She presented a cognitive approach that ascribes organization as an executive function [16,17].Executive functioning (EF) refers to higher level cognitive functions, such as formulating and executing goals, planning, organization, inhibitory control, working memory, metacognition, and directing brain activity between the cognitive and emotional processing systems [18][19][20][21][22][23].
The neurodevelopmental approach describes the concept of organization according to the sensory integration (SI) framework [1].SI is a neurological process that organizes sensory input from all sensory channels from within the body and the environment to enable effective functioning.Ayres introduced the concept of praxis, defined as a basic organizational process that incorporates motor ideation, planning, and execution of a purposeful action.Accordingly, praxis refers to the process through which people formulate an idea (what to do) and organize it (how to do it) through a timed sequence of activities, depending on their current spatial context, enabling them to participate in meaningful occupational activities [24] Organization abilities have been demonstrated in studies of 3-4-year-old children [15,25] Although executive functions in general and organization in particular evolve at a rapid pace during the preschool years [20], there are few reports of standardized assessment tools for preschoolers focusing on organization [19].Moreover, despite the fact that the young child's everyday environments, both at home and at preschool, are important venues for observing routine manifestations of executive functions in general [8] and organizational abilities in particular, standardized and ecological assessments for young children are scarce [26][27][28].A few standardized assessments, such as the Goralnik [29], the Shatil test [30], and the Kaufman Assessment Battery for Children [31], have been developed but they do not assess the child in his/her natural environment.In addition, the Israeli Ministry of Education has published "Mabatim," a tool for use with 3-5-year-old preschoolers to assist teachers in observing and tracking children's preschool performance [32,33], but it relates only to their functioning within the educational environment and has not yet been validated.Other standardized, ecological tools for preschoolers include the Brief-P Inventory of Executive Function for Preschoolers [34,35] and the Performance Skills Questionnaire (PSQ) [36].These assessments focus on EF in young children, but only partially relate to organizational abilities.Thus, the QAPOA was developed to meet the need for an ecological standardized assessment focusing on organizational abilities among preschoolers and includes both a parents' and teachers' version.This tool enables educators and parents of preschoolers to identify children with rudimentary risk for organizational disabilities at an early age and refer them for further appropriate intervention.The purpose of this study was to establish the validity and reliability of both versions of the QAPOA.with a minimum of 10 years' experience working with preschoolers who were asked whether the statements did (yes) or did not (no) represent the organizational abilities of preschoolers.In accordance with their recommendations, ten items were added.

Methods
Next, the content validity of the revised 55-statement item pool was examined through the use of feedback questionnaires sent to 15 expert professionals (preschool teachers, OTs, and SLTs), with over 10 years of experience working with preschoolers.They were asked to rate each item as suitable, somewhat suitable, or very suitable in representing the organizational abilities of preschoolers.Ten questionnaires were completed by four OTs, four SLTs, and two preschool teachers.The maximum total score for each statement was 30 (3 points * 10 questionnaires, per statement).Only the 44 statements that received 18 or more points (60% of maximum score) were retained [7].Examples of statements in the area of play include "selects preferred games at home" and in the area of social interactions "relates and responds appropriately to what others say."In addition, four statements relating to the parent's or teacher's overall impression of the child's organizational ability in the four occupational areas (i.e., play, activities of daily living, preschool, and social interaction) were included, resulting in a total of 48 statements.
Following approval of the Chief Scientist of the Ministry of Education the questionnaires were distributed to the preschool teachers and parents of the participants to initiate the validation process for both versions of the QAPOA (P/T).

Internal Consistency
Reliability.The internal consistency (ICC) of both versions of the QAPOA (P/T) was found to be very high ( = 0.97 and 0.94, resp.).

Construct Validity.
Factor analysis yielded two factors for each version of the questionnaire, namely, motor-based organizational abilities and language-based organization abilities (Tables 1 and 2) As can be seen from these tables a high ICC was found between the items in each factor.Notably, according to both the teacher's and parents' reports, a significant difference was found between boys' and girls' scores ( = −4.11, < 0.001;  = −2.69, < 0.01, resp.)(Tables 1 and 2).

Convergent Validity.
Pearson's correlational analyses revealed significant correlations between the preschool teachers' total organization score on the QAPOA and the -score on the BRIEF-P Planning and Organization scale, kindergarten teacher's version ( = 0.46;  < 0.001), and between the QAPOA total organization scores and the teacher's overall impression of the children's organizational abilities ( = 0.73. < 0.001).Similar findings were found for the parents' scores on these scales (i.e., parent's total QAPOA organization score and the BRIEF-P -score,  = −0.41, < 0.001; QAPOA-P total score, and parents' overall impression of children's organization,  = 0.57,  < 0.001).

Questionnaire Cut-Off Scores Indicating Difficulty in
Organization.The QAPOA cut-off scores were found to be different for boys and girls, both with respect to total scores and the motor and language subscale scores.Total and subscale scores at or under the 16th percentile were identified as cut-off scores, indicating a risk for difficulties in organization.Thus, the identified total organization cutoff score for boys on the QAPOA-T was 173 and 185 on the QAPOA-P, and for girls it was 192 (QAPOA-T) and 196 (QAPOA-P).The same held true for cut-off scores for each subscale, in accordance with the findings of the factor analysis reported in Tables 1 and 2.
Specifically, the total cut-off score for boys on the QAPOA-T motor organization subscale was found to be 68, and it was 71 on the QPAOA-P, representing scores at or under the 16th percentile.For girls the total cut-off score signifying risk for organizational difficulties in the QAPOA-T was 76, and on the QAPOA-P it was 77.On the languagerelated organization subscale, the following cut-off scores were identified: for boys on the QAPOA-T = 90 and on the QAPOA-P = 94; for girls on the QAPOA-T = 99 and on the QAPOA-P = 100.

Discussion and Summary
The aim of the current research was to describe the development and examine the psychometric properties of the QAPOA, a practical, user-friendly tool for evaluating children's organizational abilities in ordinary life situations at kindergarten and at home.The statistical analyses found the QAPOA to be a valid and reliable tool.

Internal Consistency.
Internal consistency refers to the extent to which the individual items in an instrument measure the same construct.The statistic generated is a correlation coefficient ranging from 0 and +1.00, and a correlation of 0.80 is considered to demonstrate adequate reliability [38].
The item-selection procedure for the QAPOA was based on cognitive and neurodevelopmental theoretical models, as well as on the clinical experience of experts in preschool education and related therapy services (OT, SLP).The meticulous item-selection process described above testifies as to its content validity, supported by the high Cronbach's coefficient values found for both versions of the tool (QAPOA-P,  = 0.94; QAPOA-T,  = 0.97).
According to the neurodevelopmental [1] and cognitive [15,17] theoretical approaches upon which the initial item selection was based, organization is conceptualized in terms of its spatial and temporal dimensions.However, during the process of item selection, the need arose to also consider the organizational abilities needed for effective social interaction, as exemplified by items such as "relates to initiatives of peers during free play" or "seeks help when needed from adults or  children through gestures, spoken words, or sentences."Thus, we propose the following expanded definition of organizational skills: the ability to set meaningful goals and plan and execute them efficiently and successfully in the appropriate order and sequence, temporally and spatially with objects and people.

Construct Validity.
Construct validity is defined as the extent to which an operationalization measures the concept it is supposed to measure [38,39].Factor analysis is one approach used to establish construct validity for the purpose of determining how unobserved or latent factors exert directional influences on participants' responses of observed variables.It helps the researcher identify the number and nature of such latent factors [40].
The QAPOA questionnaire items were categorized according to the major occupations characterizing children's everyday activities: playing, learning, and daily activities of living and social interaction [41].Therefore, it was hypothesized that factor analysis would group the items into four factors corresponding to these four occupational areas.However, surprisingly, the items clustered into two factors: motor-based organizational skills, reflecting children's motor abilities (what and how to do), and language-based organizational skills, reflecting children's language ability (what and how it is said).
In light of this finding, three different cut-off scores were established to determine whether a child is within the normal range of organizational abilities: one for total organization, for motor-based organization, and for language-based organization.
The disclosure of these two factors enable us to conduct a more in depth, comprehensive assessment process that relates to client factors underlying organizational abilities.For example, "puts personal belongings in their proper place" (QAPOA-P, item 35) represents a motor-based expression of a child's organizational abilities in the natural environment.If the child scores below the cut-off, he/she can be referred for further evaluation by an Occupational Therapist (OT).Likewise, "seeks help from an adult or a child when needed using gestures, words, or sentences" (QAPOA-P, item 40) represents a language-based expression of a child's organizational abilities in his/her natural environment.Children obtaining a score below the cut-off in this area can be referred to a Speech-Language Pathologist (SLP).If a child scores below the cut-off for both motor-and language-based organization, he/she can be referred for both occupational and speech therapy and may need to be referred to other types of professionals.

Gender Differences.
Another way to examine the construct validity of a tool is to examine whether it distinguishes between populations.The QAPOA-P/T was found to distinguish between boys and girls, such that for both versions the cut-off score for boys for total organizational abilities as well as for motor-and language-based organizational abilities was lower than that for girls.The BRIEF-P [34] also has different cut-off points for boys and girls, and there too the cut-off score for boys is lower than for girls.In contrast, no gender differences were found in the My Child's Play Questionnaire [42], a parental questionnaire that examines play abilities among 3-9-year-old children, even within the subscale of executive functioning.
Convergent validity measures the extent to which the underlying constructs of an assessment tool are similar to those measured by established, validated instruments [43].In the current study, the convergent validity of the QAPOA-P/T was supported by moderate-significant correlations found with questions from the BRIEF-P that assess planning and organization, in both the parents' and kindergarten-teachers' versions.This finding provides evidence that the construct of organization, as examined in both tools, is similar.However, the degree to which the observation of behavior is representative of this construct differs between them.Specifically, the BRIEF-P is comprised of 63 statements, of which only 10 questions refer to organizational abilities.In contrast, the essence of the QAPOA-T/P is to examine organization abilities in a more comprehensive manner, such that all 48 items examine these abilities.For example, while only one item deals with organization abilities during play in BRIEF-P, QAPOA specifies 11 of such abilities as preferred game selection, story initiation, and respecting game rules.Likewise, nonitems in BRIEF-P concern organizational abilities during social communication, while QAPOA includes nine items/statements dealing with these abilities, like focusing on the topic of discussion, experience sharing, seeking help, referring to what was said by the other (see Tables 1 and 2 for the all the items).Since several areas in the QAPOA-T/P are not covered by the BRIEF-P, further validation should be considered with other subtests or assessment tools that examine organizational abilities (e.g., 30,31,37).

Conclusion
The findings of the current study support the reliability and validity of preschool teachers' and parents' versions of the QAPOA in assessing organizational abilities among Hebrewspeaking preschoolers.Data on reliability and validity should be taken into account before using QAPOA in other cultures.
As noted above, the process of developing the questionnaire raised the need for a new, more comprehensive definition of the construct of organization that relates to motor-and language-based organization abilities and to establishing cutoff scores according to gender.This questionnaire represents a more focused foundation upon which professionals can base their continued evaluation according to the various body functions.
This process informs us of the need for a more comprehensive top-down model for evaluating children's organizational abilities and should relate to the relevant abilities needed in their natural environment.Further assessment should then include the evaluation of body functions within the various therapeutic domains.Figure 1 illustrates such a model, which provides a structured and comprehensive guide that can serve to streamline the evaluation process of organizational abilities among children.The use of the approach delineated in this model can enable the creation of a more personalized intervention program for children with organizational disabilities.