Evaluation process of a switch-based interaction technique (SIT) requires an interdisciplinary team effort and takes a considerable amount of time. Collecting subjective evaluation data from the users is a very common approach, but the subjective evaluation data alone might be manipulated and unreliable for comparing performances in many cases. Thus, therapists generally cannot succeed in determining the optimum SIT setup (i.e., determining the most appropriate combination of setup variables such as the switch type or switch site) at first attempts since it is hard to evaluate the measurable performance by collecting subjective data instead of objective data. Inevitably, each unsuccessful attempt to reach the optimum SIT setup results in a loss of serious time and effort. On the contrary, a benchmark application is also required to make performance evaluation of SITs by using a number of standard tests and empirical attributes. It is obvious that a quicker and more accurate SIT evaluation process provides a better cost and schedule management considering the increasing number of SIT users in the world. Therefore, we propose a novel benchmark for performance evaluation called SITbench that provides a quicker and more accurate switch evaluation process by collecting and saving the objective data automatically. We conducted a user study with eight participants and demonstrated that the objective data collected via the SITbench helped to determine the optimum SIT setup accurately. Result of a questionnaire applied to evaluate the SITbench itself was also satisfactory. SITbench is expected to help researchers and therapists to make a better evaluation according to any change done in SIT setup variables (switch type, activation method, etc.) with the aim of reaching the optimum SIT setup, which leads to a better cost and schedule management. As the first benchmark application compatible with all SITs, which can emulate keyboard characters or mouse clicks, it can be utilized by assistive technology professionals to make comparisons and evaluations automatically via standardized tests.
There have been many people with motor disabilities worldwide [
It is obvious that an efficient evaluation process of a SIT helps therapists to determine the optimum SIT setup for motor-impaired people, but there are many variables in a SIT setup such as switch type, switch site, users’ posture, and activation method. For example, even a button switch can be used in several ways: it can be activated by the hand or any other body part. Likewise, users can be positioned in different postures during switch usage, which might affect the performance dramatically. The main aim of a SIT evaluation is to determine the optimum SIT setup which is the most suitable combination of these variables for the users to interact with their environment. To this end, a considerable time and effort is needed by an interdisciplinary team that includes many trials with different variables of SIT setup. On the contrary, assistive technology professionals require a benchmark application [
Currently, SIT evaluation is performed in three ways: (a) collecting subjective data [ Incompatibility: switch-accessible applications might require different keyboard characters or mouse clicks from switches to work. Furthermore, each switch might emulate and send different keyboard characters or mouse clicks depending on its manufacturer. Unfortunately, commonly agreed standard is not available. For example, while some switch-accessible applications might expect to receive a keyboard space character, other applications might expect to receive a mouse right-click. Both applications expect to receive a mouse left-click to work. In other words, they are only compatible with switches which are able to emulate mouse left-click. The remaining switches are excluded, which means that just a minority of SITs are compatible and could be evaluated with these applications. Therefore, we consider that they are far from being a proper benchmark for SIT evaluation. Our novel tool SITbench is compatible with all switches, which can emulate any mouse clicks or keyboard characters, since it allows therapists to assign the expected characters from any switch. Limited number of switches: they are only capable to evaluate single-switch systems. Double-switch support is also required since double-switch usage is widely used as an alternative interaction method. SITbench is capable to allow both single- and double-switch evaluation. Limited number of tests: both applications employ only one test that measures press time (i.e., the time from the prompt to when the switch is pressed) and release time (i.e., the time from when the switch is pressed until it is released) of a switch. SITbench includes two more additional tests to evaluate SITs with the single and double switch. Database requirement: they have some reporting functions for the test results. However, we considered that a well-structured database would be useful to share the results and apply some queries or statistical tests. In addition to the reporting function, SITbench also allows to save the test results automatically into a Microsoft Access database. Sufficient attention span requirement: sufficient attention span via both applications might not be achieved especially by infants since they can become distracted and lose their attention easily during long and boring sessions. We applied gamification techniques while designing SITbench tests with the intent to make evaluations more engaging and fun.
Therefore, we propose a novel SIT evaluation tool, namely, SITbench as a benchmark application which helps to determine the optimum SIT setup with the aim of providing a quicker and more accurate SIT evaluation process. To collect the objective data, SITbench includes three different games which can be played via the single or double switch. It measures and saves the performance metrics (accuracy, precision, recall, and false-positive rate) automatically at the end of each trial.
A user study with eight participants was conducted as a part of this work in order to test and demonstrate the proposed benchmark application. We identified two different switch sites to be tested by users under the same conditions in order to determine the most suitable switch site. To this end, we collected the objective data via SITbench. Results revealed that SITbench could help to determine the optimum switch setup accurately. We also applied a System Usability Scale (SUS) [
More potential SIT users can be served at the same time period with the same workforce since a quicker and more accurate SIT evaluation process is provided by the SITbench, which might prevent governments to spend high amounts of money as a result of better cost and schedule management. As a benchmark application, it allows to make objective comparisons with standardized tests under the same conditions by collecting the performance data of SITs for assistive technology community automatically. Thus, it provides extratime for therapists to observe more subjective aspects of client needs. On the contrary, it might be used to evaluate fine-motor skills of clients as a clinical tool. Occupational therapists can track the patients’ progress by SITbench that allows to measure and record clients’ fine-motor performance and reflexes automatically in the form of quantitative objective data. The SITbench might also help to improve the contingency awareness of the ones with profound and multiple learning disabilities, or it might be useful for pupils with severe learning difficulties to assess their auditory and visual attention.
This paper begins with the section that presents the design and implementation of our novel switch evaluation tool. Then, in Evaluation, we share the objective results of our user study and the questionnaire results of the SITbench. Finally, we conclude our study and discuss our future work in Conclusion.
SITbench is designed as a novel benchmark application for assistive technology and healthcare professionals to determine the most appropriate SIT setup. It helps to collect and save the objective data automatically with the aim of the optimum SIT setup. To this end, three different switch-accessible games, depending on the single or double switch, were designed within SITbench, namely,
Welcoming screen of SITbench.
As can be seen in Figure
Expected key assignment module.
TSMG is a single switch-accessible game based on indirect selection with the automatic linear scanning method. As it is exemplified in Figure
Time-state model of an automatic linear scanning sample.
There are five different templates which could be tested via TSMG. Figure
Initial form of TSMG in template 1.
TSMG is based on automatic linear scanning where each smiley is highlighted for a given time period (i.e., scan time) one-by-one. User should activate the switch once the highlighted smiley is a red one. User also hears a click sound as an auditory prompt, as soon as the target is highlighted. When the switch is activated, it sends the expected key to SITbench as a selection signal. Once the expected key is received, SITbench gives a sensory feedback by swapping the background color of the interface like a blink. In the expert mode (Figure
Initial form of TSMG in the expert mode.
User aims to match each smiley with a tie in a way that smiley and its tie are in the same color (e.g., red smileys with red ties). To this end, the user should select all red smileys but yellow ones via a switch. A sample view of results after the user completed a trial without any mistake can be seen in Figure
A view of TSMG in the end of trial following a user performance without any mistake.
At the end of each trial, confusion matrix variables (true positives (TP), false positives (FP), false negatives (FN), and true negatives (TN)) are calculated and assigned automatically, as can be seen in Figure
A general view from TSMG in the end of trial following a user performance with several mistakes (i.e., with false negatives and false positives).
SITbench allows therapists to save the results and all data into a structured database, and it has also a reporting function to print or save the results as a document.
NDG can be played with the single or double switch: (1) a single switch-accessible version based on indirect selection (SS-NDG); (2) a double switch-accessible version based on direct selection (DS-NDG). Figure
The initial form of SS-NDG where 1 shows the left signal (i.e., orange square box), 2 is the right signal; 3 represents the green car, and 4 is the finish line.
The initial form of DS-NDG in track 2.
A view of DS-NDG in the end of trial where the user reached the finish line in track 3 without any crash.
HFG is a single switch-accessible application to measure user’s switch performance. At the beginning of each trial, therapists can select the scenario and enter user details. Each trial of game consists of ten tasks. As it is illustrated in Figure
All three frames shown to user during a task: (a) frame shown until a fly is appeared; (b) frame shown until the user activates the switch; (c) frame shown once the user activates the switch.
A view of HFG in the end of a trial.
We conducted a user study as a demonstration of SITbench. We identified two different switch sites (Figure
Positions of forefinger during experiments according to two switch sites FDP (represented by
In this section, firstly we introduce the participants. Then, we present the apparatus used and the procedure applied. At last, we share the experimental findings.
Eight able-bodied participants (mean age = 30.2, standard deviation = 3.1), including four females and four males, took part in this study. Just two of the participants were familiar with switch-accessible applications before experiments.
A laptop (model: Lenovo G505S; CPU: AMD A8-4500M 1.9 GHz; RAM: 6 GB DDR3; screen: LCD 15.6; OS: Windows 10 64 bits; resolution: 1600 × 900) was employed within this study for experiments.
At the beginning, the participant is positioned in front of a laptop in a way that the participant is able to access laptop’s keyboard easily.
Participants were informed about the SITbench and tests, and then they practised SITbench in the counterbalanced order until they become ready for tests. This practicing step took 20 minutes approximately for each participant. Following positioning and practicing steps, three tests were applied to participants to collect objective performance data: TSMG: each switch site (FDP and FPIJ) was tested by each participant ( SS-NDG: each switch site (FDP and FPIJ) was tested by each participant ( HFG: each switch site (FDP and FPIJ) was tested by each participant (
All tests were applied in the counterbalanced order to avoid learning and repetition effects. The participants were also allowed to rest (1 to 5 minutes) during experiments to prevent excessive mental or physical fatigue.
At the end of experiments, a questionnaire based on SUS [
Modified statements of the SUS questionnaire with average scale values of all participants.
Statements | Scale |
---|---|
(1) I would use SITbench for SIT evaluation tasks frequently | 4.00 |
(2) I found SITbench unnecessarily complex | 1.37 |
(3) I found SITbench easy to use | 4.12 |
(4) I would need the support of a technical person to be able to use SITbench | 1.75 |
(5) I found the various functions in SITbench were well integrated | 4.37 |
(6) I thought there was too much inconsistency in SITbench | 1.37 |
(7) I would imagine that most people would learn to use SITbench very quickly | 4.25 |
(8) I found SITbench very cumbersome/awkward to use | 1.50 |
(9) I felt very confident using SITbench | 4.12 |
(10) I need to learn a lot of things before I can use this system | 1.25 |
Results of the SUS questionnaire are listed in Table
FDP as a switch site showed quite impressive performance in comparison with FPIJ in all three tests (TMSG, SS-NDG, and HFG) as it is expected at the beginning. It is demonstrated that SITbench succeeded to determine the most appropriate switch site as FDP.
According to results of TSMG (Figure
Mean values and standard deviations of two switch sites for all the participants through evaluation metrics of TSMG (accuracy, precision, recall, and false-positive rate) (
SS-NDG results (Figure
Mean values and standard deviations of two switch sites for all the participants according to evaluation metrics of SS-NGD as (a) completion time and (b) crash count (
Lastly, HFG results (Figure
Mean values and standard deviations of two switch sites for all the participants through evaluation metrics of HFG (average press time, average release time, the fastest press time, the slowest press time, the fastest release time, and the slowest release time) (
On the contrary, we applied
Evaluation process is one of the most important tasks in order to reach the optimum SIT setup. Because the optimum SIT setup plays a vital role for people with motor disabilities to interact with their environment, any tool to achieve the optimum SIT setup for having a better cost and schedule management becomes a very important requirement considering the increasing number of SIT users. Determining the optimum switch setup by collecting the subjective data might be challenging since the subjective data alone might be unreliable and manipulated easily for performance evaluation. Therapists might have to reapply questionnaires and make new observations several times. A serious time and effort is needed for these repeated trials to collect subjective data. Therefore, subjective data collection instead of objective data does not seem a proper method for performance evaluation of a SIT. On the contrary, current evaluation methods based on collecting objective data in literature are far from being a benchmark. These methods are generally employed to evaluate just a specific SIT. In other words, they are not designed to evaluate the other SITs, which make them ineligible to be a benchmark. To the best of our knowledge, there have been just two applications [
We have conducted a user study as a demonstration with eight participants to evaluate the usage of different switch sites. To this end, objective data were collected via SITbench. FDP performed better performance than FPIJ in all tests as it is expected. Findings demonstrated that SITbench is capable to determine the most proper switch site with the aim of an optimum SIT setup. Result of a SUS questionnaire to evaluate the SITbench itself was also quite satisfactory.
A quicker and more accurate SIT evaluation via SITbench helps to serve more potential SIT users at the same time period with the same workforce. As a result of better cost and schedule management, SITbench might prevent governments from unnecessary expenses and human-resource allocations, but future studies with SITbench are required to verify that SITbench is capable to do this. On the contrary, it might be also employed by therapists and assistive technology professionals to measure the fine-motor skills and reflexes of users as a clinical tool. They can track the progress of user’s skill via SITbench since it is capable to measure and save the performance automatically as a quantitative objective data. SITbench can also be utilized to improve the contingency awareness of the ones with profound and multiple learning disabilities. Besides, it might be employed as a tool to assess auditory and visual attention of people with severe learning difficulties.
In order to improve the SITbench and overcome some of its limitations, some future studies would be quite useful. We intend to include new tests depending on several scanning methods. So as to test the efficiency of SITbench better, we aim to extend the participant group with motor-impaired people. Since the SITbench is currently compatible with only desktop computers, it might be modified to be compatible with mobile systems such as smartphones and tablets to extend the target group. We also aim to include some tests such as a speller to evaluate users’ computer access activities. Employing a group of therapists and assistive technology professionals to evaluate and demonstrate SITbench would be also quite useful.
The data used to support the findings of this study are available from the corresponding author upon request.
The authors declare that they have no conflicts of interest regarding the publication of this article.
The first author was funded by the Ministry of National Education Scholarship of Turkish Republic. We acknowledge support by the German Research Foundation and the Open Access Publication Fund of TU Berlin. We also would like to thank Brijnesh Jain and Fikret Sivrikaya for their valuable suggestions.