Networked Multimodal Sensor Control of Powered 2-DOF Wrist and Hand

A prosthetic limb control system to operate powered 2-DOF wrist and 1-DOF hand with environmental information, myoelectric signal, and forearm posture signal is composed and evaluated. Our concept model on fusing biosignal and environmental information for easier manipulation with upper limb prosthesis is assembled utilizing networking software and prosthetic component interlink platform. The target is to enhance the controllability of the powered wrist’s orientation by processing the information to derive the joint movement in a physiologically appropriate manner. We applied a manipulative skill model of prehension which is constrained by forearm properties, grasping object properties, and task. The myoelectric and forearm posture sensor signals were combined with the work plane posture and the operation mode for grasping object properties. To verify the reduction of the operational loadwith the proposedmethod, we conducted 2 performance tests: system performance test to identify the powered 2-DOF wrist’s tracking performance and user operation tests. From the system performance experiment, the fusion control was confirmed to be sufficient to control the wrist joint with respect to the work plane posture. Forearm posture angle ranges were reduced when the prosthesis was operated companying environmental information in the user operation tests.


Introduction
Upper limb prostheses are widely accepted by amputees to support their daily life, but their mechanical functions are limited.Hence, upper limb prosthesis users are required to manage the burdens of daily activities.For example, widely used transradial prostheses do not have wrist joint functions.The users need to compensate the orientation of the hand with shoulder and trunk movements for grasping and manipulation tasks [1][2][3].One solution is to increase the number of joints of the prosthetic hand.By incorporating multiarticulated prosthetic hand with number of movable joints, various digit movements are expected to contribute to reducing the compensatory movements.However, operating multiple joints is not an easy task for the upper limb prosthesis user.The multijoint operation requires multiple independent channels of operation input signals or pattern recognition algorithm to discriminate and coordinate the joint movements.Myoelectric signal has been a choice and numbers of myoelectric control system are proposed and available for upper limb prosthesis.However, the number of available myoelectric signal source channels is limited, due to socket design and/or user's arm conditions.To overcome this problem, methods to sequentially select operation functions are widely used in upper limb prosthetic products.These methods are useful in simple and short tasks.However, as the number of functions increases, more time is needed to select the intended function [4,5].Cocontraction switching is commonly used for switching between functions or prehensile types in multiarticulated hand systems or powered transhumeral prostheses.There are other methods to increase the number of myoelectric signal source channels through surgical reconstruction, such as Targeted Muscle Reinnervation [6]; however, the availability of this method is still limited.
The target of our research is to contribute to this problem by engineering approaches.We proposed a multimodal sensor control method that combines information of upper limb posture and myoelectric sensor output to operate wrist function of a powered prosthesis [7].Our aim is to extend this approach.Generally, selection of manipulative activities is constrained by the properties of the tasks and other surrounding conditions, such as the size and shape of the objects or target areas [8].Therefore, our proposal is to use the information of the surrounding environment for dexterous operation of the joints and the motions of the upper limb prosthesis.
When the information of the executing task is available on beforehand, we can select the operation mode or prehensile type, as proposed by Trachtenberg et al. [9].However, it is cumbersome to get all the required information for each task of daily life, and select the priority of the candidate tasks in advance.We believe installing identification tags and sensors to utilize environmental information is a more practical solution than extraction the information from the myoelectric signals for prosthetic operation in routine daily life activities.To implement this idea, three software technologies are required, first is for connecting environment objects, second is for communication of prosthetic components, and the third is for connecting those environment and prosthetic elements.There are several researches for connecting environment object.In this research, we applied a software platform for robot development [10] used in the development of robotic life space environment [11].For the communication between prosthetic components, a Prosthetic Device Communication Protocol (PDCP) [12] is implemented in this work.Software component were developed to connect those two software platforms [13].In this project, we composed and evaluated a system to operate powered 2-DOF wrist and a 1-DOF hand on these platforms.To verify the effect, the reduction of user's operational load was evaluated by conducting performance tests with nonamputated subjects.

System Overview.
The system composed in this project consists of networking software for robotic control software components and an interlinking hardware platform for prosthetic components.
The posturing of wrist joints during prehension mainly depends on the forearm posture, the grasping object posture, and the task.In addition to the myoelectric signal and forearm posture, we use the work plane posture and the operation mode as environmental information.To compose this structure, surface electrodes are used for myoelectric signal measurement, inertial sensor for forearm posture measurement, acceleration sensor for work plane posture measurement, and RFID for selection of object information.Prosthetic hand elements with actuator, motor driver, sensor, and main controller are modularized as prosthetic device elements (right in Figure 1) to form a prosthetic limb network.Acceleration sensor and wearable RFID reader are linked to a software component as the environmental information networks (left in Figure 1).The prosthetic limb network and the environmental information network are linked for communicating between the prosthetic and environment and consist a RTM-PDCP linkage platform (center of Figure 1).

Networking Platform.
We use RT-Middleware [10] for networking the environment.RT-Middleware was developed by a group at National Institute of Advanced Industrial Science and Technology (AIST), Japan, as a software platform for efficient development of robot system.We used Prosthetic Device Communication Protocol (PDCP) [12] for networking the sensors, motors, and microprocessors in the prosthesis.The PDCP is a communication protocol for prosthesis that was developed mainly by a group at the University of New Brunswick, Canada.The PDCP is designed to modulate the prosthetic elements [14] to satisfy the requirements of each prosthesis user.In the previous study, a RT-Middleware software component, PDCP Wrapper [13], was developed to interpret the communication between RT-Middleware and PDCP.The PDCP Wrapper connects the environment network composed by RT-Middleware and the prosthetic network by PDCP and interpret the information for communication.

Networking of Hardware.
A number of multifunctional and multi-DOF upper limb prostheses with multiple active joints are developed and available [15,16].Montagnani et al. [17] suggested that "a complex (and expensive) multigrasp hand cannot be fully exploited if fitted on a simple rotator; a much cheaper 1-DOF hand on a slightly more complex wrist could achieve a similar performance."Therefore, in this system, we composed a 3-DOF transradial prosthesis (center of Figure 2).The former 3-DOF transradial prosthesis consisted of a 2-finger and thumb hand driven with 1 geared DC motor, palmar/dorsal flexion with axis at the MP joint and a pronation/supination wrist each driven by a geared DC motor.) and connected to the PDCP-Bus using an adapter (CANUSB, LAWICEL).

Integration of Information.
Tag Manager sets information such as the priority and hand posture for RFID tag (left in Figure 1).The information is sent to PDCP Wrapper whenever the configured tag is read.The work plane posture and the upper limb posture are presented by Euler angles.The Euler angles are calculated from the rotation of the acceleration vector measured by acceleration sensor.First, the instantaneous direction vector of the acceleration is computed from the acceleration vector.Since the acceleration vector is composed of gravitational acceleration and inertia, the acceleration vector measured when the acceleration sensor is set horizontally is used to calibrate as a reference vector.Then, Euler angles representing the sensor's posture are calculated from the rotation from the reference vector to the unit direction vector.The work plane posture is calculated by Object Posture Manager and sent to PDCP Wrapper (left in Figure 1).Upper limb posture is calculated by Sensor Board and transmitted to the PDCP-Bus which is a CAN-BUS standard network (right in Figure 1).Myoelectric signals are AD converted by Sensor Board and transmitted to the PDCP-Bus.The Control Board receives the information and use it for operation.

Operation of Powered 2-DOF Wrist and
Hand.Two electrodes were used, and hand opening and closing are, respectively, operated with myoelectric signal from the flexor muscle and extensor muscle of the user.In this system, when the tag is read by the wearable RFID reader, and the myoelectric signal of the flexor muscle exceeds the threshold value, the wrist joint servocontrol traces the work plane posture angle, and when the myoelectric signal of the extended muscle exceeds the threshold value, the tracing of the wrist joint control is canceled.In the track mode of the wrist joint, two kinds of modes were prepared: a mode that keeps the hand horizontal relative to the work plane and a mode that keeps the hand perpendicular to the work plane (Figure 3).

System Performance Tests.
We conducted an experiment to investigate the performance of the system.In this experiment, the tracking performance of the wrist joint posture with respect to the work plane posture is identified.We assembled an angle representation device, which alters the work plane posture for the experiment (Figure 6).By entering the target angle to the controller component of the angle representation device, the attitude angle of the representation device is shifted with the servomotor and the acceleration sensor signal on the device is transmitted to the wrist joint angle controller through the PDCP Wrapper for the wrist joint to follow.
In the experiment, an infrared reflective markers were attached to measure the movement of the angle representation device and the transradial prosthesis's powered 2-DOF wrist joint angles with optical motion capture and motion analysis system (VENUS 3D, Nobbytech).The attachment positions of the markers are shown in Figure 6.The sampling frequency of the optical motion capture was 100 Hz.The experiments were conducted under the two conditions as follows: (I) Horizontal mode: angle representation device tilts in the roll angle direction of the acceleration sensor to identify the dynamic characteristics of the pronation/supination joint of the wrist.
(II) Vertical mode: angle representation device presents the angle in the pitch angle direction of the acceleration sensor to identify the dynamic characteristics of the flexion/extension joint of the wrist.

User Operation Tests.
To verify the reduction of operation burden, we conducted experiments to operate the powered 3-DOF transradial prosthesis in performance tests with multiple manipulating tasks.The performance was compared between the prosthesis controls: the proposed method with the environmental information fusion wrist orientation and conventional method with a locked wrist at neutral position.
In both conditions, the hand was controlled with myoelectric signals on the extensor and flexor muscle in the forearm.When RFID tag information is received and the priority of the tag information is higher than the previous value, the hand posture is set based on the tag information.The priority is set with 2 bits, and the smaller the number, the higher the priority.After the wrist joint angles (pronation/supination and flexion/extension angles) are determined based on the tag information (: roll angle of the hand; : pitch angle of the hand), the work plane posture, and the upper limb posture, the wrist joint starts action when Close Mode is set and it stops the operation of the wrist joint when Open Mode is set.
User operation test was conducted by operating the powered 2-DOF wrist with the proposed multimodal sensor control method (MM) and conventional neutral position locked wrist myoelectric hand control (NP).To evaluate the wrist orientation function, the tests were specialized to adjust the wrist joint when reaching to the selected target object.There were three user operation test experiments: (I) a task that travels the hand back and forth to pick and release single shape object in resembling position and orientation (Box and Blocks Test, BBT); (II) a task that adjusts hand positioning, orientating, and closing force to pick-carry-release the target object in two pickup wrist posture conditions (Clothespin Test, CPT); and (III) a task that adjusts the hand orientation to correspond to a thin plate target object randomly posed tilt angles (Random Angle Test, RAT).
All experiments were conducted by nonamputee subjects donning a 3-DOF transradial prosthesis in Figure 7.The operation time and forearm acceleration were measured for each trial.The operation time was measured with a stopwatch, and the acceleration was measured with an inertial sensor (MPU 6050, InvenSense) at a sampling frequency of 200 Hz by a microcomputer (NUCLEO-F767ZI, STMicroelectronics) wired to the sensor and installed on the socket.
The 3-DOF transradial prosthesis was attached to the right forearm of the subject with a offset position and angle to the user's hand as shown in Figure 7.The myoelectric sensors were attached to the body surface over the flexor and extensor muscles in the forearm, respectively.The experimental equipment for all tasks was set on the workbench with the table top height of 750 mm.The initial posture of the hand and wrist joint was set to the neutral position and the hand aperture was set to 50 mm.experimenter, the subject moves as many blocks as possible from the right side to the left side within 30 seconds operating the prosthesis.An RFID tag was set as shown in Figure 8 to change the wrist position to vertical mode at the beginning.

Tasks (I) Box and Blocks
(II) Clothespin Test.The test is to pick and move the clothespins from the horizontal bar in the low front to the three vertical bars, on the metal net in the back and release.With the hand placed at the initial position and confirming the start sign from the experimenter, the subject moves the three clothespins to the target vertical bars in the order from the left to right.A vertical mode tag was placed on the right front end of the case and a horizontal mode tag below the vertical bar as in the positions shown in Figure 9.
For BBT and CPT, the acceleration sensor for measuring the attitude of the work plane was installed on the work table.
(III) Random Angle Test.In daily life, the human hand orientation is operated by the wrist to adjust to the anonymous posture of the target object.To produce a similar condition for testing the hand orientation, a servomotor driven angle representation device randomly arranges five tilted angles of 10, 20, 30, 40, and 50 degrees toward the subject from the vertical upright posture.After each angle being presented, the subject grasps and picks up the clothespin attached to the angle representation device.As shown in Figure 10, the tag to invoke the vertical mode is installed to the right of the clothespin on the angle representation device.In the experiment, the motion is measured from the initial position till the prosthesis grips the clothespin and lifts it off the device.

System Performance Tests.
The presented angle and wrist angle were computed from the position data of the marker position measured and recorded through the experiment.The experimental environment, representation angles, and wrist angles relation in time series are presented in Figures 11 and  12.
Using the calculated data of 5143 sets in flexion/extension joint direction and 5024 sets in supination/pronation joint direction, the dynamics of the 2-DOF wrist unit with the transfer function of the first-order lag system, using the MAT-LAB 2015a System Identification Toolbox, were identified.The identified transfer function is as follows: ProSpi () = 6.087  + 5.134 .
(2)    time were compared between the fixed wrist joint at the neutral position condition and the condition with the wrist joint was operated by the proposed control method of RFID information is used to orientate the 2-DOF wrist and myoelectric control of the hand.The statistical significant difference was tested with t-test between the conditions.

Operation Time.
The means and standard deviations of operation time are shown in Table 1; a radar chart of the means and significant differences ( *  < 0.05,  = 4) is shown in Figure 13.
In the results of CPT, RAT (10 deg.), RAT (20 deg.), and RAT (30 deg.), the means of operation time increased when the proposal method was used to control the wrist.
In RAT (10 deg.), the mean is 4.29 ± SD 0.69 s when operated with fixed wrist joint at neutral position and 6.36 ± SD 1.49 s by the proposed control method.The operation time was longer in the proposed control method and there was statistical difference between the two.Similar tendency was confirmed at RAT (30 deg.), where the mean was 4.48 ± SD 1.16 s for the fixed wrist joint and 6.28 ± SD 1.72 s with the proposed control method.In RAT (50 deg.), the mean of operation time was reduced when proposed control method was applied.

Posture Angle Range during the Operation (Roll, Pitch).
The means and standard deviations of ROR (Range of Roll) are shown in Table 2; a radar chart of the means and significant difference ( *  < 0.05,  = 4) is shown in Figure 14.
The means of ROR were reduced in all tasks when proposed control method is applied.There were statistical differences in BBT, RAT (20 deg.), RAT (30 deg.), and RAT (50 deg.).
The means and standard deviations of ROP (Range of Pitch) are shown in Table 3; a radar chart of the means and significant difference ( *  < 0.05,  = 4) is shown in Figure 15.The means of ROP were reduced in CPT, RAT (10 deg.), RAT (20 deg.), RAT (40 deg.), and RAT (50 deg.) when proposed control method was applied.In RAT (50 deg.), the ROP was 61.10 ± SD 12.82 degrees when the wrist joint was fixed at neutral position and 40.34 ± SD 8.59 degrees when the proposed control method was applied.The ROP was smaller  in the proposed control method and statistical difference was confirmed in RAT (50 deg.).The angular velocities were calculated from wrist angle during the identification experiment.The angular velocities are shown in Figure 16 and the maximum value is shown in Table 4. Commercially available wrist rotator's (Electric Wrist Rotator, Ottobock) angular velocity is 81.28 deg./s [19], and, therefore, we assume that the tracking performance of our system is, in terms of cutoff frequency and the maximum angular velocity, acceptable for use in operation experiment level.Further tests are required to discuss the daily use level.

Operation Time.
By confirming the movement of the user during each test conditions when using the proposed control method, the necessity to approach the RFID tag and read the tag repeatedly is found to be the major cause of operation time to extend, such in Clothespin Test.However, when the control of the wrist joint using the environmental  information is effective in easing to grasp the object, it is inferred that the operation time is shorter compared to the grasp time with the fixed wrist, especially with deeply tilted target such as at 50 degrees of the Random Angle Test.

Posture Angle Range during the Operation (Roll, Pitch).
An example of the roll angle during operation is shown in Figure 17 and average reduction rates of the Range of Roll are shown in Table 5.The average reduction rate of the Range of Roll angle is 41.1% for all tasks.In the standing posture with shoulder joint in neutral position and elbow flexion at 90 degrees, the roll angle direction of the forearm posture calculated by acceleration sensor output roughly matches the abduction direction of the shoulder joint.From this result, the proposed control method can be said to reduce compensating motion of the prosthesis user's shoulder and the operation load.

Conclusion
In this project, we proposed a concept model that comprehensively supports the operation of the 2-DOF wrist and 1-DOF hand by fusion of myoelectric signals and environmental information by networking the prosthesis elements with a RT-Middleware and PDCP platforms.
To verify the operation load reduction with the proposed platform and control method, we conducted performance test identification experiments of a powered 2-DOF wrist tracking performance and user operation tests.From the joint tracking performance identification experiment, it was confirmed that the ability to sufficiently control the wrist joint with respect to the multiple conditions of presented work plane posture is encouraging.To verify the load reduction, we conducted user operation test with nonamputee subject to operate the powered prosthetic wrist and hand in the proposed environmental information fusion multimodal sensor control method and compared to the neutral position locked wrist hand control.The means of Range of Roll angle on condition of proposal control method were reduced in all tasks: Box and Blocks Test, Clothespin Test, and Random Angle Tests.From these findings, the proposed multimodal sensor control with environmental information fusion and the network platform are capable of reducing compensatory motion and operation burden in routine tasks with repeated reaching and hand orientation under well-established environment such as in home and office workspace.The guarantee of the effect of the proposed method is limited to transferring postural information from the sensor in the environment when triggered by the RFID tag reading.Additional sensors (i.e., force, tactile, and distance) or RFID tags can be added to the environment and information can be transmitted through the network.The drawback will be decreasing responding speed and failing to link the prosthesis control strategy to appropriate natural human behavior (i.e., response speed tolerance and other human interface design factors) will cause complexity and low reliability which will link to rejection.Therefore, therapeutic training methodology and screening methodology of the target user should be discussed with experienced therapist and prosthetist during assessment trial phase.

Figure 2 :
Figure 2: Hardware of system and sensor coordinate system.
The motors were driven by an H-bridge motor driver and controlled by a single chip MPU.The new mockup was composed of 2-DOF wrist and 1-DOF hand.The palmar/dorsal flexion and pronation/supination wrist consists of 2 servomotors (Standard Servo, Parallax).As a terminal device a 5-digit hand (YS-2002, Tokyo Denki University) is attached to the wrist.A geared DC motor drives the linked digits through a worm gear and universal joint.Surface electrode (13E200, Ottobock), inertial sensor (MPU6050, InvenSense), and powered 3-DOF transradial prosthesis (TDU) were networked by microcomputers (dspic33F, Microchip) and the development boards (Microstick II and MicrostickPlus ver.C of Microchip and Microstick IO Board of AIST/WIN Electronic Industries).Software programs for PDCP-base communicating are those released from UNB. Acceleration sensor (ADXL330, AnalogDevices) for work plane measurement (left in Figure 2) and wearable RFID tag reader (TECCO, GOV), right in Figure 2, were networked [18] by RT-Middleware.PDCP Wrapper was installed on the laptop (Latitude3340, Dell, OS: Microsoft Windows 7 Flowchart of a hand open/close is shown in Figure 4 and flowchart of wrist action is shown in Figure 5.The wrist joint angles setting in each mode are shown in the following function: (i) Horizontal Mode (Supination and flexion directions are positive.):Pronation/supination angle = −(work plane roll angle -forearm roll angle) [deg.].Flexion/Extension angle = 0 [deg.].(ii) Vertical Mode: Pronation/supination angle = −(90 + work plane roll angle -forearm roll angle) [deg.].

Figure 3 :Figure 4 :
Figure 3: Horizontal mode and vertical mode of the wrist joint tracking control.

Figure 5 :
Figure5: Flowchart of the wrist action.When RFID tag information is received and the priority of the tag information is higher than the previous value, the hand posture is set based on the tag information.The priority is set with 2 bits, and the smaller the number, the higher the priority.After the wrist joint angles (pronation/supination and flexion/extension angles) are determined based on the tag information (: roll angle of the hand; : pitch angle of the hand), the work plane posture, and the upper limb posture, the wrist joint starts action when Close Mode is set and it stops the operation of the wrist joint when Open Mode is set.

Figure 8 :Figure 9 :
Figure 8: Experimental equipment of Box and Blocks Test.

Figure 10 :
Figure 10: Experimental equipment of Random Angle Test.Ext/Flx

Figure 11 :Figure 12 :
Figure 11: Results and experiment environment of flexion/extension joint.Pro/Sup

Figure 17 :
Figure 17: Example of the posture angle during operation.
*Time (s) and BBT score.

Table 4 :
Maximum value of powered 2-DOF wrist angular velocity.