A prosthetic limb control system to operate powered 2-DOF wrist and 1-DOF hand with environmental information, myoelectric signal, and forearm posture signal is composed and evaluated. Our concept model on fusing biosignal and environmental information for easier manipulation with upper limb prosthesis is assembled utilizing networking software and prosthetic component interlink platform. The target is to enhance the controllability of the powered wrist’s orientation by processing the information to derive the joint movement in a physiologically appropriate manner. We applied a manipulative skill model of prehension which is constrained by forearm properties, grasping object properties, and task. The myoelectric and forearm posture sensor signals were combined with the work plane posture and the operation mode for grasping object properties. To verify the reduction of the operational load with the proposed method, we conducted 2 performance tests: system performance test to identify the powered 2-DOF wrist’s tracking performance and user operation tests. From the system performance experiment, the fusion control was confirmed to be sufficient to control the wrist joint with respect to the work plane posture. Forearm posture angle ranges were reduced when the prosthesis was operated companying environmental information in the user operation tests.
Upper limb prostheses are widely accepted by amputees to support their daily life, but their mechanical functions are limited. Hence, upper limb prosthesis users are required to manage the burdens of daily activities. For example, widely used transradial prostheses do not have wrist joint functions. The users need to compensate the orientation of the hand with shoulder and trunk movements for grasping and manipulation tasks [
The target of our research is to contribute to this problem by engineering approaches. We proposed a multimodal sensor control method that combines information of upper limb posture and myoelectric sensor output to operate wrist function of a powered prosthesis [
When the information of the executing task is available on beforehand, we can select the operation mode or prehensile type, as proposed by Trachtenberg et al. [
The system composed in this project consists of networking software for robotic control software components and an interlinking hardware platform for prosthetic components.
The posturing of wrist joints during prehension mainly depends on the forearm posture, the grasping object posture, and the task. In addition to the myoelectric signal and forearm posture, we use the work plane posture and the operation mode as environmental information. To compose this structure, surface electrodes are used for myoelectric signal measurement, inertial sensor for forearm posture measurement, acceleration sensor for work plane posture measurement, and RFID for selection of object information. Prosthetic hand elements with actuator, motor driver, sensor, and main controller are modularized as prosthetic device elements (right in Figure
System structure of RTM-PDCP linkage platform.
We use RT-Middleware [
A number of multifunctional and multi-DOF upper limb prostheses with multiple active joints are developed and available [
Hardware of system and sensor coordinate system.
Tag_Manager sets information such as the priority and hand posture for RFID tag (left in Figure
Two electrodes were used, and hand opening and closing are, respectively, operated with myoelectric signal from the flexor muscle and extensor muscle of the user. In this system, when the tag is read by the wearable RFID reader, and the myoelectric signal of the flexor muscle exceeds the threshold value, the wrist joint servocontrol traces the work plane posture angle, and when the myoelectric signal of the extended muscle exceeds the threshold value, the tracing of the wrist joint control is canceled. In the track mode of the wrist joint, two kinds of modes were prepared: a mode that keeps the hand horizontal relative to the work plane and a mode that keeps the hand perpendicular to the work plane (Figure Horizontal Mode (Supination and flexion directions are positive.): Pronation/supination angle = −(work plane roll angle – forearm roll angle) [deg.]. Flexion/Extension angle = 0 [deg.]. Vertical Mode: Pronation/supination angle = −(90 + work plane roll angle – forearm roll angle) [deg.]. Flexion/extension angle = − (−90 + work plane pitch angle − forearm pitch angle) [deg.].
Horizontal mode and vertical mode of the wrist joint tracking control.
Flowchart of hand open/close. “Thr” and “Thr_end” are threshold value to control of the prosthetic hand opening and closing.
Flowchart of the wrist action. When RFID tag information is received and the priority of the tag information is higher than the previous value, the hand posture is set based on the tag information. The priority is set with 2 bits, and the smaller the number, the higher the priority. After the wrist joint angles (pronation/supination and flexion/extension angles) are determined based on the tag information (
We conducted an experiment to investigate the performance of the system. In this experiment, the tracking performance of the wrist joint posture with respect to the work plane posture is identified. We assembled an angle representation device, which alters the work plane posture for the experiment (Figure
Angle representation device and marker positions.
In the experiment, an infrared reflective markers were attached to measure the movement of the angle representation device and the transradial prosthesis’s powered 2-DOF wrist joint angles with optical motion capture and motion analysis system (VENUS 3D, Nobbytech). The attachment positions of the markers are shown in Figure Horizontal mode: angle representation device tilts in the roll angle direction of the acceleration sensor to identify the dynamic characteristics of the pronation/supination joint of the wrist. Vertical mode: angle representation device presents the angle in the pitch angle direction of the acceleration sensor to identify the dynamic characteristics of the flexion/extension joint of the wrist.
To verify the reduction of operation burden, we conducted experiments to operate the powered 3-DOF transradial prosthesis in performance tests with multiple manipulating tasks. The performance was compared between the prosthesis controls: the proposed method with the environmental information fusion wrist orientation and conventional method with a locked wrist at neutral position. In both conditions, the hand was controlled with myoelectric signals on the extensor and flexor muscle in the forearm.
User operation test was conducted by operating the powered 2-DOF wrist with the proposed multimodal sensor control method (MM) and conventional neutral position locked wrist myoelectric hand control (NP). To evaluate the wrist orientation function, the tests were specialized to adjust the wrist joint when reaching to the selected target object. There were three user operation test experiments: (I) a task that travels the hand back and forth to pick and release single shape object in resembling position and orientation (Box and Blocks Test, BBT); (II) a task that adjusts hand positioning, orientating, and closing force to pick-carry-release the target object in two pickup wrist posture conditions (Clothespin Test, CPT); and (III) a task that adjusts the hand orientation to correspond to a thin plate target object randomly posed tilt angles (Random Angle Test, RAT).
All experiments were conducted by nonamputee subjects donning a 3-DOF transradial prosthesis in Figure
Powered 2-DOF wrist and hand with adapter socket for nonamputee subjects.
The 3-DOF transradial prosthesis was attached to the right forearm of the subject with a offset position and angle to the user’s hand as shown in Figure
Experimental equipment of Box and Blocks Test.
Experimental equipment of Clothespin Test.
For BBT and CPT, the acceleration sensor for measuring the attitude of the work plane was installed on the work table.
Experimental equipment of Random Angle Test.
The presented angle and wrist angle were computed from the position data of the marker position measured and recorded through the experiment. The experimental environment, representation angles, and wrist angles relation in time series are presented in Figures
Results and experiment environment of flexion/extension joint.
Results and experiment environment of supination/pronation joint.
Using the calculated data of 5143 sets in flexion/extension joint direction and 5024 sets in supination/pronation joint direction, the dynamics of the 2-DOF wrist unit with the transfer function of the first-order lag system, using the MATLAB 2015a System Identification Toolbox, were identified. The identified transfer function is as follows:
User operation tests were conducted and recorded of 56 trials (4 subjects
The means and standard deviations of operation time are shown in Table
Means of operation times and scores (
Wrist joint operation | Neutral position | Multimodal |
---|---|---|
Box and Blocks Test score | 8.25 ± SD 0.96 | 8.75 ± SD 0.50 |
Clothespin Test operation time [s] | 25.59 ± SD 4.61 | 46.86 ± SD 18.93 |
Random Angle Test (10 deg.) operation time [s] |
4.29 ± SD 0.69 | 6.36 ± SD 1.49 |
Random Angle Test (20 deg.) operation time [s] | 5.55 ± SD 2.30 | 7.34 ± SD 2.11 |
Random Angle Test (30 deg.) operation time [s] |
4.48 ± SD 1.16 | 6.28 ± SD 1.72 |
Random Angle Test (40 deg.) operation time [s] | 7.56 ± SD 2.77 | 7.12 ± SD 2.83 |
Random Angle Test (50 deg.) operation time [s] | 13.64 ± SD 8.41 | 6.74 ± SD 0.49 |
Radar chart of the means and significant difference (
In the results of CPT, RAT (10 deg.), RAT (20 deg.), and RAT (30 deg.), the means of operation time increased when the proposal method was used to control the wrist.
In RAT (10 deg.), the mean is 4.29 ± SD 0.69 s when operated with fixed wrist joint at neutral position and 6.36 ± SD 1.49 s by the proposed control method. The operation time was longer in the proposed control method and there was statistical difference between the two. Similar tendency was confirmed at RAT (30 deg.), where the mean was 4.48 ± SD 1.16 s for the fixed wrist joint and 6.28 ± SD 1.72 s with the proposed control method. In RAT (50 deg.), the mean of operation time was reduced when proposed control method was applied.
The means and standard deviations of ROR (Range of Roll) are shown in Table
Mean of Range of Roll (
Wrist joint operation | Neutral position [deg.] | Multimodal [deg.] |
---|---|---|
Box and Blocks Test |
53.82 ± SD 7.36 | 28.61 ± SD 4.59 |
Clothespin Test | 83.73 ± SD 5.04 | 51.55 ± SD 19.68 |
Random Angle Test (10 deg.) | 42.22 ± SD 3.94 | 24.18 ± SD 11.46 |
Random Angle Test (20 deg.) |
41.09 ± SD 6.28 | 26.64 ± SD 8.50 |
Random Angle Test (30 deg.) |
39.70 ± SD 7.63 | 22.91 ± SD 7.54 |
Random Angle Test (40 deg.) | 36.76 ± SD 7.98 | 22.15 ± SD 11.36 |
Random Angle Test (50 deg.) |
36.90 ± SD 5.61 | 21.12 ± SD 5.01 |
Radar chart of the means and significant difference (
The means of ROR were reduced in all tasks when proposed control method is applied. There were statistical differences in BBT, RAT (20 deg.), RAT (30 deg.), and RAT (50 deg.).
The means and standard deviations of ROP (Range of Pitch) are shown in Table
Mean of Range of Pitch (
Wrist joint operation | Neutral Position [deg.] | Multimodal [deg.] |
---|---|---|
Box and Blocks Test | 34.00 ± SD 4.05 | 34.15 ± SD 8.06 |
Clothespin Test | 56.89 ± SD 5.23 | 53.58 ± SD 4.32 |
Random Angle Test (10 deg.) | 46.39 ± SD 12.40 | 46.23 ± SD 4.17 |
Random Angle Test (20 deg.) | 47.81 ± SD 12.91 | 45.99 ± SD 7.88 |
Random Angle Test (30 deg.) | 46.71 ± SD 10.08 | 46.79 ± SD 4.83 |
Random Angle Test (40 deg.) | 52.60 ± SD 11.68 | 43.00 ± SD 7.32 |
Random Angle Test (50 deg.) |
61.10 ± SD 12.82 | 40.34 ± SD 8.59 |
Radar chart of the means and significant difference (
The cutoff frequency was calculated from the approximated transfer function, flexion/extension joint direction was 0.86 Hz, and supination/pronation joint direction was 0.82 Hz.
The angular velocities were calculated from wrist angle during the identification experiment. The angular velocities are shown in Figure
Maximum value of powered 2-DOF wrist angular velocity.
Angular velocity [deg./s] | |
---|---|
Flexion | 423.0 |
Extension | 310.9 |
Pronation | 346.8 |
Supination | 358.7 |
Angular velocity of powered 2-DOF wrist.
By confirming the movement of the user during each test conditions when using the proposed control method, the necessity to approach the RFID tag and read the tag repeatedly is found to be the major cause of operation time to extend, such in Clothespin Test. However, when the control of the wrist joint using the environmental information is effective in easing to grasp the object, it is inferred that the operation time is shorter compared to the grasp time with the fixed wrist, especially with deeply tilted target such as at 50 degrees of the Random Angle Test.
An example of the roll angle during operation is shown in Figure
Mean and average reduction rate of Range of Roll (
Wrist joint operation | Neutral position |
Multimodal |
Reduction Rate |
---|---|---|---|
Box and Blocks Test |
53.82 ± SD 7.36 | 28.61 ± SD 4.59 | 46.8 |
Clothespin Test | 83.73 ± SD 5.04 | 51.55 ± SD 19.68 | 38.4 |
Random Angle Test (10 deg.) | 42.22 ± SD 3.94 | 24.18 ± SD 11.46 | 42.7 |
Random Angle Test (20 deg.) |
41.09 ± SD 6.28 | 26.64 ± SD 8.50 | 35.2 |
Random Angle Test (30 deg.) |
39.70 ± SD 7.63 | 22.91 ± SD 7.54 | 42.3 |
Random Angle Test (40 deg.) | 36.76 ± SD 7.98 | 22.15 ± SD 11.36 | 39.7 |
Random Angle Test (50 deg.) |
36.90 ± SD 5.61 | 21.12 ± SD 5.01 | 42.8 |
Example of the posture angle during operation.
In this project, we proposed a concept model that comprehensively supports the operation of the 2-DOF wrist and 1-DOF hand by fusion of myoelectric signals and environmental information by networking the prosthesis elements with a RT-Middleware and PDCP platforms.
To verify the operation load reduction with the proposed platform and control method, we conducted performance test identification experiments of a powered 2-DOF wrist tracking performance and user operation tests. From the joint tracking performance identification experiment, it was confirmed that the ability to sufficiently control the wrist joint with respect to the multiple conditions of presented work plane posture is encouraging. To verify the load reduction, we conducted user operation test with nonamputee subject to operate the powered prosthetic wrist and hand in the proposed environmental information fusion multimodal sensor control method and compared to the neutral position locked wrist hand control. The means of Range of Roll angle on condition of proposal control method were reduced in all tasks: Box and Blocks Test, Clothespin Test, and Random Angle Tests. From these findings, the proposed multimodal sensor control with environmental information fusion and the network platform are capable of reducing compensatory motion and operation burden in routine tasks with repeated reaching and hand orientation under well-established environment such as in home and office workspace. The guarantee of the effect of the proposed method is limited to transferring postural information from the sensor in the environment when triggered by the RFID tag reading. Additional sensors (i.e., force, tactile, and distance) or RFID tags can be added to the environment and information can be transmitted through the network. The drawback will be decreasing responding speed and failing to link the prosthesis control strategy to appropriate natural human behavior (i.e., response speed tolerance and other human interface design factors) will cause complexity and low reliability which will link to rejection. Therefore, therapeutic training methodology and screening methodology of the target user should be discussed with experienced therapist and prosthetist during assessment trial phase.
The authors declare that there are no conflicts of interest regarding the publication of this paper.