Optical Enhancement of Exoskeleton-Based Estimation of Glenohumeral Angles

In Robot-Assisted Rehabilitation (RAR) the accurate estimation of the patient limb joint angles is critical for assessing therapy efficacy. In RAR, the use of classic motion capture systems (MOCAPs) (e.g., optical and electromagnetic) to estimate the Glenohumeral (GH) joint angles is hindered by the exoskeleton body, which causes occlusions and magnetic disturbances. Moreover, the exoskeleton posture does not accurately reflect limb posture, as their kinematic models differ. To address the said limitations in posture estimation, we propose installing the cameras of an optical marker-based MOCAP in the rehabilitation exoskeleton. Then, the GH joint angles are estimated by combining the estimated marker poses and exoskeleton Forward Kinematics. Such hybrid system prevents problems related to marker occlusions, reduced camera detection volume, and imprecise joint angle estimation due to the kinematic mismatch of the patient and exoskeleton models. This paper presents the formulation, simulation, and accuracy quantification of the proposed method with simulated human movements. In addition, a sensitivity analysis of the method accuracy to marker position estimation errors, due to system calibration errors and marker drifts, has been carried out. The results show that, even with significant errors in the marker position estimation, method accuracy is adequate for RAR.

The assessment of (a) patient movement compliance with the prescribed exercises and (b) patient long-term improvement is critical when planning and evaluating the efficacy of RAR therapies. In order to obtain the patient motion data to conduct the said assessments, one has to estimate patient posture (i.e., the joint angles of the limbs). Patient posture estimation methods need to be practical and easy to set up for the physician, so that the said assessments can indeed be an integral part of the therapy.
Current methods for estimating patient posture are either cumbersome or not accurate enough in exoskeleton-based therapies. In order to overcome such limitations, we propose a method where low-cost RGB-D cameras (which render color and depth images) are directly installed in the exoskeleton and colored planar markers are attached to the patient's limb to estimate the angles of the GH joint, thereby overcoming the individual limitations of each of these systems. human motion capture [5]. Reference [6] reports Optotrak errors of 0.1-0.15 mm. However, in the specific case of exoskeleton-based therapy, these systems require redundant sensors and markers to cope with occlusions caused by the exoskeletal body. Therefore, their specific usage for therapy is limited. Besides, the cost of these systems is high (50 K-300 K USD [7]) compared to nonoptical MOCAPs.
(2) Electromagnetic systems do not suffer from optical occlusions. However, they are easily perturbed by surrounding metallic objects (e.g., exoskeletal body) and electric/magnetic fields [5]. An additional drawback of these systems is their limited detection volume when compared to optical systems.
(3) Inertial and Magnetic Measurement Systems are robust, handy, and economical for full-body human motion detection (upper limb tracking in [8,9]). With the use of advanced filtering techniques, inertial sensor drift errors are reduced and a dynamic accuracy of 3 deg. RMS [5] is achieved. However, these systems require patients to perform calibration motions/postures, which may not be suitable for those with neuromotor impairments.
In exoskeleton-based rehabilitation, the prevailing approach to estimate human limb joint angles (e.g., [10][11][12][13]) is to approximate them with the angles of the exoskeleton joints. However, misalignment between the axes of the exoskeleton and human joints may produce large estimation errors [14,15]. Accurate estimation of GH joint angles is hard to achieve using this approach, since it requires an exoskeleton with a complex kinematic structure that considers the concurrent motion of the sternoclavicular and acromioclavicular joints.
Recognizing the differences in the kinematic structures of the limb and exoskeleton, [16] presents a computational method which considers the limb and exoskeleton parallel kinematic chains related by the cuff constraints joining them together. Then, the IK problem of the parallel kinematic chain can be solved to find the limb joint angles. A limitation of this method is that its performance has been demonstrated solely for analytic (1-DOF) movements of the elbow and wrist joints.
The estimation accuracy of the GH joint angles has yet to be determined.
Reference [17] presents a computational method based on the estimation of the arm swivel angle (which parametrizes arm posture) for exoskeleton-based therapy. The arm IK is solved with a redundancy resolution criterion that chooses a swivel angle that allows the subject to retract the palm to the head efficiently. The approach in [17] extends their previous work in [18,19] by considering the influence of the wrist orientation on the swivel angle estimation. Although the error of the swivel angle estimation (mean error ≈ 4 deg.) has been reported for compound movements [17], individual errors in the wrist, elbow, and GH joint angles are not indicated.
Reference [20] extends the method in [17] to estimate the wrist angles and assesses its performance for compound movements (mean RMSE ≈ 10 deg. in the swivel angle estimation). Reference [20] reports the individual errors of the arm joint angles solely for the movement task where the swivel angle was best estimated (mean RMSE ≈ 5 deg. in the swivel angle estimation). No errors of the arm joint angles were discussed for the other cases. A limitation of the work in [20] is that the MOCAP used to obtain the reference angles to assess their method performance is a custom-made inertial system with no reported measurement accuracy.

Conclusions of the Literature Review.
We remind the reader that the general context of this paper is the estimation of the GH joint angles.
(1) As per our literature review, no MOCAPs have been developed for the specific scenario of exoskeletonbased rehabilitation. Even if current MOCAPs and the exoskeleton could be set up for simultaneous use (e.g., [15,16]), the setup protocol and operation are intricate and conflicting with the usual time and resources available for patient treatment.
(2) Exoskeleton-based posture estimations present limitations in their accuracy due to kinematic mismatch of the limb and exoskeleton [15,16].
(3) The accuracy of the GH joint angle estimations provided by computational methods in [16,17] is Applied Bionics and Biomechanics 3 unknown. Reference [20] extends the work in [17] by estimating the wrist angles. Reference [20] solely reports the estimation accuracy of the GH angles for the best-case scenario and the precision of its groundtruth is not indicated.

Contributions of This Paper.
In response to the limitations discussed in the estimation of patient joint angles in exoskeleton-based therapy (Sections 2 and 2.1), this paper introduces a hybrid approach to estimate, in real-time, the GH joint angles. This hybrid system is composed of a lowcost marker-based vision system and the rehabilitation robot, overcoming the individual limitations of its constitutive subsystems: (a) Occlusions are minimized, which are a major limitation of optical systems.
(b) Accuracy of joint angle estimation is improved, which is a major limitation of exoskeleton-based systems.
This paper presents the implementation and assessment of our method using simulated human motion data. In addition, a sensitivity analysis of our method accuracy to marker position estimation errors is carried out.
We have considered the following scenarios of application for the proposed method in the RAR domain: (A) Precise estimation of GH joint angles during rehabilitation or evaluation sessions of GH joint analytic movements.
(B) Acquisition of GH joint movement data enabling validation and improvement of other posture estimation methods without using expensive redundant optical MOCAPs.

Problem Definition.
This section presents the problem of estimating the patient limb GH joint angles during the GH joint RAR using the proposed hybrid motion capture system (a detailed version of the problem definition is presented in the Appendix). This problem can be stated as follows.
(3) Marker-based optical motion capture system ( ): (a) color and depth information captured by the RGB-D cameras installed in the exoskeleton links and (b) geometry and color of the markers attached to the patient upper limb (Figure 2(c)).
Goal. The goal is to estimate the patient GH joint angles (V ( )) with minimum error during the GH joint rehabilitation exercises.

Kinematic Models.
This section discusses the main features of the kinematic models of the human limb and exoskeleton used for the posture estimation method.

Kinematic Model of the Human Upper
Body. The human kinematic model is denoted by ( , ), where and are the sets of links and joints, respectively. We use the human upper body model presented in [16] (Figure 2(a)), which includes joints of the spine, scapuloclavicular system, and arm. The upper limb is modeled with 9 DOFs: 2 DOFs of the scapuloclavicular system, 3 DOFs of the GH joint (spherical joint), 2 DOFs of the elbow, and 2 DOFs of the wrist (see further details in the Appendix). This model presents the following advantages: (a) It can be easily implemented in robotic simulators and similar tools.
(b) It is suitable for simulating human-robot interaction in real-time [16].
(c) The spherical model of the GH joint avoids limitations of other representations of such joint, like the Gimbal lock that occurs when using the three concurrent and orthogonal 1-DOF revolute joints' model [22].

Kinematic
Model of the Exoskeleton. The exoskeleton kinematic model is denoted by ( , ), where and are the sets of links and joints, respectively. In this research, the rehabilitation exoskeleton used is the Armeo Spring ( Figure 2(b)), which is a passive system that supports the weight of the patient's arm [23] with springs. The Armeo kinematic structure includes rotational joints (equipped with encoders [24,25]) and prismatic joints (enabling exoskeleton adjustment to the size of each patient). We use the Armeo Spring kinematic model presented in [16], which includes both types of joints (see further details in the Appendix).

GH Joint Angles Estimation
Method. The aim of the method is to estimate the GH joint angles with respect to (w.r.t.) a coordinate system (CS) attached to the scapuloclavicular system. Figure 2(d) shows the proposed system for the GH joint angle estimation. Our approach is based on the estimation of the upper arm orientation w.r.t. the acromion (Figure 3(a)). According to such requirements, the rationale to install the markers of the optical MOCAP is as follows: (a) Marker 0 is rigidly installed in the acromion, so the estimated upper arm orientation can be expressed w.r.t. the 0 CS (and therefore w.r.t. the scapuloclavicular system).
(b) Marker 1 is rigidly installed in the upper arm, so that all the rotations of the upper arm are captured by 1 .
The region that was chosen to attach 1 to the upper arm by using a custom-made fixation (Figure 2(d)) is the distal part of the humerus (near the elbow). Elbow rotations do not affect the orientation of 1 . Reference [26] reports a five-marker installation procedure. This reference explicitly mentions five markers as an acceptable number for clinical upper limb tracking. In this paper, we report the usage of two markers for upper arm tracking. It is not possible to compare the performance of the marker placement protocol proposed here with the one in [26] because the work in [26] addresses (a) non-RAR scenarios, (b) tracking of the entire upper limb, and (c) protocol sensitivity w.r.t. its application on the dominant/nondominant arm and w.r.t. the age of test subjects. However, the work in [26] helps to establish the number of markers compatible with the clinical application of upper limb tracking.
The cameras of the optical motion capture system are rigidly attached (using custom-made supports) to exoskeleton links so that camera 0 detects marker 0 and camera 1 detects marker 1 during the GH joint training. Camera 0 is mounted on link 0 and camera 1 is mounted on link 8 (Figure 3(a)).
The cameras used in our system are of low cost. Commercial cameras that present similar specifications to the ones simulated here (Table 1) are Intel5 SR300 (99 USD) [27,28], DepthSense5 525 (164 USD) [29,30], and CamBoard pico S (690 USD) [28,31].    A summary of the steps to estimate the GH joint angles is as follows: (1) Estimate the pose of the markers w.r.t. the cameras.
(2) Estimate the pose of the cameras w.r.t. the exoskeleton.
(3) Estimate the pose of the markers w.r.t. the exoskeleton.
The details of the mentioned steps are presented in the following sections.

Estimation of the Pose of the Markers w.r.t. the Cameras.
The purpose of this step is to estimate the position and orientation of the markers (Figure 4) w.r.t. the CSs of the cameras using the color and depth images provided by each camera : (2) Pose of cameras w.r.t. E links The approximate marker disk centers detected by camera form the set̃= {̃0 , . . . ,̃, . . . , }.
(3) Computation of the marker CS in the camera CS: an (3) coordinate frame = [̂̂̂] is attached to each marker: The submatrix [̂̂̂] is normalized to guarantee its (3) nature. The framẽdescribes the estimated pose of marker w.r.t. the CS of the camera .   of the GH joint CS w.r.t. the 0 CS, is estimated during the calibration process of the system. Hence, the GH joint center is estimated as follows:

Estimation of the
(a) Estimate , which is the pose of the GH joint CS w.r.t. the exoskeleton base CS (see (8)). (b) Extract from . The point is the position of the center of the GH joint seen from the CS: (2) Estimate the position of the elbow joint center: the rigid transformation matrix 1 elw (elbow joint CS w.r.t. the 1 CS) is estimated during the calibration process of the system. Hence, the elbow joint center is computed as follows: (a) Estimate elw , which is the pose of the elbow joint CS w.r.t. the exoskeleton base CS (see (9)). (b) Extract elw from elw . The point elw is the position of the center of the elbow joint seen from the CS:

Refer the Angles of the GH Joint w.r.t. the Acromion.
Since 0 is rigidly attached to the acromion, the upper arm orientation can be expressed w.r.t. the acromion by using the inverse of 0 :̃0 arm =̃0 * ̃a rm .
3.4. Implementation and Simulation. The arm posture estimation method was implemented by using the V-REP robotics simulator [32]. In the simulator, the scene in Figure 2(d) is created, which includes the models of (a) a human patient, (b) an Armeo Spring, (c) the RGB-D vision sensors with the couplings to attach them to the exoskeleton, and (d) the planar markers with the couplings to attach them to the human arm. The configuration of the simulated vision sensors is summarized in Table 1.
For the estimation of the coordinates of disk centers c in the image c , color segmentation and blob detection algorithms available in the simulator were used. Additional code was written to sort blob centers by color. All additional code was written in LUA (lightweight embeddable scripting language) scripts.   (2) Patient movement generation: the movements of the patient upper limb that correspond to the recorded movements of the Armeo are computer-generated with the method in [16]. The said method provides an estimation of the patient posture given the joint angles of the exoskeleton by using an inverse kinematics approach.

Generation of the Ground-Truth
In this way, four sets (one per movement dataset) of known poses of the upper arm are obtained by simulating patient movement and compared here against those estimated with our method. Our method accuracy is assessed without compensating any time offsets between the reference and estimated angles. In this way, real-time accuracy of the method is assessed. Table 2 presents the approximate amplitudes of the Euler angle decomposition of the GH joint movements of the simulated patient w.r.t. its local CS.  are the rotation submatrices of transformation matrices 0 arm and̃0 arm , respectively. (b) Express Rot error in exponential map notation [22] as → arm ori ∈ R 3 .

Measurement of the Estimation Performance
(c) Compute arm ori as the RMS of ‖ → arm ori ‖ for all samples in the movement dataset.

Sensitivity Analysis.
A sensitivity analysis is carried out to study the influence of relevant parameters on the method accuracy. Formally, the sensitivity analysis determines the effect of the perturbation of the parameter on the objective function ( ). The relative sensitivity of ( ) w.r.t. , , is given by (11) [33]. The value of is the ratio (dimensionless) between the percentual changes in and : The upper arm pose accuracy (and, therefore, that of the GH joint angles) relies on the precise estimation of the position of the centers of the elbow and GH joints (̃e lw and ) (Section 3.3.4), which ultimately depend on the following transformations involving the markers: For the sensitivity analysis (see (11)), the vector-valued function ( ) quantifies the estimation error of the arm position and orientation (see (12)) and the parameter set represents the marker translation errors. The parameter set is defined as = { 1 , 2 , 3 , 4 , 5 , 6 }, where each ∈ is a scalar representing the magnitude of a translation of a specific marker along a prescribed direction. Table 3 describes the meaning of each parameter in set : ( ) = ( arm pos ( ) , arm ori ( )) ; ( ) : The sensitivity analysis procedure ( Figure 11) entails the following steps: (1) Load the movement dataset of the GH joint to test (SFE, SAbAd, SIR, and COMB).  Translation with magnitude ‖ 6 ‖ of 1 along -axis Elbow joint (5) Compute the position and orientation components of as per (11). The derivative of ( ) w.r.t. is given by (13). The required derivatives are computed numerically [34,35]: The complete sensitivity analysis was performed for each movement dataset (SFE, SAbAd, SIR, and COMB). The directions in which marker translations occur (Table 3) are chosen so that the markers do not leave the detection volume of the cameras. Table 4 summarizes the parameters of the sensitivity analysis. Translation units are in meters (mts).

Results and Discussion
This section presents and discusses the results of (a) estimation accuracy of the marker 3D position, (b) estimation accuracy of the upper arm pose, and (c) sensitivity analysis of the estimation accuracy of the upper arm pose w.r.t. translation errors in 0 and 1 elw . Table 5 presents the RMS of the estimation errors of the position of the markers per movement dataset. The mean RMS errors of the position estimation of 0 and 1 for all movement datasets are 0.00083 and 0.00208 mts, respectively. Figure 13 shows the box plots of the estimation errors in the marker positions for all movement datasets. A greater variation in the position estimation accuracy of marker 1 , in comparison to that of 0 , is observed. We have attributed this to (a) the higher linear and rotational velocities and likewise (b) the larger translations and rotations that 1 undergoes compared to 0 .

Results of Upper Arm Pose Estimation.
The RMS of errors in the upper arm pose estimation are presented in Table 6. By averaging the results of all movement datasets, errors of 0.00110 mts and 0.88921 deg. in the upper arm position and orientation estimation are obtained. Figure 13 shows the box plots of the estimation errors in the upper arm position and orientation for all movement datasets. In motor rehabilitation, angular errors in the range of 3-5 degrees are considered acceptable for mobility evaluation of patients [6,36,37]. Figure 13 shows that our arm orientation estimation accuracy is adequate for exoskeleton-assisted rehabilitation.

Results of the Sensitivity Analysis.
The results of the sensitivity analysis per movement dataset of the shoulder are presented in Figures 14, 15, 16, and 17. In each figure, the following subfigures are presented:  Figures  14, 15, 16, and 17, one can observe that the translations of marker 1 produce larger absolute errors in the upper arm orientation estimation when compared to those produced by translations of marker 0 . Notice that the -axis and -axis of the elbow joint CS are always perpendicular to the upper arm vector ( ⃗ arm ) (Figure 12(b)). When the position of 1 is perturbed along the said axes, the angle between (i) the actual upper arm vector ( ⃗ arm ) and (ii) the estimated upper arm vector (̃a rm ) (which is inaccurate due to the perturbation of the marker position) is maximal. A side effect of the marker position perturbation is that the marker suffers modifications of scale and changes in the level of perspective distortion in the images of camera , affecting the accuracy of the system. This situation can be observed in Figures 14(b Hence, the orientation estimation of the upper arm is more robust than the position estimation w.r.t. errors in the translational components of matrices 0 and 1 elw . The results of the sensitivity analysis show that the assumption that transformations 0 and 1 elw are rigid is reasonable. Even with marker drifts of 0.02 mts, the GH joint angles can be estimated with an accuracy (RMSE 3.6 deg.) appropriate for the mobility evaluation of patients (in the range of 3-5 deg.).

Sensitivity in Arm Orientation Estimation. In
Marker drifts must be mitigated by the marker attachments to the human body. Furthermore, marker attachments should be designed to minimize the effect of errors in 0 and 1 elw on the method accuracy. For example, notice how the attachment of marker 1 (Figure 12(b)) locates marker 1 with an offset w.r.t. the elbow joint center along the direction which least affects the upper arm orientation estimation.
The results presented suggest that the method we implemented is a feasible alternative for estimating the GH joint angles in a RAR scenario. posture estimation (including the GH joint) in exoskeletonbased rehabilitation using computational methods. Among the mentioned works, only [20] reports the errors (mean RMSE 4.8 deg.) in the GH joint angles estimation. Reference [20] reports RMSE values of the GH joint angles only for the best-case scenario (swivel angle mean RMSE 5 deg.). For all the movement tasks tested, the method in [20] presents a mean RMSE of 10 deg. for the swivel angle estimations.

Comparison to Related
Given that global errors of the swivel angle double those of the best-case scenario, a report of global errors of GH joint angle estimations of the method in [20] is required to reach a conclusion regarding its suitability for clinical use. Table 7 summarizes the comparison of our contributions w.r.t. comparable works (i.e., [20]).

Conclusions and Future Work
In the context of RAR, this paper presents the formulation, implementation, and assessment, in silico, of a novel accurate method to estimate the patient GH joint angles during therapy. Our method does not require redundant markers or cameras and relies on simple geometric relationships and tools of standard robotics and computer vision libraries. These characteristics make it economical and readily applicable in RAR. The accuracy and the robustness of our method are evaluated using computer-generated human movement data corresponding to actual movement datasets of the Armeo Spring. We present a formal sensitivity analysis of the pose estimation accuracy w.r.t. marker position estimation errors produced by (a) system calibration errors and (b) marker drifts (due to skin artifacts). This analysis indicates that even in the presence of large marker position errors our method presents an accuracy that is acceptable for patient mobility appraisal.
Future work includes (a) implementation of the method using commercially available RGB-D vision sensors, (b) evaluation of the method accuracy with actual human movement data, (c) adaptation of the method using solely RGB cameras, and (d) extension of our method to address other limbs.
(3) A marker-based optical tracking system composed of two RGB-D cameras and two planar markers (Figure 2(c)). Consider the following remarks: (a) A set = { 0 , 1 } of planar markers that are detected by the cameras of and are installed on the patient upper limb.
(i) All present the same 2D square geometry, with a disk in each corner. The position of each disk w.r.t. the marker CS is known. The set of disks is = { 0 , . . . , , . . . , }.
(A) presents a color ∈ that can be detected by (Figure 2(c)). (B) renders a depth image associated with the scene in c , defined as d , of × pixels; ≤ and ≤ . The pixel coordinates ( , V) in d take values 0 ≤ ≤ − 1 and 0 ≤ V ≤ − 1. The CS of images c and d is coincident.
(C) presents a truncated square pyramid detection volume parametrized by the minimum and maximum detection distances and the horizontal and vertical field of view of . Table 1
(a) ‖ ‖ is the Euclidean norm of vector .

Glossary
Acromion: Region of the scapula bone above the GH joint Clavicle: Bone of the shoulder girdle located at the root of the neck CS(s): Coordinate system(s) COMB: Combination of movements of the GH joint (SAbAd, SFE, and SIR) DOF(s): Degree(s) of freedom GH: Glenohumeral Humerus: Upper arm bone MOCAP(s): Motion capture system(s) mts: Meters RAR: Robot-Assisted Rehabilitation RMS: Root mean square Scapula: Bone that connects the humerus to the clavicle SAbAd: Shoulder horizontal abduction-adduction SFE: Shoulder flexion-extension SIR: Shoulder internal rotation VR: Virtual Reality V-REP: Virtual Robot Experimentation Platform w.r.t.: With respect to : Exoskeleton kinematic model : Human upper body kinematic model = { 0 , 1 }: Set of planar markers mounted on the patient 3-tuple of joint angles of the GH joint at instant V ( ): Tuple of joint angles of the exoskeleton kinematic model at instant : Transformation matrix of marker w.r.t. the base CS : Transformation matrix of marker w.r.t. the CS 0 : Transformation matrix of the GH joint w.r.t. the 0 marker 1 elw : Transformation matrix of the elbow joint w.r.t. the 1 marker Notation : can be a position, transformation, and so forth, of object w.r.t. object CS.