A vision-based approach for estimating contact forces : Applications to robot-assisted surgery

The primary goal of this paper is to provide force feedback to the user using vision-based techniques. The approach presented in this paper can be used to provide force feedback to the surgeon for robot-assisted procedures. As proof of concept, we have developed a linear elastic finite element model (FEM) of a rubber membrane whereby the nodal displacements of the membrane points are measured using vision. These nodal displacements are the input into our finite element model. In the first experiment, we track the deformation of the membrane in real-time through stereovision and compare it with the actual deformation computed through forward kinematics of the robot arm. On the basis of accurate deformation estimation through vision, we test the physical model of a membrane developed through finite element techniques. The FEM model accurately reflects the interaction forces on the user console when the interaction forces of the robot arm with the membrane are compared with those experienced by the surgeon on the console through the force feedback device. In the second experiment, the PHANToM haptic interface device is used to control the Mitsubishi PA-10 robot arm and interact with the membrane in real-time. Image data obtained through vision of the deformation of the membrane is used as the displacement input for the FEM model to compute the local interaction forces which are then displayed on the user console for providing force feedback and hence closing the loop.


INTRODUCTION
The primary goal of this research is to provide force feedback using vision-based techniques.Our formulation is based on the finite element method (FEM) formulation for a linear elastic model of the membrane.We have chosen vision to estimate the interaction forces rather than using a force sensor because a force sensor will provide only the local forces at a point.However, by obtaining the nodal displacement using vision-based techniques will provide the user with force feedback for an area of interest rather than a single point from the force sensor.One possible application of this work is in robot-assisted surgery, where the surgeon can interact with the model on the screen based on the vision feedback received from the surgical site.Another advantage of using vision is the noninvasiveness of the displacement estimation method.We have developed a mathematical model of interaction with the membrane using FEM whereby vision is used to provide displacement information to the FEM modal for computing the interaction forces.
To achieve the goal of realistic haptic interaction with soft tissues, we have developed an experimental system for "reality-based" modeling of biological tissues (Figure 1).By reality-based modeling, we are referring to the process of experimentally determining the mechanical properties of the tissues so that our experimental data can be fit to an appropriate constitutive model for the tissue that is used for haptic interaction by the surgeon (Chanthasopeephan et al 2003;Hu and Desai 2003).We have previously demonstrated the capability of our system to measure forces due to probing (Hu and Desai 2003) and cutting (Chanthasopeephan et al 2003) of porcine liver tissue.Our previous work also indicates that force feedback has a significant effect on a surgeon's ability to characterize soft tissues during laparoscopic surgery (Tholey et al 2003).In this paper, we describe an extension of our previous work whereby vision information is used to provide force feedback to the surgeon (Kennedy 2002;Kennedy  et al 2002; Kennedy and Desai 2003).Our goal is to track the displacement of the membrane using stereo vision, then overlay the membrane displacement on our FEM model to determine the interaction forces at the membrane's surface.A set of experiments were designed for tracking a membrane and providing haptic feedback to the user based on the vision information and the material properties of the membrane.
Visual servoing is an essential tool for tracking deformable objects.Visual servoing encompasses many different areas such as kinematics, dynamics, image processing, control theory, and real-time computing.An excellent overview of visual servoing is presented in Hutchinson et al (1996).A novel method for tracking objects using a deformable dual-mesh is presented in Akgul and Kambhamettu (1999).There has been significant research in this area, both for surgery simulation (Cotin et al 1999) and in actual procedures in laparoscopic hepatic surgery (Stefansic et al 2000).Research has also been done on deformation modeling of virtual objects (Cotin et al 1996;James and Pai 1999).The work presented in this paper aims to track deformable objects rather than models of the objects.This will require tracking a membrane (where the deformations are produced through a robot arm interacting with the membrane, which simulates a tissue) and overlaying the vision data onto a FEM model which computes the forces of interaction and displacement in real time based on the vision data.The user interacts with the FEM model and the forces exerted by the user are reflected back onto the robot arm which reacts to the exerted force.
A major constraint for any surgical system is the realtime calculation of the various parameters necessary for a visually and haptically (force and tactile feedback) realistic experience.For a more realistic visual experience, images must be updated at a rate greater than 30 frames per second, and preferably around 200 frames per second (Astley and Hayward 2000).Haptic systems require a much higher update rate (typically greater than 500 Hz) of the sensing area, but preferable around 1000 Hz (Astley and Hayward 2000).
The disparity in the update rates for visual and haptic data suggests that the ideal robotic surgical system should be a multi-frequency system to efficiently use the computational resources (Astley and Hayward 2000;Burdea 2000).Numerous methods have been considered for performing the necessary calculations surgical manipulations in a virtual surgical environment (Astley and Hayward 1998;Cavusoglu and Tendick 2000;Cotin et al 1996Cotin et al , 1999;;James and Pai 1999;Zhuang and Canny 2000).A plausible approach that requires local deformation information would be appropriate in order to calculate the contact forces felt by the user during the procedure (Mendoza and Laugier 2001).The use of visual information for tracking the deformation of surfaces has also been studied extensively (Vedula et al 1999;Zhang and Kambhamettu 2000).
The remainder of this article is structured as follows.In the second section, a brief description of the various experimental components is presented.In the third section, mathematical model of the membrane using FEM techniques is presented along with the algorithm for detecting membrane contact in the virtual model on the haptic display.In the forth section, we present two experiments which validate our approach and finally some concluding remarks are made.

SYSTEM DESCRIPTION
The testbed for validating the model and carrying out experiments consists of a PHANToM haptic interface device (manufactured by Sensable Technologies, Inc.), seven degree of freedom PA-10 robot arm system (Mitsubishi Heavy Industries, Ltd.), 6-axis force/torque sensor model Gamma F/T, (ATI Industrial Automation, resolution 0.05 N in the x and y directions and 0.01 N in the z-direction), and a pair of Elmo micro-cameras (see Figure 2).Our experimental testbed enables the acquisition of image data from the membrane and transmitting it to and from the PHANToM haptic device.The pair of micro-cameras observes the deformation of a membrane.The deformation is controlled and measured precisely by the robot arm, while the force data is acquired from the force/torque sensor attached to the robot end-effector.A force sensor is used as an external verification tool to compare with the force data computed from the FEM model and displayed on the haptic interface device.The results from the computed force values are displayed on the haptic device and updated at 1000 Hz while the graphics on the console updates at 30 Hz.

Vision
The vision system comprise of two micro-cameras in a fixed configuration, with image data being acquired from each camera simultaneously.The apparatus allows a fixed stereo rig consisting of the two micro-cameras to view a circular membrane.The surface has an inscribed array of dots that are used to calculate the contact forces.The locations of these points are tracked in real-time (30 Hz) and overlaid on a finite element model of the surface to obtain contact force information.This requires (1) accurate calibration of each camera's internal and external parameters, as well as the baseline of the stereo-rig; (2) a method of performing the correspondence of given surface points in a sequence of images acquired by the vision system; and (3) a method of calculating the three-dimensional location of a surface point, given data acquired from a stereo image pair.

Robot arm and controller
The Mitsubishi PA-10 robot arm is used to control the deformation of the membrane.The robot arm has seven degrees of freedom and an open controller architecture.The four-layer control architecture is made up of the robot arm, servo controller, the motion control card, and the upper control computer.The upper control computer runs the QNX real-time operating system, and we have been able to achieve communication rates of up to 700 Hz with the robot servo driver.The robot joints are actuated through three-phase AC servomotors and harmonic gear transmissions.Joint positions are measured through resolvers at the motor axis, with a resolution of 0.000439 o over ±3 output revolutions.The robot joints are able to achieve approximately 1 µm of positional repeatability.The control software is developed using Visual C++.

Haptic device
The haptic device consists of a PHANToM Premium Model 1.5A.This device has six degrees of position feedback and three degrees of force feedback.Haptic rendering is performed using Visual C++ and the Ghost SDK (developed by Sensable Technologies, Inc.).Force calculations are performed at 1000 Hz based on the surface deformation measured by the vision system.The membrane is represented visually in the virtual environment using OpenGL, which is updated at a rate of approximately 30 Hz.The user is able to interact with the virtual deformable surface through the PHANToM device and experiences the same forces as those applied by the robot arm on the membrane.

FORCE INTERACTION
Force feedback allows a user to feel and exert forces on an object.For a realistic force feedback, contact forces must be communicated to the haptic device at a rate of at least 500 Hz.In our experiments, we use the PHANToM as our force feedback device and the vision system to sense the forces on the membrane.Force estimation through vision is computed based on (a) the deformation of the fiducials on the membrane and (b) the geometric and mechanical properties, such as thickness, circular area confined at the boundary, Poisson's ratio and Young's modulus of the membrane.Our goal is to model a deformable surface to allow a user to interact with it in real-time.The algorithms we have developed for this purpose address the following issues: (1) a physical model for the surface that accurately simulates the actual forces that would be experienced by a user as if he/she were interacting with the real surface, and (2) a means of haptically and visually rendering the membrane to create a realistic interaction experience for the user.

Surface model-Force calculation through FEM
We model the membrane in our experiments using a linear elastic finite element model.Use of a linear elastic model allows us to precompute the response of the mesh to elementary displacements in each degree of freedom, allowing us to perform real-time calculations of contact forces through the principle of superposition.Though the applicability of this methodology in a surgical setting will require nonlinear viscoelastic modeling (due to finite strains in the tissues), the primary goal of this paper is to illustrate the feasibility of the concept of using vision to update the deformations on the finite element mesh in real time and force feedback the forces to the user.The membrane is first discretized into elements using the locations of the fiducials on the membrane as the node locations.We employ triangular plate elements in our calculations, which allow forces to be calculated for five degrees of freedom per node (three translations and rotation about the x and y axis of the element).The degrees of freedom for both the planar element and the element in bending are shown in Figure 3.
Classical finite element procedures in structural mechanics involve solving the linear system given by where k is the global stiffness matrix of a structure, u is the vector of displacements, and f is a vector of corresponding forces.Typically, the stiffness matrix is inverted to calculate the nodal displacements in response to a set of applied forces to the structure.In our approach, however, we require the force calculations to be made in response to a set of displacements induced by the robot end-effector.This approach greatly simplifies the calculation in equation (1) because the displacement matrix is provided by the vision system, thereby eliminating the need to invert the stiffness matrix.This allows for calculation of contact forces with the membrane at haptic servo rates (greater than 1000 Hz).The computation time is further simplified by the need to only calculate the nodal forces for the element with which the robot end-effector is in contact.We perform an additional step of precomputing the response of the mesh to elementary deformations.The stiffness matrix for triangular plate elements can be separated into two parts, one describing the response of the structure for planar strains and the other for bending strains (Przemieniecki 1968).The stiffness matrix for planar strains is 2n × 2n, where n is the number of nodes in the mesh.The stiffness matrix for bending strains is 3n × 3n, and allows for the calculation of reactions of the mesh to transverse deflections and rotations.
The stiffness matrix for in-plane forces for a triangular plate element can be separated into two parts (Przemieniecki 1968): where k n represents the stiffness due to normal stress and k s represents the stiffness due to shearing stresses.For triangular plates in bending, the stiffness matrix can be derived as where xy, F = 12(x + vy)y, with x i and y i representing the coordinates of the ith node as shown in Figure 3, v is the Poisson's ratio, and E is the Young's modulus of the specimen.
After assembling the stiffness matrices over all the elements, the pre-calculations of elementary deformations for each degree of freedom are performed.We employ the Lagrange Multiplier Method for the imposition of constraints.
We first consider the variational formulation for a discrete structural model which is given by To impose a constraint on the displacement for a given degree of freedom U i with U i = U * i , we augment the right side of equation ( 4) to obtain * = 1 2 Since δU and δλ are arbitrary, we obtain where e i is a vector with all entries equal to zero except its ith entry, which is equal to one.Inverting the system of equations given in equation ( 6) results in a vector of resultant displacements U appended by the Lagrange multipliers (λ's), which are equal to minus the force required to impose the displacement constraint at U i .
For each degree of freedom of the membrane, the system of equations given in equation ( 6) is constructed by imposing an elementary constraint for the corresponding degree of freedom, and a zero displacement constraint for every degree of freedom at the boundary nodes which are assumed to remain fixed.The system is then inverted to obtain the resultant force due to the constraint.These calculations must be performed 2m times for the planar stiffness matrix, and 3m times for the bending stiffness matrix, where m is the number of free nodes.After completing the precalculation phase, real-time determination of forces at each node can be calculated as multiples of the elementary force calculated previously.
In our experiments, we compare the results of the PHANToM interacting with the virtual membrane using the above FEM model with the results of the robot interacting with the real membrane.In the case of the deformation of the real membrane, the displacement of the nodes is given by the vision information.In the case of the virtual membrane, the global deformation of the mesh must be updated online using the elementary displacements calculation given above.Using the principle of superposition, the displacement of the entire mesh can be calculated as a linear combination of the displacement induced by each node moved by the PHANToM.
The calculation of the displacement nodes that the PHANToM is in contact with requires both a collision detection scheme to determine the contact point within an element and a kinematic model for movement of the element based on the PHANToM's motion.For a realistic haptic experience, it is necessary to detect the collisions between the haptic interface device and the rendered surface.The stylus of the PHANToM is represented as a single point in the graphic display.The task of the collision detection algorithm is to determine when the stylus (point) is in contact with any of the segments of the surface defined by the node locations.The steps in the collision detection algorithm are enumerated below: (1) During each iteration of the servo loop, the distance between the current PHANToM position and each node is calculated.
(2) The distances between the nodes and the PHANToM are arranged in ascending order, the first three elements of this list are the three closest nodes to the PHANToM.(3) The collision of the PHANToM with the area enclosed by these nodes is found by calculating the angles between the lines between each node and the PHANToM.If the sum of these angles is equal to 2π ± ε, a collision is identified.(4) The force sent to the PHANToM is found through an interpolation function based on the calculated force value at each node.

EXPERIMENTS
In this section, we describe two experiments that we conducted, namely, (a) comparison of forces computed by the FEM model based on vision feedback versus the actual forces measured by the force sensor attached to the robot tool tip, and (b) the use of the PHANToM to control the robot arm with real-time forces reflected to the PHAN-ToM based on the robot's contact with the membrane.We used a membrane made of epicholohydrin rubber (radius 39 mm) for which the Poisson's ratio (v) for this type of rubber was assumed to be 0.45, and the modulus of elasticity was 27.2 MPa.

Experiment 1
The goal of the first set of experiments was to verify that the vision system can correctly measure the deformation of the membrane when the membrane is deformed by the robot arm.Once this is verified, we can input the displacement data into the FEM model to compute the interaction forces for display on the haptic device.The deformation of the membrane is measured by tracking an array of dots attached to its surface.The centroid of each dot represents a node on the surface that can then be used to reconstruct the surface both visually and haptically in a virtual environment.
The robot is positioned in such a way that it deforms the membrane in a direction normal to the contact surface.The deformation of the membrane is then progressively increased in 0.5 mm increments (from 0 to 5 mm).Concurrently, a six degree of freedom force sensor attached to the robot end-effector measures the forces and torques experienced by the robot while deforming the membrane.The stereovision apparatus is positioned approximately 250 mm from the membrane to enable real-time acquisition of the dots fixed to the membrane (see Figure 2).
The representative results from tracking the displacement of the center node of the mesh are shown in Figure 4.There is a good match between the displacement of the node as measured by the vision system and that computed form the forward kinematics of the robot arm.Figure 4 shows the mean displacement detected by the vision system, the error in the vision systems measurements (SD = 2), and the commanded displacement by the robot arm.The maximum difference between the mean displacement detected by vision and the commanded displacement by the robot was 0.16 mm, and the maximum error was 0.05 mm.
On the basis of the above success in accurately estimating the displacement through vision, we tested out FEM model for displaying the interaction forces.We compared the displayed force with the actual forces measured by the force sensor attached to the robot arm.Figure 5 shows the membrane being deformed by the PHANToM Stylus along with the locations of the randomly selected nine nodes (in this experiment) probed by the PHANToM.
The results of the PHANToM's interaction with the membrane and the actual measured forces by the force sensor are shown in Figure 6.Results show a reasonably good match in the interaction forces computed by the FEM model and those recorded by the force sensor attached to the robot tool tip.The maximum difference between the measurements was 0.9 N. The FEM model was tested by probing the mesh in nine locations by 1 mm in the direction normal to the membrane (z-axis) with the PHANToM stylus and computing the resultant force in the z-direction.Only the z-direction force data is presented in Figure 6 since all other forces and torques were found to be negligible in this experiment.
We then repeated the above experiment by probing the real membrane with the robot arm.The displacement locations and magnitudes induced by the robot arm on the real membrane were the same as those induced by the PHAN-ToM on the virtual membrane.The displacements were calculated by the vision system through triangulation of the node locations, and the contact point of the robot arm with the membrane was assumed to be the node with the highest magnitude displacement from its baseline position.
Experiment 2 One of the most common practices in minimally invasive surgery is palpation of the organ of interest by pressing the end of the endoscopic tools against the tissue and  observing the deformation of the tissue on the overhead monitor.Though in conventional endoscopic surgery, haptic (force and tactile) feedback is absent, this experiment illustrates the process of interaction with the deformable membrane through motion commands from the PHAN-ToM to the Mitsubishi PA-10 robot and "feeling" the forces of interaction on the PHANToM based on the motion of the membrane (caused by poking the back of the membrane by the probe attached to the end-effector of the PA-10) as recorded by the vision system.On the basis of the observed deformation and the finite element model that we have developed, we will illustrate the force interaction experienced by the during the real-time communication between the PHANToM, PA-10 robot arm, and the vision system.The robot end-effector was positioned approximately 10 cm from the real membrane and equipped with a cylindrical probe to deform the membrane.Communication between the user's computer and the robot-control computer was then established using TCP/IP.The control loop was then initiated, acquiring stereo image data at approximately 30 Hz, with force-feedback reflected to the user at approximately 1000 Hz through the PHANToM device.
The robot was then instructed to move according to the user's movement of the PHANToM Stylus, but constrained to move in a direction normal to the surface of the membrane (z-axis).When the robot endeffector came in contact with the membrane surface, the movement of the surface nodes was detected by the vision system.Force calculations were then made based on the FEM model described in "Force Interaction" section, again assuming the point of contact between the robot end-effector and the membrane was the node with the greatest displacement from its baseline position.The results of this experiment are shown in Figure 7.The point of contact was calculated to be the center node of the mesh.The figure shows the zdisplacement and resultant force in the z-direction reflected to the used as the real mesh is probed several times by the robot end-effector at the same location with varying displacements (the PHANToM controls the movement of the robot in real-time).The PHANToM accurately reflected the force of probing the membrane to the operator.

CONCLUSIONS
In this paper, we have presented a proof of concept for using vision-based techniques for estimating the interaction forces on the membrane using a linear elastic finite element model.The experimental results presented in this paper verify that vision-based techniques combined with the material properties of the membrane can provide a reasonably good estimate of the interaction forces without using a force sensor.The advantage of this approach is that the FEM model provides force information in an area of interest rather than a single point (which normally the case with using a force sensor).This methodology would allow the surgeon to measure forces at points other that the point of contact, and interact with those points through a virtual model of the surface without actually contacting those points with the robot.Although we have demonstrated the proof of concept with one-dimensional probing, the extension of the experiments to 3-D (x, y, z) or 5-D (x, y, z, pitch, roll) is straightforward based on the triangular plate elements we have used in the model.In practice, we plan to provide additional degrees of force feedback as required for a given surgical procedure.The experimental data show a good match between the results of the PHAN-ToM interacting with the FEM model of the membrane and the robot end-effector interacting with the real membrane.The experiments described in the paper verify the ability to (1) track the deformation of a membrane in real-time based on the 3D movement of fixed points on the membrane and render it in real time and provide force feedback to the user, (2) implement a robust tracking algorithm that is capable of accurately tracking the location of such points on a textured surface, (3) create a realistic FEM model of the membrane for interaction with the PHANToM device and, (4) allow a user to control a robot arm with the PHANToM device and experience real-time forces at the robot endeffector interacting with a membrane.
Although the results presented in this paper are preliminary and provide proof of concept, there are several things that must be addressed in future work.One of the key issues will be tracking a textured surface in the presence of uncontrolled lighting conditions.Wetness of tissues can cause specular reflection.In the current system, the maximum error in measuring the nodal displacements was 0.0561 mm.Similarly the maximum error in measuring the forces through the FEM model and that through the actual force sensor was 0.9 N. Another issue of significance that will need to be addressed in future work is dealing with occlusion in an actual surgical environment.Finally, using the approach proposed in the paper will also require us to estimate the material properties of the tissues in the non-preconditioned state, as real surgeries are not done on preconditioned tissues.This is currently one of the topics of our ongoing research.

Figure 1
Figure 1 Experimental system for "reality-based" modeling of soft tissues.Mechanical properties of soft tissues are experimentally determined and used for haptic interaction by the surgeon.

Figure 2
Figure2The experimental system for vision-based force feedback from the membrane.

Figure 3
Figure 3 Degrees of freedom for a triangular plate element in-plane and in-bending.

Figure 4
Figure 4 Comparison of the measured displacement of the center node of the mesh by the vision system along with the displacement of the same node by the Mitsubishi robot.

Figure 5
Figure 5 Virtual membrane being deformed by PHANToM stylus and approximate locations of points probed by PHANToM.

Figure 6
Figure6Comparison of force at specified nodes measured by the vision system and that measured by the force sensor.

Figure 7
Figure 7 Results of probing the real membrane with robot arm driven by movement of PHANToM stylus.