We introduce a novel parameterization of facial expressions by using elastic surface model. The elastic surface model has been used as a deformation tool especially for nonrigid organic objects. The parameter of expressions is either retrieved from existing articulated face models or obtained indirectly by manipulating facial muscles. The obtained parameter can be applied on target face models dissimilar to the source model to create novel expressions. Due to the limited number of control points, the animation data created using the parameterization require less storage size without affecting the range of deformation it provides. The proposed method can be utilized in many ways: (1) creating a novel facial expression from scratch, (2) parameterizing existing articulation data, (3) parameterizing indirectly by muscle construction, and (4) providing a new animation data format which requires less storage.

Recent interests in facial modeling and animation have been spurred by the increasing appearance of virtual characters in film and video, inexpensive desktop processing power, and the potential for a new 3D immersive communication metaphor for human-computer interaction. Facial modeling and animation technique is a difficult task, because exact classifications are complicated by the lack of exact boundaries between methods and the fact that recent approaches often integrate several methods to produce better results. The classification of these methods is described in the survey report [

In this paper, a novel parameterization of facial expressions is introduced. The parameters can be learned from existing face models or created from scratch. The obtained parameters can be applied on target face models dissimilar to the source model from which the parameters are taken in order to generate similar expressions on the target models. We also adopt a muscle-based animation system to obtain the parameters indirectly. It is tedious and difficult to make expressions by manipulating each control point. The proposed system provides a new alternative to make expressions which is easier and more intuitive. Facial animation by the parameterization requires less storage especially for highly complex face models with huge articulation data without reducing the range of deformation it provides.

In Section

Applying facial expressions from human faces to computer-generated characters has been widely studied [

To compute facial parameters from existing models, we assume that there is a “point-to-point” correspondence between them in order to derive motion vectors for each expression. This assumption might be too restrictive in some cases; however there are several techniques to establish correspondences between two different models [

In this section, the underlying theory of the elastic skin model is introduced. An intuitive surface deformation can be modeled by minimizing physically inspired elastic energies. The surface is assumed to behave like a physical skin that stretches and bends as forces are acting on it. Mathematically this behavior can be captured by the energy functional that penalizes both stretching and bending [

In a modeling application one would have to minimize the elastic energy in (

The Laplace operator in (

Figure

The surface is deformed by minimizing elastic surface energy subject to the user constraints. The gray area is the fixed region

In general, the order

In this section, we parameterize a set of existing face models using the elastic surface model. The facial parameters are calculated so that obtained parameters are precise enough to approximate the deformation of certain facial expression. The input consists of a face model with neutral expression and a set of face models with key expressions. To match up every vertices, all the models share the same number of vertices and triangles and have identical connectivity. Equation (

The solution of (

Let the

The left-hand side of the (

To obtain the basis functions

The control points are a subset of MPEG-4 feature points (FPs) and some additional points. The total number is forty-seven for the test models. We put more control points around eyes and mouth since those facial parts need to be deformed more flexibly than other facial parts to make facial expressions.

If no fixed region

Figure

First row shows the original face model and the second row shows the face models generated using facial parameters calculated from the original models.

Anger

Blink

Disgust

Smile

Surprise

Facial expression blending is a common technique for facial animation to create a novel expression by blending existing expressions.

Given the set of facial parameters

Figure

Expressions are blended by changing weight for each expression. (a) Blending smile (1.0) and blink (1.0), (b) blending smile (1.0) and disgust (0.5), (c) blending all expressions : anger (0.3) + blink (0.6) + disgust (0.3) + smile (0.1) + surprise (0.4).

Mixed expression

Expression cloning. In order to copy expressions from the source model, exactly identical facial control points of the source model have to be defined on the target model. The selection of facial control points is done manually.

Target model 1

Target model 2

Target model 3

We can also attenuate the displacement motion of each control point independently by adopting the importance map as suggested in [

Expression cloning is a technique that copies expressions of a source face model onto a target face model. The mesh structure of the models needs not to be the same. Our proposed facial parameterization can be used for this purpose.

The first step selects the facial control points on the target model, each of which is exactly correspond to the control point on the source model. It takes no more than twenty minutes to select all facial control points on the target model. The second step computes the basis functions

Expression cloning. Time took to compute the basis functions for each target model.

Vertex | Face | Time (basis function) | |
---|---|---|---|

Source model | 755 | 473 | 2.5 seconds |

Target model 1 | 5226 | 3139 | 17.1 seconds |

Target model 2 | 2362 | 1159 | 6.2 seconds |

Target model 3 | 2254 | 997 | 5.5 seconds |

To compensate for the scale difference between the source and the target model, each element of facial parameters

In Figure

First row shows the source models and the other rows show target models generated by expression cloning. The expressions are retrieved from the source model and are copied on each target model. Left most column shows the neutral expression of each model.

Neutral

Anger

Disgust

Smile

Surprise

At the end of this paper we show the various expressions generated by a set of facial parameters in Figure

The facial animation by muscle contraction has been studied by many researchers [

We define two types of muscles: a linear muscle that pulls and a sphincter muscle that squeezes the nearby skin elements. Similar pseudomuscle model is first proposed in [

The major human facial muscles:

Most of the methods proposed before using a facial muscle model try to attach nearby skin elements to a pseudomuscle in the registration stage then deform the skin elements as the muscle contracts. The magnitude of the deformation is determined from the relative position of the skin element from the muscle. In a physically based facial modeling approach [

In order to generate natural expressions and provide the animator easier operability, a set of simplified major facial muscles is defined and each of them is attached to the mesh by referring a anatomical model. These tasks are done manually and must be adopted for each different face model. Note that each muscle is a virtual edge connecting the two vertices (attachment points) of the mesh.

The fundamental of the linear muscle is that one end is the bony attachment that remains fixed, while the other end is embedded in the soft tissue of the skin. When the muscle is operated, it contracts along the two ends of the muscle. Each muscle has maximum and minimum zones of influence; however there is no analytic methods to measure them since only the surface points could be measured and the range of influence varies greatly from face to face [

The sphincter muscles around the mouth and the eyes that squeeze the skin tissue are described as an aggregate of linear muscles radially arranged from a pseudocenter point. The pseudo center is calculated by fitting an elliptic curve to the points defining the sphincter muscle.

Figure

The blue vertex is the fixed bony attachment point, and the red vertex is the skin attachment point. The green vertex is a facial control point within the zone of influence. When the muscle contract, the red point is moved along the muscle endpoints. Rs: radial distant at the muscle's registration, Rf: radial fall off distance,

Linear muscle

Sphincter muscle

To compute the zone of maximum and minimum influences, we adopt the method proposed in [

Figure

Facial muscle registration. The control point (index 24) is influence by two linear muscles. (a) Right levator (

In our simplified muscle models, the linear muscle contracts only along the muscle's endpoints. When the muscle contracts, the displacement vector

Since the facial control point might be registered by other muscles, the final displacement, the total of the displacement by each muscle, is given by

Finally the deformed model by the muscle contractions is calculated by the product of the basis functions

In Figure

Six canonical facial expressions by muscle contraction.

Disgust

Anger

Surprise

Fear

Sad

Happy

Amounts of muscle contraction for each expression. If the amount is positive, the muscle stretches and if negative the muscle shrinks.

Surprise, fear, disgust

Anger, happy, sad

First column: the source model and expressions. Second column through the last column: the cloned expressions. Models have different shapes but expressions are well represented.

The proposed parameterization of facial expression has advantages in terms of storage size. The conceptual storage size required to store the animation data is given by

Meanwhile for a traditional system that simply stores the displacement vector at every vertex, it is given as

So the required storage size is decreased by the ratio

As this formula indicates, the proposed method requires less storage and has much lower memory footage if the size of the mesh is very large and the number of the expressions exceeds the number of the control points. It is a possible scenario since highly detailed facial animation might require a large number of blendshapes, for instance, the facial animation of Gollum in the feature film.

Character animation specifically facial animation requires continuous deformation in animation time frame.

Key framing is a common technique to continuously deform a model by interpolating key framed poses or channel data. The various methods of interpolation and extrapolation control the visual effects of the continuous deformation in the animation time frame. By using the proposed parameterization, the displacement of each facial control point can be set directly by tweaking the position at each key-frame. It is also possible to set the facial control points indirectly by blending canonical expressions described in Section

The deformation by the limited control points requires low computational cost since only the sum of the scalar products of the basis functions

The elastic deformable surface model has been used as a deformation tool especially for elastic organic models. We have introduced a novel parameterization of facial expression using the elastic deformable surface model. The parameterization uses a small number of facial control points to deform a higher resolution surface smoothly and continuously.

In order to obtain parameters for each expression two approaches are introduced. The first approach retrieves parameters directly from existing models by least square minimization method. The parameter can also be created from scratch by moving each control points using the imagination of the animator.

The other approach indirectly obtains parameters by manipulating the facial muscles. The method by the muscle contraction is more intuitive and less daunting task compared with the former method.

The obtained facial parameters can be applied on other target model even if the mesh structure (number of vertices, number of triangles, and connectivity) is different from the source model. A key-framed facial animation can seamlessly use the proposed parameterization. The parameter values could be provided from mocap data.

The method could be used as a postprocessing tool to compress existing animation data, since it requires less storage especially for highly complex mesh objects with huge articulation data without sacrificing the quality of the original animation as much as possible.

Future research could explore the automated detection of the control points on the face model. Several heuristic approaches have been studied [