Displaying a variety of fabrics on a customized character could help customers choose which fabric is more suitable for themselves and help customers choose clothing. However, it is not an easy task to show realistic garment on customized virtual character. As a result, we propose a stable finite element method (FEM) model which is stable to approximate stretching behaviors. At first, we measure four kinds of cloth materials with measurement techniques to research elastic deformations in real cloth samples. Then, we use the parameter optimization method by fitting the model with measurement data. For promoting the display of realistic fabrics, we recover 3D human in shape and pose from a single image automatically. Human body datasets are constructed at first. Then, CNN-based image retrieval in shape and skeleton-based template matching method in pose are combined for 3D human model recovery. To enrich human body details, we synthesize the human body and 3D face with spatial transformation. We compared our proposed method of recovering 3D human from a single image with the state-of-the-art methods, and the experimental results show that the proposed method allows the recovered virtual human to put on garment with different fabrics and significantly improves the fidelity of virtual garment.
Unlike physical try-on, which requires try on of several clothes before the shopper makes decision, the virtual try-on system allows shoppers to visualize what a garment might look like on themselves before purchasing and enhances the shopper’s shopping experience. Thus, enabling realistic clothes try on the recovered human body of their own has the potential prospect. Some virtual try-on systems, on the other hand, do not typically account for the effects of fabric materials. With the right set of parameters, these systems could simulate real cloth to a high degree of fidelity. But some parameters should be tuned based on the animator’s intuition about the fabric. And during this tuning process, it is difficult to tell which models and which parameters are giving results more like the real fabric. In addition, almost all of the virtual try-on systems assume that the users have selected a predefined set of avatars or have captured accurate measurements of their own bodies through 3D scanning. And these avatars or captured bodies do not have the generated 3D human face details.
In this study, we consider the problem of exhibiting realistic clothes on recovered virtual character with less input information, such as the capability that enables users to virtually try on different material clothes by a single photograph of themselves. Instead of previous cloth models, which are inaccurate for stretching materials, we introduce a stable FEM cloth model to approximate the stretching behaviors of various materials. Then, elastic deformations on stretching in real cloth samples are studied using stretching measurement and parameter optimization method. A recovery human body method is needed to show realistic fabric. Instead of the scanned human model or human shape relying on multiview images, we recover the human model in shape and pose from a single image. For improving the accuracy of garment models with animated characters, we enhance the visual realism effects of digital human models by incorporating human body with 3D face.
And the contributions of this paper can be summarized as follows: (1) We propose a stable FEM cloth model which could approximate various materials accurately and stably. With stretching measurement and parameter optimization method, we find the stiffness parameters of the elastic model (in Section
Our work builds on previous efforts including cloth simulation, cloth parameter estimation, and 3D human recovery.
Cloth simulation is a traditional research problem in computer graphics. Early physics-based cloth simulation adopts the particle system. Macklin et al. [
In addition to discrete models like mass-spring, there are also continuous models based on FEM [
As we know, garment modeling is built upon cloth simulation. Some methods start from the 2D design patterns; then, they use physical simulation or iterative optimization of related parameters to stitch the planar pattern and obtain the desired realistic 3D garment. Umetani et al. [
Despite a large amount of works on cloth simulation models, little work has been done on estimating the parameters of these models which could match the behavior of real fabrics. To this end, previous cloth parameter estimation works [
Another approach to model the cloth is to fit cloth parameters by experimental data. Wang et al. [
Traditional methods that capture full skeletal motion mostly rely on multiview camera systems. Baak et al. [
There are lots of studies on estimating 3D human body from multiview images or video sequences [
All human body recovery methods do not take into account face details; thus, we provide a fully automatic method for estimating a 3D human model in pose and shape from a single image and combining it with 3D face. Although constructing 3D human model dataset is a bit trivial, we only need to do it once.
The specific process of our system is shown in Figure
Process of exhibiting realistic garment.
A list of symbols used in this study is presented in Table
Symbol list.
|
|
---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
The 3D human model
We have attempted to accelerate stability and improve the accuracy for continuous models based on FEM. While realistic fabric is possible to be simulated with existed techniques, it is time consuming to adjust many parameters to achieve an authentic appearance of the particular fabric. In this section, we first propose a stable FEM cloth model that is stable and approximates stretching behaviors. Then, we use the parameter optimization method to study the real cloth samples of four kinds of materials. And we apply the optimization parameter of four different materials on our proposed stable FEM cloth model.
We adopt the FEM for in-plane elastic force in this study. And assume the nonlinear Green–Lagrange strain tensor as the strain metric. To avoid the process oscillation caused by the elastic force during the simulation process and to accelerate the convergence speed of the simulation, this model applies a damping force on each particle according to the cloth property.
It is known from the elastic mechanics that the deformation caused by the internal tension of the cloth is described by the strain
Among them,
The deformed points are also represented by the same weight:
To facilitate the calculation of the formula, it can be assumed that
According to equation (
When equation (
In the process of garment simulation, the elastic force is easy to cause oscillation effect, and the simulation process cannot be stabilized quickly. To solve the problem, this study introduces a linear damping force model, which is linearly related to the velocity of the particle:
Among them,
We adopt the iterative optimization of related parameters to stitch the planar pattern and obtain the desired realistic 3D garment. The initial 2D garment which is around the human body is shown in Figure
Comparison results between cloth models during stitching process. (a) Before stitching. (b) During stitching. (c) Comparison results between stabled FEM and FEM cloth model.
We know that a given model describes a particular piece of cloth by fitting the model to the measurement data. It could adjust its parameters to minimize the difference between the model’s predictions and the measured behaviors. In this study, we do this by solving an optimization problem, measuring cloth behavior under conditions, and estimating cloth deformation models following an incremental parameter fitting procedure. Like other cloth testing systems, we just focus on tensile forces, because it is hard to measure compression forces in a sheet repeatedly.
To obtain the stiff parameters, we design experiments for stretching, shearing, and bending behaviors of cloth, which demonstrate a sufficient set of cloth behaviors. The square fabrics on the normalized size of 25 cm
We select four kinds of common fabrics for testing, which are silk, fleece, denim, and linen, respectively. They exhibit distinct elastic behaviors, for example, rib knit material has little effect on linen likely due to its vertically rigid structure. And we have first measured the tensile stretching, shearing, and bending force-elongation curves with our proposed method on the fleece sample, as shown in Figure
Stretching, shearing, and bending behaviors of real fleece fabric. (a) Original fabric. (b) Stretching behavior. (c) Shearing behavior. (d) Bending behavior.
Our optimization method is to find optimal parameters by minimizing the difference between captured features and simulated features. And the optimization formulation is as follows:
Among them,
The parameter optimization test was performed based on the stable FEM cloth model, and we compared the motion of cloth in simulation with measured real fabric motion by parameter fitting incremental. The cloth parameters of four fabrics are obtained after
Cloth parameters of four kinds of real fabrics.
Parameter material | Silk | Fleece | Denim | Linen |
---|---|---|---|---|
|
1.0 |
6.9376 |
7.7720 |
8.4776 |
|
50 | 19.3822 | 20.2884 | 20.1119 |
|
50 | 69.654 | 32.6492 | 50.830 |
Simulation results of four kinds of fabrics. (a) Fabric of silk. (b) Fabric of fleece. (c) Fabric of denim. (d) Fabric of linen.
The recovery of 3D human pose from 2D is critical in many applications. However, most existing techniques for human shape recovery rely on multiview images and are insufficient to constrain a 3D shape from a single image. Thus, we provide a fully automatic method to recover a 3D human model both in shape and pose from a 2D image. First, we construct human model dataset using a 3D generative model called SMPL. Then, we combine the CNN-based image retrieval and skeleton-based template matching method to match the shape and pose in the human model dataset. We add a last step, which synthesize independent human body with 3D face.
To reconstruct the 3D human model from a single image, we prepare human model dataset in advance. In this study, we employ the recent SMPL model introduced by Loper et al. [
Generation of human model dataset by shape and pose parameters.
When recovering the 3D model that is closest to the 2D human in image, a CNN-based image retrieval method could not accurately estimate the 3D human body model due to the particularity of the human body posture. Using a skeleton-based template matching method requires a pose comparison with each of the 3D human bodies in the human model dataset Preprocess: for each human body in the human body dataset Step 1: semantic segmentation on the input image Step 2: the human skeleton for the input image
Process for recovering shape and pose of the human model.
For recovering the 3D model that is closest to the 2D human in the input image, it is essential to find the model that is closest in shape to the 2D image. CNN-based image retrieval could help us retrieve the candidate images that have the most similar content to the query image. The key concern for CNN-based image retrieval is feature extraction. It is important that the features extracted from the
We adopt the popular pretrained CNN model VGG-16, which could get more discriminative representations for object recognition and be used as the basic model for image retrieval [
For recovering the 3D human model that is similar in pose from the input image
Among them,
Overall and partial human skeleton. (a) Partial human skeleton in coordinate system. (b) Overall human skeleton.
From Section
In 3D space, when the human body and the 3D face are synthesized, the human body without spatial transformation is used as a reference object, and the spatial transformation is performed on the 3D face. The transformation calculation is shown as follows:
The synthesis process is shown in Figure
Synthesis details for 3D face and human body.
With our proposed stable FEM cloth model in Section
Displaying various fabrics on customized character allows shoppers to visualize the effects of trying on various fabrics on their own. This is a good application prospect for virtual try-on. In this section, we will show garments that are made up of four fabrics and try them on the customized human model.
Our initial 3D garments are created from 2D panels. However, the 2D panels are triangulated using a Delaunay algorithm since we use triangular meshes to represent garment surfaces. And seam lines are explicitly specified by choosing pairs of panel boundary edges. We apply the cloth parameters obtained in Section
Figure
Displaying results of four kinds of fabrics. (a) Fabric of silk. (b) Fabric of fleece. (c) Fabric of denim. (d) Fabric of linen.
Given a single image, our goal is to reconstruct the 3D geometry and texture of a clothed human while preserving the detail present in the image. Existing state-of-the-art virtual try-on systems require a depth camera for tracking and overlay the human body with the fit garment. Saito et al. [
One of the image-based virtual try-on result is shown in Figure
An example of image-based virtual try-on. (a) Input image. (b) Result of 3D model.
All of our experiments were tested on a 3.4 GHz AMD Phenom II x4965 processor machine, with 4 GB of RAM, and a NVIDIA GTX260 graphics card. For cloth simulation, we use the stable FEM cloth models with continuous collision detection and constraint-based collision response on this configuration. And we implement parallel virtual garment display on custom character on CUDA.
Compared to the previous human model recovery method, our proposed method has some advantages. The benefits include the follows: (1) Requiring no visual cues, such as shading cues, internal edges, or silhouettes, to promote the fitting of the template model. (2) There is no need to use strong priors, such as a manually defined skeleton or body parts, which facilitates recovery of the human model accurately. (3) Our proposed method achieves enriched face details. And the comparison results on human image are shown in Figure
Comparison results of human recovery between the inferring 3D shape [
Another comparison results on human image are shown in Figure
Comparison results of human recovery between Human Mesh Recovery [
Figure
Comparison results of human recovery between Human Mesh Recovery [
The following is the comparison results of image-based virtual try-on. Yang et al. [
Comparison results of image-based virtual try-on. (a) Input image. (b) The result of ref [
For displaying dynamic effect of the fabric, we added wind into the natural environment. When the human body is in wind-blown environment, the clothe fabric will swing to a certain extent with the size and direction of the wind. In order to simulate the true natural wind blowing effect, it is necessary to randomly take the direction and size of the wind to obtain irregular wind. And we select the garment of linen material as virtual try-on. Figure
Result of draped garment in wind environment. (a) Windless environment. (b) Wind environment.
We have presented a procedural method that exhibits realistic garments on a recovered 3D human model. It allows general pose of virtual human model to put on garment for displaying realistic fabrics. A stable FEM cloth model is proposed, and the stiffness parameters of elastic models are obtained by stretching measurement and the parameter optimization method. We show that four simulation fabrics which draped on recovered 3D human model. This method is easy to implement and significantly improves the fidelity of virtual garment, which make evident that our system has great potential value, such as commercial applications like virtual dressing or interactive applications like VR game.
Although our method has some limitations, it points out the direction of our future research. The first limitation is that we have found the stiffness parameters of the elastic model for four real fabrics. And the parameters of elastic model for more fabrics are waiting for research. Another limitation is that the accuracy of our recovered human body depends on the human model dataset. In the future, we will devote a lot of time to remove the two limitations and so we could get more reasonable and realistic garment immediately.
The research data used to support the findings of this study are available from the corresponding author upon request (corresponding name: Yanjun Peng, e-mail:
The authors declare that there are no conflicts of interest.
The authors would like to thank Prof. Zhigeng Pan at Hangzhou Normal University for his valuable comments and Assoc Prof. Mingmin Zhang at Zhejiang University for discussion. This work was supported by the National Natural Science Foundation of China under Grant no. 61976126, the National Key Research and Development Program of China under Grant no. 2018YFB1004902, the Natural Science Foundation of Shandong Province under Grant no. ZR2019MF003, and the Natural Science Foundation of Shandong Province under Grant no. ZR2017FM054.