A Novel Medical Freehand Sketch 3D Model Retrieval Method by Dimensionality Reduction and Feature Vector Transformation

To assist physicians to quickly find the required 3Dmodel from themassmedicalmodel, we propose a novel retrievalmethod, called DRFVT, which combines the characteristics of dimensionality reduction (DR) and feature vector transformation (FVT) method. TheDRmethod reduces the dimensionality of feature vector; only the topM low frequency Discrete Fourier Transform coefficients are retained. The FVT method does the transformation of the original feature vector and generates a new feature vector to solve the problem of noise sensitivity. The experiment results demonstrate that the DRFVTmethod achieves more effective and efficient retrieval results than other proposed methods.


Introduction
The medical 3D model retrieval becomes a hot research topic due to the rapid development in clinic and teaching.The medical 3D model can not only show doctors the anatomical structure of a particular part inside the human body, but also reveal the function of human organs to a certain degree [1].The current diagnostic techniques, especially in the cases which cannot immediately get the diagnosis, need the auxiliary diagnosis by previous similar cases.At this time, 3D model retrieval technology is applied to obtain the similar 3D model from the database.In the face of vast numbers of medical 3D models, how to find the required 3D model rapidly and extract the valuable information accurately has become an urgent issue [2].
The most popular way for retrieving 3D models is example-based paradigm [3], where the user provides an existing 3D model as query input and the retrieval method can return similar 3D models from the database.However, it is difficult for a user to have an appropriate example 3D model at hand.An alternative way is to use a 2D sketch as a query where users can describe a target 3D model by quickly drawing it.But a 2D sketch is merely a coarse and simple representation which only contains partial information of an original 3D model.Hence, it is more challenging to realize freehand retrieval [4] than example-based retrieval.
For the freehand sketch 3D model retrieval how to create efficient feature descriptors [5] is the most important part.Several freehand sketch 3D model retrieval methods have been proposed recently.Funkhouser et al. [6], Chen et al. [7], and Li et al. [8] extracted feature descriptors from the outline to describe the query sketch and 2D views of 3D models, such as the spherical harmonics descriptor, light field descriptor, and shape context features.Recently, Wang et al. [9] provided a sketch-based retrieval approach by utilizing both global feature and local feature.In the practical applications, we find that these features are not effective and efficient in evaluating the relevance between the query sketch and 3D models.There exist two main disadvantages for the current freehand sketch 3D model retrieval methods.Firstly, the traditional Fourier Transform [10] only considers describing the model and the argument of the coefficient.Compactness [11] and simplicity of feature descriptors are necessary for minimizing the storage overhead and the computing time.Therefore, it is necessary to describe the outline by a limited number of coefficients in the frequency domain.Secondly, a freehand sketch is a simple line drawing which has a high level of abstraction, inherent ambiguity, and unavoidable noise [12].For effective retrieval, the feature descriptors should be robust to noise and invariant to transformations.
To tackle these problems, this paper proposes a novel freehand sketch 3D medical model retrieval method, which combines the characteristics of dimensionality reduction (DR) and feature vector transformation (FVT) method, called DRFVT.The DR method reduces the dimensionality of feature vector; only the top  low frequency Discrete Fourier Transform (DFT) coefficients are retained.These  coefficients are able to provide all of the abovementioned requirements and obtain good efficiency in retrieval [13].The FVT method does the transformation of the original feature vector and generates a new feature vector to solve the problem of noise sensitivity.On this basis, the similarity between the sketch and the 3D model is calculated [14].To evaluate our method, we test our method on the public standard data set and also compared with other leading 3D model retrieval approaches [15].The experiments demonstrate that the DRFVT method is significantly better than any other retrieval techniques.

Our Method
This paper researches on how to get user's retrieval intention and the feature extraction of 2D freehand sketch.Users express their retrieval intention freely by freehand sketch.In order to realize the 3D model retrieval, this paper calculates the similarity between 2D sketches feature vector and 3D model projection view feature vector.The framework of the medical freehand sketch 3D model retrieval method is proposed, as shown in Figure 1 [16,17].
The 2D freehand sketch framework contains five modules, respectively, 2D freehand sketch input module, 2D outline representation module, DR method module, FVT method module, and 2D sketch feature vector set module.The 2D freehand sketch input module would get user's retrieval intention with hand drawing or mouse drawing.The 2D outline representation module extracts the outline of sketch based on the eight-direction adaptive tracking algorithm [18].The DR method module reduces the dimensionality of feature vector; only the top  low frequency Discrete Fourier Transform coefficients are retained.The FVT method module does the transformation of the original feature vector and generates a new feature vector to solve the problem of noise sensitivity.The 2D sketch feature vector set module contains the feature vector set generated by the DR method and the FVT method.
The medical 3D model framework contains four modules, respectively, medical 3D model set module, 2D projection view of 3D model module, feature vector extraction module, and 2D projection view feature vector set module.The medical 3D model set module contains 300 3D model files of medical organs from the Princeton Shape Benchmark [19].The 2D projection view of 3D model module chooses and generates 2D projection views of each 3D model.The feature vector extraction module creates feature descriptors which describe the structural information and recognize interior content from 2D projection views of 3D model.The 2D projection view feature vector set module contains the feature vector set of 2D projection views of medical 3D model, which we use to compare with the 2D sketch feature vector set.
2.1.Feature Extraction of 2D Sketch.In the general sketch retrieval method, the sketches are composed of element figures, and the relationship between the element's figures is complex.For the 3D model retrieval method, the sketches are not expressed by simple element figures, because the outline of the sketch is complicated and has abundant feature information; it cannot be expressed by simple element figures.Therefore our method firstly extracts the outline of the sketch based on the eight-direction adaptive tracking algorithm and parameterizes the outline to obtain a complex discretetime periodic signal whose period  equals the number of points.Secondly, we adopt the DR method to reduce the dimensionality of feature vector, which obtains good efficiency in retrieval.Finally, we propose that the FVT method does the transformation of the original feature vector, which uses the new feature vector to solve the sensitive noise problem [20][21][22].

2D Outline Representation.
For the sketches submitted by users, this paper extracts the outline of the sketch based on the eight-direction adaptive tracking algorithm.Let (, ) represent the original image [23] and (, ) represent the outline binary image.As shown in Figure 2, we search the outline in the 3 × 3 region with point  as centre counterclockwise.
This algorithm selects the first point of (, ) as  .If there are no other points in direction 2, 3, 4 of   counterclockwise, we save   into the outline binary image (, ).The algorithm is described as follows.
Step 2. Scan the original image (, ) line by line in order, from top to bottom and left to right.We obtain the starting point   of the outline and initialize the current point   with  .Step 4. Judge whether   coincides with the starting point  ; if yes then exit; otherwise returns Step 3.

The DR Method Based on Fourier Transform
(1) Method Description.The outline of the sketch is closed and cyclical.We can use Fourier Transform to describe its characteristics.First of all, we consider the outline of an object as a discrete-time complex periodical signal  = ⟨ 0 ,  1 , . . .,  −1 ⟩ and define   =   +   ( = √ −1), where   and   are the real coordinate values of the th ( = 0, . . .,  − 1) sampled point.The signal  is then mapped to the frequency domain by the DFT [24], where   and   are the module and the argument of the th DFT coefficient, respectively.
(2) Invariance Requirement.In order to guarantee the translation, scaling, rotation, and starting point invariance, we have to modify the DFT coefficients accordingly.When a change is in the position, size, orientation of the object, or the initial point used to parameterize the outline, the DFT coefficients are modified [25].In Table 1, we show how normalized DFT coefficients    =         satisfying the invariance requirements can be obtained.
If one wants to achieve rotation and starting point invariance, the two corresponding modifications have to be integrated, leading to For rotation and starting point invariance, the derivation is as follows [26][27][28].
We consider the original outline signals  and   , where   is obtained from  by rotating each point counterclockwise Thus,    can be expressed as In particular, when  is 1 or −1,   1 and   −1 are By referring to Table 1, it is obtained that By simply performing the Inverse DFT on normalized DFT coefficients, we obtain a modified normalized signal   which satisfies all the requested invariance (2/) ( = 0, 1, . . .,  − 1) .(7) (3) Dimensionality Selection.In order to concentrate the outline information of the sketch, we use only low frequency DFT coefficients.In particular, we keep only the  ( ≪ ) coefficients whose frequency is closer to 0. The DR method uses the Bartolini et al. [29] spectral characteristics () to determine the value of .The energy of the signal retained by the  coefficients is defined as The choice of a suitable value for  has to trade off the accuracy in representing the original signal with the compactness and the extraction efficiency.

The FVT Method. The feature vector extraction algorithm is as follows.
Let the centroid be ( 0 ,  0 ); the distance   from the outline point to the centroid is shown as below: Then the distance from each of  outline points to the centroid ( 0 ,  0 ) generates the  dimension feature vector  = ( 1 ,  2 , . . .,   ).

Similarity Comparison.
From the user input retrieval intention, retrieving the user's satisfactory models requires comparing the feature vector distances between the input sketch and the 3D model.The similarity of models is determined by the differences between feature vectors [30][31][32][33][34].
In order to realize the medical freehand sketch 3D model retrieval, this paper compares the similarities of the medical 3D model based on Euclidean distance [35], where ,

Experiments and Results
We implement the medical 3D model retrieval method in C++ under Windows.The system consists of a computer with an Intel Xeon CPU E5520@2.27GHz and 12.0 GB of RAM.On average, each 3D model takes 6.1 seconds to extract features; before the sketch retrieval we can extract and store the 2D projection view feature vector.Moreover the 2D freehand sketch takes 0.06 seconds to extract features.The freehand sketch retrieval takes 0.1 seconds in the medical 3D model retrieval system.For performance evaluation, we select 300 3D model files of medical organs from the Princeton Shape Benchmark [18] (http://shape.cs.princeton.edu/benchmark/); each 3D model   is manually classified based on 6 categories ("Body" (130 images), "Heart" (30), "Lung" (25), "Ovary" (13), "Kidney" (22), and "Liver" (18)).The 3D models do not belong to any category and were assigned to a default class (62).Table 2 shows some 3D models in the data set, along with their category.
In this paper, we provide users' interface with hand drawing or mouse drawing, which is convenient for users to draw and modify the freehand sketches.As shown in Figure 3, the left is the tablet which is used for hand drawing or mouse drawing, and the right shows 3D model retrieval results.

The 2D Sketch Feature Extraction Experiments
3.1.1.Extract the Outline of Sketch.This paper extracts the outline of sketch based on the eight-direction adaptive tracking algorithm.It is assumed that when the sketch is drawn, the mouse point sequence is stored in (, ).After that (, ) is applied to generate the outline binary image (, ); Figure 4 shows a freehand sketch and the outline extraction.parameterize  points outline.The DR method reduces the dimensionality of feature vector; only the top  low frequency DFT coefficients are retained.We consider the spectral characteristics () of the data set to determine the value of .
As an example, Figure 5 plots the value of ()/() for the heart 3D model set.From the graph, when choosing  ∈ [16,64], the retained energy varies from 75 percent to 91 percent of the total.On the other hand, when obtaining ()/() = 95%,  should be equal to 512.The  value will represent 512 outline points and then construct 512 vertices of graph, which is not obviously appropriate.Therefore choose the smaller , such as  = 32;  is a power of 2; we can easily use the Fast Fourier Transform.
Experiments of the heart 3D model set show that the effectiveness of using  = 16 is much lower than  = 32, whereas increasing the value of  to 64 does not lead to further improvements.Thus, in this experiment, we will use  = 32; the result of the DR method is as shown in Figure 6.Using 32 generated outline points calculates the distance from each point to the centroid ( 0 ,  0 ) and then generates 32D feature vector  = ( 1 ,  2 , . . .,  32 ).3.1.3.The FVT Experiments.This paper considers the outline of the transformed human body, such as widening or cutting the human body.We select the  = 64 value using the DR method and calculate the distance from the outline point to the centroid   = √(  −  0 ) 2 + (  −  0 ) 2 .The feature vector  = ( 1 ,  2 , . . .,  64 ) is shown as follows.Figure 7 shows the original human body feature vector, Figure 8 shows the widening human body feature vector, and Figure 9 shows the cutting human body feature vector.Figures 8 and 9 show that the feature vectors have certain sensitivity to noise.Therefore we transform the existing feature vector  = ( 1 ,  2 , . . .,   ) and generate new feature vector   = (  ,  +1 , . . .,   ,  1 , . . .,  −1 ) to solve noise sensitive issue.

Algorithm Performance.
For evaluation purposes, any 3D model in the same category of the query is considered relevant to that query, whereas all other models are considered irrelevant.To measure the retrieval effectiveness, we considered classical precision () and recall () metrics [36,37] averaged over the set of processed queries.They are defined as follows: precision = relevant correctly retrieved all retrieved , recall = relevant correctly retrieved all relevant . (11)

Effect of the DR Method.
In our first experiment, we compare the DR method with the one based on the extraction from the boundary of the  points having the highest curvature values, hereafter called MAXC [38].We do not consider methods that reduce dimensionality by simply resampling the original boundary every / points since this easily leads to missing significant shape details.Results in Figure 10(a) clearly show that DR method consistently outperforms MAXC in precision and that it is always positive even for higher recall levels.A similar trend can be also observed on specific query, as shown in Figures 10(b) and 10(c).

Effect of the FVT Method.
In order to consider the effects of the FVT method, we compare the DRFVT method with the other two methods.One method uses the DR method and discards the FVT method, hereafter called DR-NoFVT.The other method uses the MAXC and the FVT method, called MAXC-FVT.For one specific query, Figure 11 shows the / graph for the query "Human Body" and Figure 12 for visualization of retrieval results.The experiment results demonstrate that the FVT method is very useful, and the DRFVT method gets better precision than other proposed methods.

Discussion and Conclusion
In this paper, we propose a novel medical freehand sketch 3D model retrieval method DRFVT which uses dimensionality reduction and feature vector transformation.It firstly provides a convenient interface to satisfy the user's design process and extract the outline of sketch based on eightdirection adaptive tracking algorithm.Secondly, the DR method reduces the dimensionality of feature vector; only the top  low frequency DFT coefficients are retained.These  coefficients are modified so as to achieve the invariance and then stored in the database.Finally, The FVT method is proposed, which generates a new feature vector to solve the problem of noise sensitivity.On this basis, the similarity between the sketch and the 3D model is calculated; thus the final retrieval results would be presented to the users.The experiment results demonstrate that our method achieves retrieval results more effectively and efficiently than other previously proposed methods.
The future of our work is to develop a user interaction feedback mechanism.After the users submit the query sketch, the system first provides a list of retrieved 3D models.Then, the users can refine the retrieval results by selecting the 3D models which they take as the good results.This feedback mechanism can not only provide more desirable retrieved 3D models to the user, but also enhance the user interaction just by making some easy choices.It would largely improve the effectiveness of our retrieval system [39,40].

Figure 1 :
Figure 1: The framework of the medical freehand sketch 3D model retrieval method.

Figure 2 :
Figure 2: The point  eight directions.

Figure 3 :
Figure 3: The medical freehand sketch 3D model retrieval system.

Figure 4 :
Figure 4: The effect of the outline extraction.

Figure 6 :Figure 7 :
Figure 6: The result of the DR method for the heart 3D model.

Figure 12 :
Figure 12: Results for the query "Human Body."

Table 2 :
Sample 3D models based on 6 categories.
by a constant factor  0 and by shifting the starting point by  0 positions,    =  − 0   0 .The corresponding DFT coefficients are