Challenges and Methodologies of Fully Automatic Whole Heart Segmentation: A Review

Whole heart segmentation from magnetic resonance imaging or computed tomography is a prerequisite for many clinical applications. Since manual delineation can be tedious and subject to bias, automating such segmentation becomes increasingly popular in the image computing field. However, fully automatic whole heart segmentation is challenging and only limited studies were reported in the literature. This article reviews the existing techniques and analyzes the challenges and methodologies. The techniques are classified in terms of the types of the prior models and the algorithms used to fit the model to unseen images. The prior models include the atlases and the deformable models, and the fitting algorithms are further decomposed into four key techniques including localization of the whole heart, initialization of substructures, refinement of boundary delineation, and regularization of shapes. Finally, the validation issues, challenges, and future directions are discussed.


BACKGROUND AND MOTIVATION
According to World Health Organization (WHO), cardiac vascular diseases are the number one cause of deaths globally.An estimated 17.3 million people died from cardiac vascular diseases in 2008, accounting for 30% of deaths around the world [1].To improve diagnosis and treatment of cardiac vascular diseases, there has been much emphasis on developing and introducing novel technologies in medical imaging and image computing to clinical practice and medical research of cardiology [2][3][4][5].The morphological and pathological information from the three-dimensional (3D) medical imaging has made revolutionary impacts on cardiology, including the cardiac computed tomography (CT) or computed tomography angiography (CTA) [6], and the cardiac magnetic resonance imaging (MRI) [7].The 3D data, with good contrast and wide fields of view, provide valuable anatomical information of the heart.To enable the development of new clinical applications and thus improving cardiology, accurate extraction and precise interpretation of this anatomical information becomes particularly important.
(2) The right ventricle blood cavity (referred to as right ventricle or short as RV).
(3) The left atrium blood cavity (referred to as left atrium or short as LA).(4) The right atrium blood cavity (referred to as right atrium or short as RA).

372
Challenges and Methodologies of Fully Automatic Whole Heart Segmentation: A Review The myocardium of the left ventricle is also of interest in the whole heart segmentation: (5) The phrase myocardium (MYO) is commonly used to refer to that of the left ventricle, and endocardium (ENDO) and epicardium (EPI) are used to refer to the endocardial surface and epicardial surface of the left ventricle, respectively.The great vessels include the aorta and the pulmonary artery.Due to the different fields of view, the coverage of the great vessels can be different in different scans.To have a consistent definition of the great vessels across different subjects, the major trunks of the vessels can be defined as follows: (6) The aorta is defined as the beginning trunk of the ascending aorta from the aortic valve to the superior level of the atria (referred to as ascending aorta or AO); for example, the mean length of this trunk is about 41.8 mm as we measured in 25 healthy subjects.(7) The pulmonary artery is defined as the beginning trunk between the pulmonary valve and the bifurcation point (referred to as pulmonary artery or PA); for example, the mean length of this trunk is about 41.9 mm as we measured in 25 healthy subjects.
Other (parts of) great vessels were also segmented, including the descending aorta, the pulmonary veins, the superior and inferior vena cava, and the coronary sinus [8].
Whole heart segmentation results are important to clinical studies and applications, including the following:

•
To extract functional indices such as chamber volumes, ejection fraction (EF), myocardial mass, myocardial thickness/thickening/motion, etc.These functional indices are important measures in clinics [4,[48][49][50][51][52][53][54].For example, the EF determines how well the heart is pumping out blood and is important in diagnosing and tracking heart failure [51,52]; the myocardial motion is linked to the contractility of the heart and is shown to be strongly correlated to the remodeling of post cardiac resynchronization therapy [53,54].It is also anticipated that the functional analysis of the whole heart may detect subtle functional abnormalities or changes of the heart, and this is important to patients who otherwise have normal systole in ventricles but are suspected to have abnormal functions in other regions [55,56].

•
To enable 3D surface rendering, as shown in Figure 1 (d).The rendering result has a wide range of applications such as investigating the congenital cardiac malformations in a morphological study [57].Also, the latent definition of the landmarks from the whole heart segmentation result can define the short axis (SAX) and long axis (LAX) to display any desired two-dimensional (2D) views, such as the four-chamber view and SAX view [58].The geometrical information from the whole heart segmentation is useful in guiding interventional procedures by fusing the 3D surface of the heart constructed from an anatomical scan to the real-time interventional scans [59].For example, in the radio frequency ablation surgery, the surgeons can obtain a better relative position of the interventional catheter tip to the heart by setting certain transparence to the 3D surface model.Also, the landmarks from the segmentation result enable fast navigation of the catheter tip during surgeries, such as accessing the pulmonary veins in the radio frequency ablation surgery.
The goal of this work is to review the whole heart segmentation studies in the literature, and discuss the challenges and methodologies for fully automating such segmentation.It should be noted that this review does not cover articles and methods for ventricle segmentation or isolation of the heart, topics already reviewed in previous papers [47,[60][61][62][63][64][65][66].The rest of this paper is organized as follows.Section 2 presents the major challenges of the fully automatic whole heart segmentation.Section 3 reviews and analyzes the methodologies of the existing works.The databases used to search the papers include the PubMed (http://www.ncbi.nlm.nih.gov/pubmed), the IEEE Xplore (http://ieeexplore.ieee.org/Xplore),Google Scholar (http://scholar.google.com/),and all the papers published in IEEE Transaction on Medical Imaging and Medical Image Analysis since 2002.Section 4 discusses the validation issues.Section 5 discusses the remaining challenges and potential solutions.Section 6 concludes this work.

CHALLENGES
Manually delineating all the substructures of the whole heart is labor-intensive and tedious.More importantly, manual delineation is subject to intra-and inter-observer variations.Therefore, automatic whole heart segmentation has become increasingly popular in recent years.This fully automated processing is particularly useful in the clinical studies where a large number of images need segmentation.However, fully automatic whole heart segmentation is difficult.The challenges are the following.
Firstly, the heart organ, with multiple chambers and great vessels, is complex in geometry.The heart shape can vary significantly in different subjects or from the same subject at different cardiac conditions.This shape variation is especially evident when pathological cases are involved.For example, Figure 2  Reproduced with permission [23].
segmentation, even when the statistical shape priors are used to estimate the unseen cases.Unseen refers to the medical images that need segmentation.It is practically difficult, if not impossible, to capture all possible heart shapes from different pathologies using a prior model trained by a limited training dataset.Secondly, some boundaries between anatomical substructures are visually indistinct based on the intensity distributions, namely texture patterns, of the medical images.This happens in situations including the following: (1) Some substructures are not separated by physical tissues or structures; instead their boundaries are defined based on anatomical knowledge.These boundaries include the valve planes separating the atria and great vessels from the ventricles, the boundaries between the left atrium and pulmonary veins, and the boundaries between the right atrium and superior/ inferior vena cava.Figure 3 (a) shows an example of a cardiac MRI image, where the tricuspid valve plane was delineated at a wrong location by a traditional atlas propagation and segmentation scheme [29].(2) The intensity distributions between adjacent tissues or substructures are similar or even identical.For example, the intensity levels of the papillary muscle and myocardium are similar while the papillary muscle tends to be included in the ventricular cavities.The intensity of the myocardium is also similar to the adjacent liver and body muscle in cardiac CT data.In cardiac CTA, the intensity distribution of the unenhanced cavities, such as the right ventricle, is also similar to that of the myocardium.The cardiac CTA image in Figure 3 (b) shows that the intensity distributions of the myocardium, the right ventricle, and the adjacent tissues of the body muscle and liver are similar.It is therefore difficult to separate these regions from each other if a segmentation method does not utilize prior models for guidance and solely relies on the image intensity.Figure 3(c) shows an example of incorrect segmentation of the myocardium by a traditional atlas propagation and segmentation method, due to the indistinct boundary [29].(3) The thin walls of the atria and vessels, as well as the partial volume effects, make the boundaries between these substructures unclear when they are adjacent to each other.As illustrated in Figure 3 (d), an axial view of a cardiac MRI volume shows that the boundaries between the left atrium, the right atrium, and the aorta are unclear.
Due to these indistinct boundaries, the lower level techniques, which do not use shape priors for guidance [62][63][64][65], have difficulty achieving the fully automatic whole heart segmentation.
Finally, due to the complex motions and blood flow within the heart, the imaged data may contain heavy motion artifacts, intensity inhomogeneity, and noise.This contributes to the unsmooth and suboptimal delineation of the boundaries.Smoothness constraints and shape regularization are generally adopted to deal with this challenge.

Key Techniques for Automatic Whole Heart Segmentation
Automatic segmentation of complex organs, such as the heart, is generally formulated as a fitting process from a prior model to the unseen images for segmentation propagation [9][10][11][12][13][14][15][16][17][18][19][20][21][22].The prior models contain the segmentation information and are used to guide the segmentation procedure, and the fitting procedure is generally implemented in a hierarchy manner.

Prior model:
The main content of a prior model, also referred to as prior knowledge, can contain information of the heart shapes and important intensity features of the images.The common representations of prior models include the atlas [17,19,29,67] 376 Challenges and Methodologies of Fully Automatic Whole Heart Segmentation: A Review and the deformable model [11][12][13][14][15]68].In the atlas-based segmentation, one can construct an atlas pool, which contains atlases with different heart shapes, and use a multiple classification strategy to improve the segmentation [17,24,69,70].In the deformable model-based segmentation, the statistical shape model (SSM), also known as active shape model (ASM) [71], is commonly used for the segmentation of complex organs [72].When an SSM also includes the statistical information of image intensity distributions, the model is also known as an active appearance model (AAM) [73].The flexibility of an SSM plays an important role in accurate segmentation.As the variability of a statistical model is limited to the available training data, several techniques have been proposed to extend the flexibility of the SSM in the segmentation framework.
Fitting process: The fitting process requires a global localization to perform a deformable fitting.To improve the robustness, several studies proposed to use the hierarchical scheme, where the deformable fitting can be further decomposed into two stages: the initialization of substructures and the refinement of boundary delineation in detail [11][12][13][14][15]19].In the deformable fitting, shape regularization is required to maintain a realistic and smooth heart shape in the resultant segmentation.

METHODOLOGY
This section introduces the key techniques and analyzes the existing methodologies based on the classification of prior models and algorithms of fitting process.The works and the methodologies are summarized in Table 1 and Table 2, and the performance is presented in Table 3 and Table 4.

Prior Model
The prior model in the whole heart segmentation can contain information of the heart shape, the intensity distributions of the medical images, and sometimes the salient features of the landmarks and boundaries [9, 11-17, 19, 22, 62, 63, 68, 74, 75].The representation of such prior knowledge includes the atlas and the deformable model using statistical shape model (SSM).

Atlas
An atlas is defined as an image pair consisting of an atlas intensity image and an atlas label image, as well as other useful prior information.Figure 1 shows an example of the atlas intensity image (Figure 1a) and atlas label image (Figure 1b).The atlas-based segmentation, referred to as atlas propagation and segmentation, employs image registration algorithms to fit the atlas image to the unseen image [19,29,46,67,69], as Figure 4 illustrates.
The atlas intensity image is generally constructed from an image or a set of images from the same modality as the expected unseen images, since it is used as the source image in the registration between the atlas and the unseen images.Original cardiac images can also be used as the atlas intensity images such as the cardiac CTA volumes [16,17,24] and the MRI volumes from the same patient [29].This is also widely used in the brain image segmentation [76][77][78][79], where the brain MRI images are of high quality.In [19,20], the atlas intensity image was constructed from the intensity mean image of a set of cardiac MRI images.
The atlas label image shown in Figure 1b is a label image with segmentation information of all the substructures of interests.The segmentation information will be propagated to the unseen image using the resultant transformation of the registration, to achieve the automated segmentation, as Figure 4 demonstrates.
In atlas propagation and segmentation, the segmentation propagation, i.e., the process of fitting the atlas to the unseen images, is achieved through a voxel-based registration method.In the registration, the atlas and the unseen image are regarded as the source space and the target space, respectively.The registration can use mutual information [80,81] or normalized mutual information [82] as the similarity metric for cardiac images.Therefore, an atlas constructed from one modality or of one intensity distribution can potentially be used in the segmentation of images from the other modalities or of other intensity distributions.
To improve the robustness and accuracy of the registration, the atlas may also include other useful prior information, also known as priors.These priors can be the labels of other substructures of the heart, such as the descending aorta, or the adjacent organs (such as the liver), which may not be the interest of the segmentation itself.This information can be used to assist the registration, for example, to better initialize all substructures in the locally affine registration method [18,19].Other useful priors include the boundaries and anatomical landmarks.For example, the boundary information can be used to adaptively set the status of the control points in the free-form deformation registration, in order to improve the segmentation of the epicardium [19].

378
Challenges and Methodologies of Fully Automatic Whole Heart Segmentation: A Review

2007
Four chambers and Deformable model + Generalized Hough transform and similarity parametric [12] beginning trunks of adaptation + piecewise affine parametric adaptation + search for optimal edge response aorta and pulmonary along normal direction to mesh surface for parametric adaptation of vertices using artery + 3D MRI simulated search + shape constrained deformable model to increase variability of the shape model  Four chambers: The segmentation results consisted of aorta and pulmonary artery as they were constructed in the atlas(es), but with the segmentation results of these great vessels not validated due to lack of gold standard.The volume compared in the Dice is the myocardium of the left ventricle.

Atlas Selection and Multiple Atlas Strategy
In atlas propagation and segmentation, it is important to construct atlases with high quality and to select the best atlas(es) for segmentation propagation.An ideal atlas not only should have a high quality intensity image, but also should have a heart shape similar to the unseen image.This is practically important and represents one of the major challenges for automatic processing of cardiac images, as discussed in Section 2. Since an atlas does not naturally contain the statistical information of shape variations, a solution proposed was the multiple atlas strategy [17,69,70].
The multiple atlas strategy selects a set of atlases and uses each of the selected atlases to segment the unseen image with a segmentation result.The final segmentation is a fusion from these results using an algorithm such as the majority voting approach [83].The multiple atlas strategy is in particular efficient and effective when the medical images with manual labeling can be directly used as atlases.For example, in the cardiac CT/CTA segmentation and brain MRI segmentation, the original medical images are of high image quality and can be employed as the atlas intensity images [16,17,24,29].This method is referred to as the multiple atlas propagation and segmentation (MAPS).In whole heart segmentation of cardiac MRI volumes, MAPS did not demonstrate convincing improvement compared to the single atlas propagation and segmentation (SAPS) using a mean intensity atlas [19].It was proposed to use multiple path propagation and segmentation (MUPPS) to achieve the multiple classifications [20].In MUPPS, the common atlas was constructed from the mean of a set of training images, including the mean of intensity and the mean of shapes.By registering the atlas label image to the corresponding label images of the training data, a set of transformations were determined.These transformations were used as priors and referred to as paths [20].Each transformation could be employed to deform the common atlas, including the intensity image and the label image, to generate a new atlas if this transformation was selected in the path-ranking process.The new atlases had the same intensity distributions as the common atlas, but with different heart shapes defined by the prior transformations.Therefore, MUPPS resembles the MAPS except that the atlases were generated using the prior transformations, instead of the original cardiac MRI images.In both MAPS and MUPPS, an atlas/path ranking scheme should be adopted to efficiently select a small number of "good" atlases from the atlas pool which generally has a large number of candidates.that one can generate new atlases from existing ones, instead of requesting more training samples to construct new atlases.Figure 5 shows an example of a new atlas with right ventricle hypertrophy generated using such techniques [20].

Statistical Shape Model for Deformable Model-Based Segmentation
The deformable model was first introduced to the computer vision field and had gained its popularity since late 1980's, in particular after the "snake" method was proposed [84,85].Deformable model-based segmentation employs shape priors to maintain an optimal shape in the resultant segmentation.The statistical shape model (SSM) [71] is one of the most investaged methods to build the statistical information of the shape priors.In the SSM-based segmentation, the deformation of the model is generally constrained to be within or close to the shape manifold defined by the SSM.These segmentation approaches combine the advantages from both the SSM and the deformable model-based segmentation framework, and they are more suitable for the segmentation of organs with complex shapes [72].
The SSM, originally known as the active shape model (ASM) [71], has been widely investigated and used in modeling and segmentation of complex objects in the medical image computing field [72].Compared to the atlas, the SSM has the advantage of being able to build in the statistical information of shapes of the heart.The procedure of constructing an SSM for whole heart segmentation includes the following steps, as outlined in Figure 6: • Represent images using a shape model.Cootes et al. employed the point distribution model (PDM) [71].The connection between the dense points can be formed to create a surface mesh model for a deformable model-based segmentation [9, 11-15, 25, 34].The shape model embeds the segmentation information and will be propagated to the unseen images.Therefore, it is important to choose a representation, such as the extracted volume or surface of a substructure, from which the segmentation information can be efficiently interpreted.To this end, the triangulated surface mesh model has been preferred and commonly adopted for the whole heart segmentation in the SSM-based methods [11][12][13][14][15].

•
Align image coordinates to normalize the shapes of the training data.To build an SSM, the training data need to be aligned to a common space, as they can be in different coordinate systems and with different orientations and sizes.By aligning the training data to this common space, the variations that are not of interests to the resultant SSM are then excluded from modeling.• Find correspondence between each training shape.To construct the covariance matrix, one needs to find the correspondence of each vertex between two training shapes or between a training shape and the common space.Peters et al. [12,13] and Ecabert et al. [11] employed a common mesh model and deformed this mesh model to each training image to extract its shape.The correspondence between the vertices of the common model and those of the training shapes was then automatically formed.Zheng et al. [14,15] allocated vertices to the surfaces of the manually segmented whole heart volume of each training case using a well defined protocol.Each of the vertices was indexed and then used to form a triangulated mesh model.Since the number and relative positions of the vertices to the whole heart surface were the same for all training data, the correspondence of each training case was then defined implicitly by the indices of the vertices.• Construct mean shape and covariance matrix.Given the shape of a training case is represented by a set of n vertices as X i = {v 1 , v 2 ,…v n }, the mean shape of N training data is computed as follows: ( and the covariance matrix is given by • Apply PCA to compute SSM.Applying the eigen decomposition to the covariance matrix C or singular vector decomposition (SVD) to A results in ordered eigen Challenges and Methodologies of Fully Automatic Whole Heart Segmentation: A Review

Figure 6.
Flowchart of constructing statistical shape model.
values, λ 1 , λ 2 , …, λ dN (λ k ≥ λ k+1 ) where d is the dimension of the vertex coordinates, and the corresponding eigen vectors, V 1 , V 2 , …, V dN , are known as variation modes or eigen modes of the SSM.A valid shape can be estimated from these modes, as follows: ( where {b k } are the weights for the first N' eigen modes which are the most significant variation modes in the shape space.

Feature Priors
To employ a deformable model for automatic segmentation, the feature priors should also be constructed in the prior model.These feature priors are used in the model fitting process to align the model elements to the ground truth positions in the unseen images.Three methods have been reported to construct and apply these feature priors.The first method is to learn the texture and location features of each vertex using a learning-based algorithm, such as the steerable features and Adaboost-based classifiers [14,15].The feature priors of each vertex are the resultant feature bank which forms a strong classifier to detect the corresponding position in unseen images.These approaches are computationally efficient in detecting landmarks which are defined to the vertices of the model, though the training stage can be computationally expensive and memory demanding.
The second method is the simulated search approach [11][12][13].In this approach, a locally optimal boundary detection function was assigned to each mesh triangle in the deformable model.The optimal functions were determined in the model training phase, where the performance of any boundary detection function was evaluated for each element of the model.As a result, the boundary detection can be optimized for each element during the model training.This simulated search was demonstrated to be able to identify the suitable classes of features when a new segmentation task is presented and to be applicable to multi-modal cardiac image segmentation using a single algorithmic framework [13].
The third method constructs texture variations and combines it with the SSM to form a statistical appearance model, which is referred to as active appearance model (AAM) [73].The construction of AAM is similar to that of SSM.An AAM which has statistical information of both the shapes and the texture patterns enables employing the voxel-based registration to fit the AAM to an unseen image.The parameters to be optimized are the weights assigned to the eigen modes [40].However, the dimension of vectors representing the texture patterns of a whole heart image can be very large, and applying PCA to the covariance matrix can be practically challenging.Therefore, Lotjonen et al. and Koikkalainen et al. used the mean intensity image of the AAM for voxel-based registration [9,22].The mean intensity image of the AAM is equivalent to the atlas intensity image constructed from the mean of the training data in [19,20].

Extending Flexibility of Statistical Shape Model
In deformable model-based segmentation, the flexibility of an SSM plays the key role in high-performance segmentation.As the SSM only represents the shape variations of the training data, the variability of the SSM is restricted and the accuracy of segmentation is limited if the shape of an unseen image is very different from the training samples [22].Since it is practically demanding to obtain a large number of training data, several techniques have been proposed to extend the flexibility by either artificially enlarging the training set or releasing the shape constraint of the SSM.Koikkalainen et al. [9] investigated five approaches to artificially enlarge the training set for an SSM, including the Nonrigid Scaling method [86], the Nonrigid Movement method [86], the Smoothing method [87], the PCA & FEM method [88], and the Partitioning method [89,90].The Nonrigid Moving method and the PCA & FEM method may be the two most promising techniques in the whole heart segmentation framework.The Biomechanical Model method [91] and the Shape Constrained Deformable Model method [68], which were not covered in [9], also demonstrate potentials of effectively extending the flexibility of an SSM.
Let {X i }, i = 1…N be the set of N training cases after shape normalization using, e.g., the Procrustes alignment [92], and X be the mean shape of the training set; let v j (i), j = 1…n, be the jth vertex of the training case X i and v j be the mean coordinate of all the jth vertices in the training set.
The Nonrigid Scaling method [86], extending the original idea of the adaptive focus method in [93], scales the surface inside a deformation sphere to generate new samples.The new coordinate of a vertex is given by (4) where t j is the scaling factor, and c and r are the center and radius of the sphere, respectively.The center of the deformation sphere is randomly positioned on surfaces of the training samples.By deforming the surfaces of the original training data using eqn.( 4), new training samples are generated.
The Nonrigid Movement method [86] is similar to the Nonrigid Scaling method: ( where u is a random vector and its elements are normally distributed within The Smoothing method [86,87] combines the statistical variation modes of the original training data and the smoothness modes by constructing a mix covariance matrix: (6) where C smooth is the smoothness covariance matrix and The elements of C smooth are set to 1 if they are corresponding to identity vertices, set to 0.5 if they are corresponding to neighboring vertices [87] or vertices belonging to the same triangle [86], and set to 0 for else.C smooth represents synthetic variations to the subsequent SSM.
The PCA & FEM method [88] calculates the vibrational modes of elastic materials.These vibrational modes are used to generate synthetic variations and added to the original training shapes to generated new samples: (7) where Φ i is the matrix of eigenvectors representing the modes of vibration about training example X i , and b is a weight vector.Similar to the Smoothing method, the PCA & FEM method can also include the synthetic variations in the resultant SSM by constructing a mix covariance matrix: (8) where Λ i is a diagonal matrix consisting of the corresponding eigen values of the stiffness matrix [88].
The Partitioning method, including the hierarchical ASM [89] and the partitioned ASM [90], divides the surface model of a training image into N P patches, {P k }, k = 1, …, N P .The rationale is to separate the local shape variations of a patch from the global shape and specifically model such local variations within the subsequent SSM.This can be done by constructing an individual SSM for each patch [80,81], or achieved by generating more training data, which only exhibit the shape difference in the patch, for the SSM [9].The latter method was tested in the whole heart segmentation.For each training case X, N P new samples can be simulated by keeping the vertices of one patch and replacing the other patches using the mean model: (9) The enlarged training set, including the original training data, then has N(P+1) samples.

ˆ( ) ( ), ( ) , () Xi
The Biomechanical Model method [91], to the best of our knowledge, has not been applied to the SSM study of cardiac modeling.The simulated training shapes using the methods mentioned above do not necessarily represent a realistic shape of the heart, although the methods can extend the flexibility of the resultant SSM.For example, the PCA & FEM method only add vibrational modes of elastic materials to the resultant SSM, while the elasticity does not fully represent the complex tissue property of an organ such as the heart.Davatzikos et al. developed a framework for modeling anatomical deformations using a biomechanical model [91].The emphasis of the work was on the function of predicting anatomical changes due to surgical planning or tumor growth.This idea was subsequently extended to estimate intra-operative deformations of the prostate [94].Hu et al. employed a biomechanical model to simulate the deformations of soft-tissues caused by the random placements of an ultrasound probe [95].The deformations were then applied to a patient-specific surface model constructed from an MRI image to generate new samples for the training of an SSM of the prostate.The patient-specific SSM was used to register the interventional 3D ultrasound images for the fast nonrigid alignment of the pre-operative MRI and the interventional ultrasound.
The Shape Constrained Deformable Model method [11-13, 68, 93, 8] does not artificially enlarge the training set; instead, it relaxes the constraint from the SSM by incorporating the deformable shape model in a regularization term of the energy function in the active contour segmentation framework: (10) In this framework, E ext is the external energy measuring the distance between the deformable model to the detected boundaries in the unseen image, and E int is the regularization term which penalizes the deformations when the surface model is distant from the hyper space of the SSM or simply the mean shape model [11][12][13]: (11) where V={v i } are vertices of the deforming mesh model, {m i } are the corresponding vertices in the deformable mesh model, P are parameters weighting the eigen modes of the PCA model, N(i) are the indices of the neighbor vertices of vertex v i in the mesh model, and T is a geometric transformation.The parameters P can disappear if the mean shape model is used.Given the corresponding vertices in the epicardial and endocardial surfaces also be connected in the mesh model, eqn.(11) also implements the penalty term to maintain the connection distance between the endocardium and epicardium [25].The piecewise affine transformation model is used to model the geometric transformation T [11][12][13].The piecewise affine transformation model, which can be implemented in a hierarchy fashion, further extends the flexibility of the shape constrained deformable model and improves the capability of modeling local details.

∑ ∑ (
) Challenges and Methodologies of Fully Automatic Whole Heart Segmentation: A Review

Other Potential Modeling Techniques
Modeling the anatomy of the heart can be challenging as the heart has a complex structure.For segmentation propagation, a number of simplified representations of the shape model have been used, including manually delineated contours, mathematical models, point distribution models, surface mesh models, volumetric models with elastic material or bio-mechanical properties, and the sparse shape composition.
Hautvast et al. semi-automatically segmented the ventricles and myocardium of cine cardiac MRI by propagating the manual delineation of a selected phase to other phases [74].The prior model was the initial manual delineation.
Mathematical models, such as a conic equation, were used to model the ventricle cavity as an ellipsoid [32].This model is simple in construction and can be elegantly formulated in mathematics for deriving the internal energy.However, modeling the whole heart structure can be too complex for this method.Medial representation (M-Reps) [96] is a more sophisticated tool to model objects with irregular shapes.The applicability was demonstrated in image segmentation for several applications in [97], except for cardiac images.Pilgram et al. built a heart model based on M-Reps which may be applied to this task [98].However, its application to whole heart segmentation, to the best of our knowledge, has not been reported.
The point distribution model [88] provides dense point cloud on the surfaces of the modeled objects.For segmentation propagation, the connection between these points is generally formed to create a surface mesh model, for example, for the segmentation of the ventricles [25,34] and the whole heart [9,[11][12][13][14][15].The volumetric models with built-in material property, such as nonlinear elasticity [75] or biomechanical property [99], may also be applicable to whole heart segmentation.Constructing these models is one challenging task, while efficiently solving the deformation fields from the applied forces is the other.
Zhang et al. [100,101] proposed a sparse shape representation to approximate the shape of an unseen image using sparse linear combination of other shape instances.These shape instances had been previously annotated.Without using statistical shape decomposition methods, they employed the sparsity constraint to derive a smooth and realistic shape.This sparse shape representation method has been used in the liver segmentation where the objective organ can be considered as one simple structure.In whole heart segmentation, the objective organ consists of several substructures which need to be separated from each other.Hence, it can be a challenge to apply the method directly.More details can be found in the review papers for cardiac modeling [3] and shape representation and description techniques [102].
Prior models are generally embedded in the segmentation algorithms and determine the detail and representation of the segmentation results.Therefore, the content of anatomical modeling of the heart can be different for different applications.For example, both the atrioventricular valves and the virtual valve planes [14,15] can be modeled to define the separating boundaries between the atria and ventricles.The segmentation of ventricular cavities can either include or exclude the papillary muscle, depending on the definition of the prior model.Most of the reported cardiac segmentation articles modeled the smooth endocardial surfaces without considering the details inside the heart, although these finer structures, such as the trabeculae, can be captured and delineated using the state-of-the-art CT scanners [103].

The Fitting/ Registration Process
A prior model enables fitting the model to an unseen image for segmentation propagation.In general, the shape difference between the model and the unseen image is nonrigid, so that the global rigid or affine transformations are not sufficient for the registration between them.Therefore, the fitting process should include a global affine registration to correct the global difference such as the position, size, and orientation, and then a local deformable registration to correct the local difference.To achieve robust and accurate segmentation propagation, the deformable registration is generally implemented in a coarse-to-fine fashion.Therefore, the fitting process is considered as a hierarchy scheme, consisting of the three stages: localization of the whole heart, initialization of substructures, and refinement of boundary delineation [11][12][13][14][15]19].In the deformable registration, shape regularization is needed.The flowchart of the fitting process is shown in Figure 7.The sections below introduce the methodologies of the following four key techniques that are used to classify published works (Table 1 and  Table 2): • Localization of the whole heart, including location, orientation and size, etc. • Initialization of substructures of the heart.

•
Refinement of the boundary delineation in detail.

•
Regularization of Shapes to maintain a realistic and smooth heart shape in the resultant segmentation.

Localization
The goal of the localization is to align the global shape of the prior model to that of the unseen image.Therefore, this localization stage can correct global mismatch of translation, orientation (rotation), size, and sometimes shearing as well between the model and the unseen image.An affine transformation can be used to model this difference.Since the shearing difference is not evident and thus is less important, a similarity transformation is commonly adopted to correct the translation, rotation, and size difference.The global affine registration has been well developed in the past two decades [104].It is commonly used for the localization in the atlas propagation and segmentation frameworks [16][17][18][19].The mean image of the AAM was employed to register to unseen images for localization [22,9].
Learning-based detection methods train and record the common and discriminative features in cardiac images using a large number of training data.Haar-like features are used to train a detector to find the center of the heart [14,15].To estimate the orientation and size, Zheng et al. proposed the steerable features which can be efficiently computed without rotating or scaling the unseen images [14,15].Also, they further proposed the marginal space learning (MSL) to gradually increase the dimension of the parameter space.MSL significantly reduces the number of testing hypothesis, of which each corresponds to one case in the quantized similarity transformation parameter space.Weese et al. employed the generalized Hough transform to localize the center of the heart and estimate its size using a set of quantized isotropic scaling parameters [11][12][13].To further perform the localization with orientation and anisotropic scaling, they modified the surface mesh model with a global affine constraint.Both the localization methods in Zheng et al.'s and Weese et al.'s works registered their prior models to an unseen volume using a similarity transformation.In general, the learning-based localization approaches are computationally efficient and robust in searching the center and size of the heart.To estimate an affine transformation, Shi et al. [38] and Zhuang et al. [24] employed the Haar-like feature training-and-detection method to quickly estimate the center and size of the heart, and then applied an affine registration to further correct the difference for their cardiac ventricle segmentation.

Initialization
Initialization of substructures of the heart is an important stage to bridge the global heart localization and the follow-up refinement of local details.It is also the key technique to achieve the robust whole heart segmentation for a wide range of pathological cases with large variations of shapes.
The locally affine registration method (LARM) was proposed for the initialization of substructures in [18,19].To apply LARM, one needs to define a set of local regions or landmarks.Each of them is assigned a locally affine transformation.LARM has a mechanism to maximally maintain the affinity of the pre-defined local regions by simultaneously optimizing all the locally affine transformations.It produces diffeomorphic resultant transformation to preserve the topology of the prior model after the propagation.An inverse deformation field can be efficiently computed using the proposed dynamic resampling and distance weighting interpolation scheme, which theoretically can produce an inverse transformation with a maximal error less than the voxel size of the reference image.The initialization was implemented in three stages, a coarse-to-fine hierarchy framework [19]:

•
In the first stage, only two local affine transformations are performed.The corresponding local regions are defined to the atrial region, including the two atria and the great vessels, and the ventricular region, including the two ventricles.

•
In the second stage, the right ventricle and the right atrium are separated from the regions defined in the first stage to add another two local affine transformations.• Finally, seven local regions, including two ventricles, two atria, pulmonary artery, ascending aorta, and descending aorta, are used to define a seven-local-region LARM.
In all three stages, the atlas image was used as the target image.Therefore, an inverse transformation from the unseen image to the atlas was also computed.
Weese et al. [11][12][13] proposed the piecewise affine model for the initialization process.In the piecewise affine adaptation, each piece of the pre-defined substructures was assigned an affine transformation.The change of the shape of the substructure was constrained within the local affine transformation.The adaptation was also implemented in a hierarchy scheme as follows: • First, a global affine transformation was used for the adaptation of the whole heart.

•
Each of the anatomical substructures, such as the four chambers and the great vessels, was then assigned an affine transformation.• Finally, the anatomical substructures could be further divided into finer substructures, and more pieces of affine transformations were assigned to them.
The realization of substructure initialization was embedded into the model deforming and shape regularization process [14,15].After the localization, they employed the learning-based landmark detection to detect a small set of control points of each substructure.The control points are mainly scattered to the separating positions and the skeleton of the substructures, such as the valve annulus and the ventricular divergence planes.These control points not only can estimate the boundaries between the atria and ventricles, where there is no distinct boundary from the intensity distributions of the CT or MRI volumes, but also can initialize the global shape of the four chambers.To regularize the global shape of the model after the control point adaptation, the deformed model was projected onto a shape subspace with 25 dimensions (first 25 eigen modes) of the prior SSM.
Finally, it should be noted that the initialization stage can be optional in a whole heart segmentation method if other techniques are employed to guarantee the substructures to be also well initialized after the global localization stage [9,16,17,22].The multiple atlas strategy was used in [16,17], the training data set was artificially enlarged to extend the shape variability of the SSM [9], and an SSM whose shape variations covered the test data (the test data were the training data) was constructed for the segmentation [22].

Refinement
Refinement is the final stage of the fitting process.It produces more local deformations to the model and can achieve better accuracy compared to the previous processes.
In atlas propagation and segmentation, nonrigid registration is used to refine the final propagation.Kirisli et al. [16,17] employed the classic FFD registration [105] which uses the bending energy as the penalty term to guarantee the smoothness of the resultant 394 Challenges and Methodologies of Fully Automatic Whole Heart Segmentation: A Review deformations.Similar to other generically applicable nonrigid registration methods, such as the fluid registration [106], the registration in these works does not use the prior knowledge in the atlas to guide the registration.This unsupervised refinement is however prone to generate erroneous results, particularly in the segmentation of myocardium [19].Figure 2c shows an example that the epicardium was erroneously delineated as the adjacent liver tissue using the classic FFD registration.The FFD with active control point status (ACPS FFD) was proposed to incorporate the shape of the heart and guide the nonrigid registration [19,20].Deformable model-based methods refine the final segmentation by searching boundary/edge points in the unseen image, and then locally adapting the mesh model to align the vertices or barycenter of triangles of the model to the detected corresponding points [9,[11][12][13][14][15]. Koikkalainen et al. adopted the method of Cootes et al. [71,73] in the fitting process.This method searches the optimal weights for the eigen modes of the SSM to minimize a cost function, which measures the difference between the texture built in the SSM and the corresponding texture in the unseen image.Peters et al. employed the trained boundary detectors to detect the boundary points and refined the final segmentation by minimizing the cost function in eqn.(11) [11][12][13].The detection functions were selected (trained) in the model training stage using the simulated search approach, and the optimized parameters included the piecewise affine transformations, the weights of eigen modes of the SSM, and the vertex coordinates.These parameter sets were optimized by iterating two steps: first to optimize the transformation parameters, and then to optimize the weights of eigen modes and vertex coordinates.Zheng et al. [14,15] searched boundary points, as well as other salient landmarks for initialization, using learning-based point detection algorithm.The refinement step searched all the boundary points which were predefined according to the anatomy of the heart and were trained using steerable features.The detection of a boundary point was guided such that the searching range of the point was constrained within a region initialized by the corresponding point in the deformed surface model.

Shape Regularization
The shape regularization is crucial in automatic segmentation of the organs with complex shapes and multiple components.In general, it is embedded into the deformable fitting processing, including the initialization and refinement stages, to maintain the global shape.The objective of the shape regularization is to maintain the deformed model with smooth surfaces, a realistic heart shape, and robust propagation of indistinct boundaries which are defined according to the anatomical knowledge.
In atlas propagation and segmentation, the shape regularization is mainly handled with the original shape of the atlas and the deformation regularization process in the registration.Kirisli et al. [16,17] used the bending energy implemented in the classic FFD registration [105] as the shape constraint.Zhuang et al. [18,19] used the LARM and the ACPS FFD for the registration processes in the initialization and the refinement stages, respectively.Both methods adopted a hierarchical scheme and incorporated the shape of the heart in the atlas to guide the registration process.This shape-guided registration was shown to improve the performance of the segmentation propagation [19].
In the SSM based segmentation, the widely used approach is to constrain the deformation of the shape model within the variability of the SSM [9,14,15,28,40].The disadvantage is that the accuracy of the segmentation is limited by the variability of the SSM, namely the training dataset.Zheng et al. [14,15] regularized the shape of the model in each nonrigid estimation step by projecting the deformed model onto a subspace of the SSM, which is equivalent to using the first few eigen modes to estimate the new shape.Koikkalainen et al. [9] adopted the fitting method of Cootes et al. [71] to limit the weights of the eigen modes within three times of the standard deviation, but extended the variability of the SSM by artificially enlarging the training set [9].Peters et al. [11][12][13] employed the shape constrained deformable model to regularize the shape of the segmentation result.This method relaxed the constraint from the SSM by incorporating the deformable model in a regularization term of the energy function in the active contour segmentation framework.In the multi-stage adaptation process, a mean mesh of the whole heart structure could be used and matched to unseen images, where no variation mode of the SSM was needed in the fully deformable adaptation thanks to the piecewise affine transformation model.

VALIDATION ISSUES
The results of whole heart segmentation can either be presented as a binary volume or a surface mesh for each substructure, depending on the requirement of clinical applications.For example, for computing functional indices such as myocardium mass or ejection fraction, the binary volume is preferred, while for 3D rendering of the heart structure, the surface mesh is more useful.The volumetric and surface presentations are also convertible to each other using existing techniques.To objectively evaluate a whole heart segmentation algorithm, three issues need to be considered.
Firstly, a large amount of clinical data, with a wide range of pathologies, are needed.Since the major challenge of automated whole heart segmentation stems from the large shape variation of the heart, this test dataset should cover the different shapes that the clinical data would have in practice.
Secondly, a ground truth or gold standard segmentation of each test image is required.In general, manual delineation is used to construct gold standard segmentation.This manual work can combine semi-automatic tools with friendly and efficient user interface to interactively correct the delineation.Also, the definition of boundaries of substructures should be consistent for both the prior model and the gold standard of all test data.For example, the blood cavity of the left ventricle can either include or exclude the papillary muscle, and the boundary between the left ventricle and atrium can be defined using either the actual position of the mitral valve or a flat plane positioning on the annulus of the mitral valve.Similar situation applies to the right ventricle and right atrium.For the aorta and pulmonary artery, since they extend to other organs of the body and the field of view may not cover the whole structure, the validation of whole heart segmentation can focus on the major trunk of the arteries as discussed in the Introduction section (Section 1).
Finally, objective evaluation protocols should be employed, particularly for cross study comparisons.For a binary volume of a substructure, one can use the Dice score [107] or Jaccard index [108] to evaluate the similarity between the segmentation 396 Challenges and Methodologies of Fully Automatic Whole Heart Segmentation: A Review volumes from automatic segmentation V auto and the gold standard V gd .The generalized Dice score and Jaccard index [109], which summarize the overlap of multiple labels, can be used to assess a whole heart segmentation result [20]: (12) To evaluate the accuracy of boundary delineation, surface distance between the segmentation result and its gold standard can be used.Mean and standard deviation of the surface distance calculated from a set of discrete sample points of the surfaces are widely used to represent the distance between the two surfaces [11][12][13][14][15]. Table 3 and Table 4 present the validation results of the whole heart segmentation methods reported in the literature.The best accuracy reported were from Weese et al. [11][12][13] and Zheng et al. [14,15].The computation time of them were also efficient, as the former took 10 to 30 seconds on average to segment one volume, while the latter only needed 4 seconds, making it widely applicable and easily integrated into clinical practice.For the data used in the validation, Zheng et al. tested their algorithm on more than 500 CT volumes from 137 subjects [14,15], and Zhuang et al. [19] validated their method on a set of clinical data involving nine different pathologies.The two tools demonstrated good robustness against clinical datasets with large shape variations, although the computation speed of the latter method needs to be improved (as long as 231 seconds [24]).It should be noted that due to the differences in test data sets and hardware, an objective comparison between these methods is difficult.Therefore, a grand challenge can be useful to achieve this goal in the future.Several useful challenges on cardiac segmentation, although not on the whole heart, have been organized, including the following:

DISCUSSIONS
The challenges of whole heart segmentation are mainly due to the large shape variations of the heart and the indistinct boundaries between substructures.To tackle these challenges, segmentation generally employs prior models to guide the procedure, which is formulated as a fitting process from the prior model to the unseen image.According to the different representations of the prior model, the existing segmentation frameworks can be classified into two groups: (1) the atlas propagation and segmentation methods, and (2) the deformable model using statistical shape model (SSM) approaches.
In the atlas propagation and segmentation methods, the atlas is relatively easy to construct and the construction only needs a small number of training data.This is because the atlas does not have built-in statistical information of the shape or appearance, and therefore the segmentation propagation mainly relies on the registration algorithms to deal with the challenges.Hierarchical registration approaches are adopted to improve the robustness of segmentation propagation.The shape regularization is implemented by combining a deformation smoothness term with a similarity term.Although researchers in the image computing field continue improving the registration algorithms, there is an increasing interest in using a large number of atlases for multiple atlas propagation and segmentation (MAPS).In MAPS, a small number of atlases, whose shapes (or both shapes and appearance) are most similar to that of the unseen image, are selected for segmentation propagation.Therefore, fast atlas-ranking algorithms play an important role in these segmentation frameworks.Recently, a new atlas-based segmentation method, the patch-based segmentation (PBS), was proposed for segmentation of brain MRI [110], and was also extended to segment multiple organs from abdominal CT scans [111].PBS does not need to accurately register the selected atlases to the unseen images; for example, an affine registration can be used [110].The segmentation propagation is achieved by ranking patches from the selected atlases and fusing label information of patches according to their rankings.
A practical problem of the atlas propagation and segmentation methods is the computation time.This is because a nonrigid registration process is generally needed in the atlas propagation, and this registration can be computationally expensive.In whole heart segmentation, one atlas propagation could take about 20 minutes [16,17] or more than one hour [19].The computation time of nonrigid registration represents a more significant issue in MAPS, although the computation time can be reduced to 231 seconds by simply reimplementing the registration algorithms [24].PBS has similar computation complexity as MAPS, although nonrigid registration may not be needed [110,111].Recent advance in image registration can reduce the nonrigid registration to within a minute or even a few seconds when GPU programming is used [e.g., 112,113].Therefore, it is anticipated that these techniques will be applied to atlas propagation for whole heart segmentation.
The deformable model-based segmentation is normally computationally efficient.For example, the whole heart segmentation only takes 4 to 30 seconds [11][12][13][14][15].This is because the fitting process is generally performed using local texture information of the unseen image.Such local search requires the segmentation approaches to closely initialize the surface elements of the prior model to the ground truth boundaries.SSM is widely adopted for the automatic segmentation of complex organs such as the whole heart.In the SSM-based framework, deformation of the model is constrained to be within the shape variability of the SSM, and thus the segmentation result is maintained with a realistic shape [22,14,15].To capture the large shape variations of the heart, a large number of data, which represent all unseen images in clinics, are required for the training of the SSM.This is however practically difficult.Several methods have been proposed in the literature to artificially enlarge the training set or relax the constraint from the prior shape model for automatic whole heart segmentation.Since the shapes of the segmentation results are not exactly guaranteed by the SSM, the fitting process should be elaborately designed to regularize the deformation of the surface model.For example, in the shape constrained deformable model framework, the hierarchical piecewise affine adaptation is used before the fully deformable adaptation [11][12][13].Similar to the registration-based techniques, these approaches combine a shape regularization term and a similarity term within an energy function.The regularization term penalizes the deviation of the deformations to the prior model, and the similarity term determines the accuracy of the local adaptation of the model surface to the detected boundaries.The balance between them is controlled by the weighting parameters of the two terms.
Boundary detection is crucial in the deformable model-based segmentation, as the accuracy of the local adaptation is determined by the detected boundaries points.AAM builds the statistical information of the texture patterns of the objective organ, as well as the statistical information of the shapes.AAM can then be used in the intensity-based registration framework, where the parameters weighting both the shape eigen modes and the appearance eigen modes are optimized [28,40,73,114].These approaches can combine the advantages from both the atlas-based segmentation and the deformable model-based segmentation methods to improve the robustness and accuracy.However, besides the same practical problem of demanding a large number of training data, the dimension of the vectors representing both the texture and the shape of a whole heart image can be too large to apply the PCA.Therefore, methods which can compactly describe the texture patterns of the whole heart are desirable.
The AAM segmentation framework can be considered as a method to generate a subject-specific atlas for an unseen image using the learned image manifold.Therefore, other atlas generation methods can be considered in the future.Several works were proposed recently for learning a nonlinear manifold of face images [115], brain MRI [116,117], and cardiac MRI [118].The metadata, such as the subject age and sex, were included in the modeling of brain images, and the variables of the metadata controlled the training of the model [117].Therefore, new instances can be easily generated for segmentation propagation, given that the specific values of these variables are available.In practice, the information of subject age and sex may not be enough to create a good atlas, and generating a subject-specific atlas can be computationally demanding using regression methods of Gerber et al. [116] and Rohlfing et al. [117].Fast computation on a shape manifold was proposed to generate subject-specific atlases [118].The new atlas of an unseen image is formulated as the Frechet mean of existing atlases, which can be directly computed via fast vector operations on velocity fields.Sparse shape composition is another modeling technique which creates new instances by linearly combining a small number of existing samples [100,101].The method has been shown to be efficient for shape modeling and automated segmentation of multiple organs.In the future, the method is expected to include appearance of images and to be extended to the segmentation of the whole heart.
It should be noted that most of the prior model-based segmentation methods can only deal with the images which have the same topology as the prior model.The difference of topology represents a more challenging task than the shape variations from different pathologies.Different pathologies may induce very different heart shapes, but the topology of the heart may still be the same.No work has been reported to study the segmentation of images with different topologies, and the existing approaches reviewed in this article may fail in such situation.For instance, it is difficult to segment an unseen volume from a patient with dextro-transposition of the great arteries using a prior model constructed from cardiac images with normal topology of the heart.In such patients, the aorta and the pulmonary artery are malformationally connected to the right ventricle and the left ventricle, respectively.To deal with such cases, a potential solution is to construct an atlas (or deformable model) pool which has at least one instance corresponding to one anatomical topology of the heart, and design an efficient atlas ranking-and-selection algorithm to automatically select the corresponding atlases for segmentation.In practice, the metadata, such as pathologies and stages of the pathology, can be useful in the atlas selection algorithm.

CONCLUSIONS
This work reviews the existing techniques of fully automatic whole heart segmentation, and analyzes the challenges and methodologies of the computation task.The challenges are in large variability of the heart shape, indistinct boundaries, and limited image quality.The reported methods generally employed prior models and formulated the segmentation as a fitting process from the prior model to the unseen images.
To analyze the methodologies, the existing works are classified in terms of the prior models and the techniques used in the fitting process.The prior models include the atlas (single atlas or multiple atlases) and the SSM-based deformable models.In general, the fitting process is implemented in a hierarchy fashion, which is further decomposed into three stages: localization of the whole heart, initialization of substructures, and refinement of boundary delineation.Also, to guarantee a segmentation result with a realistic shape, shape regularization is needed in the deformable fitting.Table 1 classifies whole heart segmentation works in terms of the types of techniques, Table 2 lists the techniques for the works, and Table 3 and Table 4 summarize the reported performance of each method, although an objective comparison is not possible and an open-access dataset with gold standard is in need.
Future research is needed to build robust, fast and computationally inexpensive whole heart segmentation methods for clinical applications with a wide range of pathologies.In the model-based segmentation framework, to ease the fitting process, the prior model is expected to resemble the unseen image by simple transformations.Therefore, all available priors, ideally with information of statistical distributions, should be built into the model.For example, a large amount of data are used for the multiple atlases-based segmentation or for the training of statistical shape models.To reduce the complexity of model construction, it is necessary to design fitting algorithms that are capable of registering the model to an unseen image even when they are very 400 Challenges and Methodologies of Fully Automatic Whole Heart Segmentation: A Review different.Finally, extending the applicability of the existing priors is an important technology, especially when the available training data are limited, such as generating new atlases using existing ones and extending flexibility of the SSM.

Figure 1 .
Figure 1.Whole heart segmentation result from a cardiac CTA volume.(a) A slice from the axial view of a cardiac CTA volume.(b) The corresponding whole heart segmentation labels, in different color.(c) The whole heart segmentation labels superimposed onto the CTA image.(d) The volume rendering of the whole heart segmentation labels.

Figure 2 .
Figure 2. (a) MRI volume from a healthy volunteer with successful segmentation, defined by the green contour.(b) MRI volume from a patient with right ventricle hypertrophy, with erroneous segmentation using a traditional atlas propagation and segmentation scheme, defined by the red contour.Reproduced with permission [23].

Figure 3 .
Figure 3. Indistinct boundaries.(a) A cardiac MRI image with segmentation result.In the MRI image, no flat tricuspid valve plane between the right ventricle and the atrium is visible, and the segmentation incorrectly delineates the plane, as indicated by the red arrow.However, the true boundary should be at the white dash line position.(b) Cardiac CTA image showing that within the elliptic region (marked by orange boundary), there are three tissues or substructures with similar intensity distributions, including the myocardium (septum), the right ventricle, and the liver tissue.The intensity distributions between the myocardium and the body tissue adjacent to the heart, as the arrow indicates, are also similar.(c) An example of erroneous segmentation on myocardium which has a similar intensity distribution as the adjacent liver tissue.(d) Cardiac MRI image showing that the boundaries between the left atrium, right atrium, and the aorta are unclear, as indicated by the arrow, due to the thin walls and partial volume effects.Images reproduced with permission [23].
or multiple) + Affine registration + Hierarchical locally affine registration + [18-20] ACPS FFD registration + Smooth deformations and shape-guided registration Zhuang et al. 2013 Four chambers and Atlas (multiple) + Whole heart detection plus affine registration + Hierarchical locally [24] beginning trunks of aorta affine registration + ACPS FFD registration + Smooth deformations and shape-guided and pulmonary artery + CT registration Kirisli et al. 2010 Four chambers and Atlas (multiple) + Affine registration + N/A + FFD registration + Smooth deformations [16, 17] beginning trunks of by including bending energy into the cost function of registration aorta + CT a Multi-slice cine MRI: The data included 4-5 slices from short-axis view and 4-7 slices from long-axis view, and were constructed as volumetric data using shape based interpolation.b

Figure 4 .
Figure 4.Workflow of the atlas propagation and segmentation.

Figure 5 .
Figure 5. Example of generated atlas using MUPPS techniques.(a) The common atlas.(b) The pathological image showing a hypertrophic right ventricle.(c) The generated atlas from the common atlas with right ventricle hypertrophy.
[-r/2 … r/2].Koikkalainen et al. used 31 spheres and set r = 50 mm for both the Nonrigid of Fully Automatic Whole Heart Segmentation: A Review Scaling method and the Nonrigid Movement method, generating (31N+N) samples for the training of SSM.

Figure 7 .
Figure 7.Flowchart of fitting the prior model to unseen data for segmentation propagation.

Table 2 . Existing techniques for fully automatic whole heart segmentation (Continued) Method Reference Goal + Modality Prior
model + Localization + Initialization + Refinement + Shape Regularization Zhuang et al.

Table 3 . Validation results of the existing whole heart segmentation methods with accuracy reported in surface-to-surface distance (in mm) or/and Dice score (italicized) Whole Heart Run Time Size of dataset More information about data source and validation Reference Substructure Accuracy Training Test
Lotjonen et al.LV,EPI,RV,LA,RA 2.53 ± 0.70 mm N/A 25 25 Multi-slice cine MRI, resolution of 1.0

Table 3 . Validation results of the existing whole heart segmentation methods with accuracy reported in surface-to-surface distance (in mm) or/and Dice score (italicized) (Continued) Whole Heart Run Time Size of dataset More information about data source and validation Reference Substructure Accuracy Training Test
Zhuang et al.LV,MYO,RV,LA,RA 1.47 ± 1.55 mm 2-4 hours 10 37 SSFP MRI, acquired resolution of 2 LV,EPI,RV,LA,RA, 1.17 ± 1.39 mm 231 seconds 14 15 CT, from 15 patients; leave-one-out cross validation.