This paper presents a new, automatic method of accurately extracting lesions from CT data. It first determines, at each voxel, a five-dimensional (5D) feature vector that contains intensity, shape index, and 3D spatial location. Then, nonparametric mean shift clustering forms superpixels from these 5D features, resulting in an oversegmentation of the image. Finally, a graph cut algorithm groups the superpixels using a novel energy formulation that incorporates shape, intensity, and spatial features. The mean shift superpixels increase the robustness of the result while reducing the computation time. We assume that the lesion is part spherical, resulting in high shape index values in a part of the lesion. From these spherical subregions, foreground and background seeds for the graph cut segmentation can be automatically obtained. The proposed method has been evaluated on a clinical CT dataset. Visual inspection on different types of lesions (lung nodules and colonic polyps), as well as a quantitative evaluation on 101 solid and 80 GGO nodules, both demonstrate the potential of the proposed method. The joint spatial-intensity-shape features provide a powerful cue for successful segmentation of lesions adjacent to structures of similar intensity but different shape, as well as lesions exhibiting partial volume effect.

Accurate and automatic segmentation of medical images is an essential component of a computer-aided diagnosis (CADx) system. However, medical image segmentation is typically a difficult task due to noise resulting from the image acquisition process, irregular shape and variable size of anatomical objects, as well as the characteristics of the object’s neighborhood. For example, a lung nodule or a colon polyp is usually embedded in a complex surrounding region. In CT imaging, the intensities of such lesions (e.g., juxtavascular nodules, juxtapleural nodules, or colonic polyps) are usually very similar to their adjacent tissues. In this case, traditional intensity-based or morphological methods [

Energy minimization techniques for image segmentation have shown much promise in medical image computing. In particular, graph cut methods [

In most graph cut methods, the graph vertices are constructed at the image pixels, and the segmentation energy is composed of intensity terms only. For example, Zheng et al. [

The goal of this paper is to develop an automatic and robust superpixel-based graph cut method for accurate segmentation of different types of lesions in CT imaging including solid and GGO nodules, as well as colonic polyps.

One of the original contributions of this paper is that our graph is built on mean-shift superpixels. In this paper, a

A second original contribution is our energy function, which incorporates the image intensity and the shape feature into a Markov Random Field (MRF) minimized with graph cuts. In particular, the intensity and shape information appear in both our unary and pairwise terms. Compared to our previous work in [

The paper is organized as follows: Section

Our approach is a combination of five-dimensional mean shift clustering followed by energy minimization based on graph cuts. The method is also guided by prior knowledge about the lesion (e.g., nodule/polyp). The flow chart of our method is illustrated in Figure

Flow diagram of the proposed graph cut-based method.

The method first computes the JSIS features, which are then clustered in a five-dimensional space using mean shift. In this section, we review the volumetric shape index feature and our five-dimensional mean shift approach.

The volumetric shape index (SI) at voxel

The calculation of the Gaussian and mean curvatures is based on the first and second fundamental forms of differential geometry. A practical approach for computing these forms is to use the smoothed first and second partial derivatives of the image as suggested in [

The shape index provides a local shape feature at each voxel. Every distinct shape, except for the plane, corresponds to a unique shape index. For example, a shape index value of 1.0 indicates a sphere-like shape (e.g., lung nodule or colonic polyp), while 0.75 indicates a cylinder-like shape (e.g., vessel or colonic fold). Based on the definition, the volumetric shape index directly characterizes the topological shape of an isosurface in the vicinity of each voxel without explicitly calculating the isosurface. This feature provides rich information that complements the image intensity and is useful for automated segmentation. In particular, lesions in a CT image may appear within an area of complicated anatomy (such as a lung nodule neighboring a blood vessel or colonic polyp attached to the colon wall) where adjacent structures have similar image intensities but different shapes.

Note that we use the term

For each voxel, the 3D spatial location, intensity, and volumetric shape index features are concatenated in the joint spatial-intensity-shape domain of dimension

In the mean shift framework [

It is noted that the mean shift algorithm estimates the

An attached nodule with its intensity and shape mode maps. (a) Original CT subimage; (b) shape index map based on (

The mode map obtained by the above JSIS mean shift technique expresses the local structure of the data in a given region of the feature space. The number of resulting superpixels depends on the kernel bandwidth and the data itself; a key advantage of the mean shift clustering is its ability to locally group regions of similar intensity and shape, reducing the data variance in superpixel.

The aim of this section is to group superpixels into two classes: foreground (lesion) and background (nonlesion). It is known that the image labeling problem can be formulated using an energy function in a Bayesian framework in the context of maximization a posteriori (MAP) and MRF theory and can be solved by energy minimization. The energy function includes both unary and pairwise terms, the latter providing smoothing by modeling the interaction between neighboring superpixels. In this section, the MAP-MRF is transformed to a graph cut problem. A novel energy formulation that incorporates shape and intensity in both unary and pairwise terms is defined on superpixels and minimized with graph cuts using the maxflow/mincut method [

Two key issues are addressed in the following subsections: (

In this paper, we assume that a nodule (or polyp) is generally either spherical or has a local spherical concentration, while a blood vessel (or colonic fold) is usually oblong. The initial seeds for the graph cut are computed automatically based on a spherical concentration.

A spherical concentration

All voxels in the region have a shape index greater than or equal to a low threshold

At least one voxel in the region has a shape index greater than or equal to a high threshold

Typical thresholds are in the range

The spherical concentration defines the foreground seeds in the graph cut initialization. The background seeds are determined using adaptive enlargement of the spherical concentration. A distance transform [

An example of the initialization based on spherical concentration on an attached nodule. (a) 3D nodule in three contiguous CT slices; (b) initial foreground based on high spherical concentration; (c) initial background region.

The thresholds of

An example of the segmentation of an attached solid nodule based on the automatic calculation of the initial foreground and background. (a) 3D nodule in two contiguous CT slices; (b) initial foreground based on high spherical concentration; (c) initial background; (d) segmentation results by the proposed graph cut based method.

An example of the segmentation of a part-solid nodule based on the automatic calculation of the initialization (a) 3D nodule in three contiguous CT slices; (b) initial foreground based on high spherical concentration; (c) initial background; (d) segmentation results by the proposed graph cut based method.

In this section, we employ a graph

The lesion segmentation problem is formulated as a binary labeling problem, so the goal is to assign a unique label

We are given the initial foreground

The foreground cost (

Similarly, the background cost

Schematic diagram of different types of superpixels: foreground (green) superpixels, background (red) superpixels, and uncertain (gray) superpixels.

The second term

As can be seen from (

It is noted that when two adjacent superpixels have the same shape, namely,

Figure

A example of an attached nodule segmentation using different pairwise smoothing energies. (a1–a3) 3D nodule in three contiguous slices in CT; (b1–b3) nodule segmentation using the first term only in (

Example of cost functions. (a1, b1) 3D attached nodule in two contiguous slices in CT; (a2, b2) unary cost; (a3, b3) pairwise smoothing cost; (a4, b4) minimization of both energy terms based on (

In this section, we present results demonstrating the effectiveness of the proposed algorithm applied to clinical CT scans. The evaluation of the proposed segmentation method has been conducted in two parts. The first, visual analysis is performed on several different types of CT pulmonary nodules and colonic polyps to provide insight into the method’s performance for any visual gross missegmentation, such as a failure in separating nodules from vasculature. The second, quantitative experiments evaluate the segmentation results on large nodule datasets, containing 101 solid nodules and 80 GGO nodules. It is noted that, all the nodules in our database are confirmed by three experienced thoracic radiologists. However, each nodule boundary was manually delineated by one qualified radiologist.

In all experiments, the three kernel bandwidths (spatial

Figure

The nodule superpixels can be merged using the proposed graph cut method. Figure

Figure

An example of a solid nodule (attached to a small vessel) segmentation. (a) Original 3D nodule on 5 contiguous slices; (b) and (c) intensity and shape index (multiplied by 100) mode maps from 5D mean shift clustering; (d) segmentation results without considering the shape feature; (e) results based on the proposed method.

An example of segmentation of an attached nodule. (a) Original 3D nodule on three contiguous slices; (b) intensity mode map and its corresponding intensity mode values based on 4D mean shift clustering (without shape); (c) intensity mode map and its corresponding intensity mode values based on 5D mean shift clustering (with shape). (d) and (e) segmentation results without and with the shape feature (proposed method), respectively.

A pleural nodule is defined as a nodule attached to the pleural wall. Figure

An example of a pleural nodule segmentation. (a) Original 3D nodule on four contiguous slices; (b) and (c) segmentation results without and with the shape feature (proposed method), respectively.

A GGO nodule appears as a hazy increased CT attenuation of the lung. It is challenging to properly segment GGO boundaries due to the faint contrast, irregular shape, and fuzzy margins of GGO nodules. Figure

Quantitative evaluation of GGO nodule segmentation and further discussion will be given in next subsection.

An example of a large GGO nodule segmentation. (a) Original 3D nodule on eight contiguous slices; (b) and (c) segmentation results without and with shape feature (proposed method), respectively.

The segmentation of colonic polyps in CT images is a complex task due to the irregularity of polyp morphology and the complexity of surrounding regions. The boundaries between polyp tissues and nonpolyp tissues are much less obvious. Results showing the proposed method applied to colonic polyp segmentation appear in Figures

Segmentation of a colonic polyp. In (a), we show a 3D polyp in three contiguous slices, in (b) segmentation results without the shape feature, and in (c) segmentation results based on the proposed method.

Another example of polyp segmentation. (a) 3D polyp in five contiguous slices; (b) segmentation results without shape feature; (c) segmentation results using the proposed method.

The performance of our superpixel-based graph cut algorithm was also compared with that of a standard pixel-based method, where the graph was constructed on each pixel and only the intensity was considered in the energy function.

Figure

Comparison of pixel-based and superpixel-based graph cut algorithms on three different types nodules.

Nodule type | Pixel-based method | superpixel-based method | ||

Number of vertices | Time(s) | Number of vertices | Time(s) | |

GGO (a) | 46195 | 16 | 946 | 1.2 |

part-solid (b) | 22740 | 6.6 | 409 | 0.6 |

Solid (c) | 17254 | 3.4 | 208 | 0.5 |

Comparison of pixel-based and superpixel-based graph cut algorithms on three different types of nodules. For each nodule in (a), (b), and (c), the first row shows the 3D nodule in the original CT subimages; the second row shows segmentation results based on the pixel-based method, and the last row shows results using the proposed superpixel based method.

Since the superpixels from the five-dimensional JSIS mean shift algorithm express the local structure of the data, it produces better results and improves the speed significantly.

In this section, the proposed algorithm has been evaluated on a large set of 130 thoracic CT scans with a slice thickness ranging from 0.5 mm to 2.0 mm. The tube current ranged from 30 mA to 250 mA. Each scan was read individually by three experienced thoracic radiologists to produce a gold standard of 181 nodules (101 solid nodules and 80 GGO nodules). Among those solid nodules, 28 nodules are isolated nodule (in this paper,

Example of different types of solid nodules. (a) Isolated nodules; (b) vascular nodules; (c) pleural nodules.

Dice’s coefficient (

Figure

Dice coefficients based on the two different methods for all the solid nodules.

For comparison, we split all the solid nodules into three different types (isolated, vascular and pleural nodules) and evaluated the segmentation performance for each type separately. Table

Summary of Dice coefficients for different types of solid nodules based on two different methods.

Type | Number nodules | 4D-based method | Proposed method | ||

Mean coefficient | Std ( | Mean coefficient | Std ( | ||

Isolated | 28 | 0.80 | 0.05 | 0.81 | 0.04 |

Vascular | 53 | 0.68 | 0.1 | 0.80 | 0.06 |

Pleural | 20 | 0.67 | 0.09 | 0.73 | 0.06 |

For the isolated nodules, both methods give good results. For these nodules, the mean Dice coefficient for the 4D method is 0.80, and 0.81 for the 5D method. As can be seen in Figure

For both vascular nodules and pleural nodules, the proposed method gives a much better mean Dice coefficient. Especially for vascular nodules, the mean Dice coefficient increased from 0.68 to 0.8 with

Example of a pleural nodule segmentation. Top row: original 3D nodule on three contiguous slices; bottom row: segmentation results based on the proposed method for which a small part of nodule on the last slice was incorrectly segmented.

Another example of a pleural nodule segmentation. Top row: original 3D nodule on three contiguous slices; bottom row: segmentation results based on the proposed method in which the segmented nodule includes a small amount of lung wall.

The proposed method has also been evaluated on 80 GGO nodules (with pure and part opacified components). Most GGO nodules exhibit low contrast, irregular shapes and are often attached to vessels with very similar intensities. Figure

Dice coefficients for GGO nodules based on the proposed method.

Segmentation results on a GGO nodule; (a) one slice of the GGO nodule; (b) initial foreground seeds located on the top border of the nodule; (c) initial background; (d) segmented nodule which is missing parts of the opacified component.

GGO nodules pose a more difficult segmentation challenge than solid nodules. The proposed method provides a robust approach that combines both of local shape features and intensity. More discussion for GGO nodule segmentation will be provided in the next section.

Experimental results presented in the previous section demonstrate the promise and generalization of the proposed method to different types of lung nodules in CT. Most of nodules can be properly delineated from the lung parenchyma despite the presence of other nontarget structures such as vessels or the lung wall.

Based on the quantitative evaluation shown in Table

Figures

Also, in this paper, the initial foreground seeds for graph cut segmentation are automatically obtained based on spherical concentration. For our data, the initialization, described in Section

The proposed method has also been applied to colonic polyp segmentation. Visual analysis (Figures

We have presented a new, automatic method of extracting lesions from CT data. A five-dimensional JSIS mean shift clustering is firstly used to produce superpixels comprised of intensity and shape index mode maps. A graph cut algorithm is then applied using a novel energy formulation that considers not only image intensity but also shape. The initial foreground and background can be automatically obtained based on shape index concentration. The JSIS features provide rich information for lesion segmentation. Both by visual inspection on different types of lesions (lung nodules and colonic polyps), as well as using a quantitative evaluation on 101 solid nodules and 80 GGO nodules, demonstrate the potential of the proposed method. The method can not only successfully segment lesions adjacent to structures of similar intensity but different shape, but also can correctly identify some part of lesions with different intensity (due to PVE in CT imaging) but similar shape.