^{1}

^{2}

^{3}

^{4}

^{4}

^{4}

^{5}

^{1}

^{2}

^{3}

^{4}

^{5}

A multilevel aggregation method is applied to the problem of segmenting live cell bright field microscope images. The method employed is a variant of the so-called “Segmentation by Weighted Aggregation” technique, which itself is based on Algebraic Multigrid methods. The variant of the method used is described in detail, and it is explained how it is tailored to the application at hand. In particular, a new scale-invariant “saliency measure” is proposed for deciding when aggregates of pixels constitute salient segments that should not be grouped further. It is shown how segmentation based on multilevel intensity similarity alone does not lead to satisfactory results for bright field cells. However, the addition of multilevel intensity variance (as a measure of texture) to the feature vector of each aggregate leads to correct cell segmentation. Preliminary results are presented for applying the multilevel aggregation algorithm in space time to temporal sequences of microscope images, with the goal of obtaining space-time segments (“object tunnels”) that track individual cells. The advantages and drawbacks of the space-time aggregation approach for segmentation and tracking of live cells in sequences of bright field microscope images are presented, along with a discussion on how this approach may be used in the future work as a building block in a complete and robust segmentation and tracking system.

There is extensive current interest in the high-content, high-throughput screening of live cell populations, and the experimental techniques being developed for this purpose are leading to important biological insights with applications to new clinical therapies [

Figure

Bright field microscope image with approximately two dozen cells on a grey background. Some interior structure can be discerned in cells (including the cell membrane, the dark grey cytoplasm, and the lighter cell nucleus with dark nucleoli inside). Cells that are close to division appear as bright, nearly circular shapes. Some cells are touching, and some cell parts overlap. Our goal is to obtain separate segments for each cell in the image.

Our goal is to obtain separate segments for each cell in images like Figure

We use a variant of the Segmentation by Weighted Aggregation technique [

In the second part of the paper, preliminary results are presented for applying the multilevel aggregation algorithm in space time to temporal sequences of microscope images, with the goal of obtaining space time segments (object tunnels) that track individual cells. This parallels previous research results on space time segmentation using level set methods [

While the general ideas and concepts of the SWA framework are described in several places [

Automatic segmentation and tracking of live cells in bright field microscopy images is a difficult task [

For example, the experiments reported in [

In other types of experiments, fluorescent markers are used to measure concentrations of certain specific proteins that are under study. A limited number of fluorescent marker channels with nonoverlapping spectra (typically up to three) are used in combination. If one or two of these channels are used solely for tracking purposes, the number of channels available for measuring protein concentrations is reduced, which may be a significant limitation.

For these reasons, it can be an advantage if methods can be derived that manage to track cells based on bright field images [

In addition, bright field images typically contain many features of interest, and accurate segmentation methods for bright field images are intrinsically useful since they allow the study of cell morphology and internal cell structure, and their dynamics. For example, the morphology of a cell can be indicative of cell health or can indicate various stages of pathology [

The microscope images used in this paper (including Figure

The rest of this paper is structured as follows. In Section

In this section, we describe the multilevel segmentation algorithm that we employ in this paper. We start with an overview of the basic SWA algorithm from [

Figure

Schematic representation of the SWA algorithm.

We now give a more detailed description of the algorithm.

The problem of image segmentation can be viewed in terms of segmentation of a weighted undirected graph, with each node of the graph representing a pixel, and each edge corresponding to a link between neighbouring nodes, weighted by the similarity in intensity between the two neighbouring pixels. The SWA algorithm starts with this weighted graph on the finest level (which we call level 1) and forms a weighted graph of reduced size on level 2, with the nodes of the level-2 graph representing overlapping blocks of level-1 nodes. Each level-2 block is formed around a level-1 seed point, which is called a

In more specific terms, we start from the pixel graph of the original image, with the intensities of the pixels stored in level-1 intensity vector

On each level, the blocks are tested for saliency: a salient (or “prominent”) block is a block that is sufficiently different from all the blocks it is connected to, as determined by a saliency measure. Once a block is designated salient, it is not allowed to merge with other blocks on coarser levels. (In our implementation, salient blocks are propagated to all coarser levels.) The coarsening process terminates at the level where all blocks have become salient, at which point we have found all segments in the image. In Figure

The right branch of the diagram in Figure

The SWA algorithm is in many ways similar to AMG methods for solving linear systems of equations [

Figure

Flow chart for the algorithm with the recursive and nonrecursive parts.

The following are further details and enhancements of the SWA algorithm.

In order to coarsen the graph at the current level, we use the so-called first pass of the standard AMG coarsening algorithm [

Calculating

Another blockwise quantity we use to better connect similar blocks is multilevel variance in intensity (as a measure of texture) [

In order to avoid small segments, we only allow detection of salient segments on levels larger than level

In summary, we list the free parameters in our algorithm, to be chosen such that correct segmentation is obtained for the application at hand: top-level intensity scaling factor

We propose a saliency measure that is a modified version of the saliency measures used in [

The saliency measures used in [

On each level

The SWA algorithm seeks segments (Boolean vectors

At level

Saliency measure (

Consider Figure

A simple example image with a square block of white pixels in the center. We assume that the white pixels are aggregated to a single, nonoverlapping block on level 2.

Let us now calculate the

First, we want to show that diagonal element

Next we want to show that

Then multiply this with row

It is now easy to see that the

With these interpretations of the diagonal elements of

Example shapes for analyzing scale behaviour of saliency measures.

Small

Large

In order to remedy the potential scaling difficulties of saliency measure (

We have experimented extensively comparing the scale invariant saliency measure (

To finalize this section on a scale invariant saliency measure, we want to make three remarks. First, the example in Figure

In this section we evaluate the performance of the multilevel segmentation algorithm by applying it to bright field phase contrast cell microscopy images. First we demonstrate the advantage of segmenting taking the multilevel intensity variance into account, versus not taking it into account.

The image of Figure

Parameters used: (b)

Original image

1 segment found, not using variance

2 segments found (patterned shape and background), using variance

Next we apply our algorithm to the single isolated cell of Figure

Parameters used: (b)

Original image

4 segments found (including the background segment), not using variance

2 segments found (including the background segment), using variance

Figures

Parameters used: (b)

Original image

4 segments found

Parameters used: (b)

Original image

4 segments found

Parameters used: (b)

Original image

5 segments found

Finally, Figure

Parameters used: (b)

Original image

4 segments found

5 segments found

The results of Figures

A first type of improvement can be offered by including more features in the feature vector. For example, geometrical shape moments can be calculated for overlapping blocks at all levels, giving information about block shape and orientation that can be used to preferentially group together blocks that have similar shape or orientation [

A second type of improvement, however, may be achieved by simply considering the extra information that is present in temporal sequences of images. The SWA algorithm can take immediate advantage of this information by applying it directly to these image sequences in space time. However, before exploring SWA segmentation in space time in Section

To apply the algorithm to an image, we need to choose values for segmentation parameters

To increase the contrast level of the image, increase top-level intensity scaling factor

For images containing broad bright or dark boundaries of regions (such as the halos in the bright field cell images), we do not want these boundaries to become salient segments themselves. Since a large coarse-level intensity rescaling factor

In order to separate desired segments that differ more in average intensity and less in intensity variance, choose a larger intensity rescaling factor

If the algorithm finds too many (small) segments, try

decreasing

decreasing

increasing

On the contrary, if too few (large) segments are found, then shift the parameter values in the opposite direction

The process for segmenting image sequences in space time (see next section) is similar but we usually start with a smaller

In this section we describe results obtained when applying the multilevel SWA algorithm to sequences of images. In our experimental technique we take images frequently enough that moving cells overlap significantly between frames. (The motion of the cells is slow compared to the image frequency.) Cell trajectories in space time thus form “object tunnels’’ that are found efficiently by the SWA algorithm. The extra temporal information makes it easier to resolve difficult cases such as touching cells, dividing cells, and cells that temporarily overlap. The resulting space time segments can also be used for tracking cell motion.

By stacking up the images, the problem of segmenting multiple images can be viewed as segmenting one three-dimensional (3D) data set. It is not difficult to modify the SWA algorithm to suit a 3D problem because the algorithm is already designed to coarsen arbitrary graphs, which can represent geometric grids of any dimension. The details of this simple modification are given in Appendix

We now describe segmentation results for sequences of bright field cell images.

Figure

Original image sequence.

2 segments found with parameters

Figure

3D representation of the cell segment.

Looking down from

Looking down from

The next set of images (Figure

Original image sequence.

Annotations by a human expert.

2 segments found with parameters

3D representation of the segments.

Looking down from

Looking down from

Figure

Original image sequence.

2 segments found with parameters

3D representation of the segments.

Looking down from

Looking down from

3 segments found with parameters

3D representation of the segments.

Looking down from

Looking down from

Multilevel algorithms often enjoy the desirable properties of fast execution time and low memory cost that are linearly proportional to the number of data elements [

We test runtime scaling by fixing a set of segmentation parameters and running the algorithm on images of different resolution, from

Runtime versus image size. (Single images.)

Runtime versus image size. (Space time sequences of images.)

In this paper, we have investigated the use of a multilevel aggregation algorithm as a method for segmenting live cell bright field microscope images. We use a variant of the Segmentation by Weighted Aggregation (SWA) technique [

While the robust application of multilevel aggregation to bright field cell images requires further research, it is already clear that the SWA segmentation approach may have several advantages over more commonly used segmentation techniques, which include level set, active contour, and watershed methods. First of all, multilevel aggregation is fast and linearly scalable in the number of pixels to be segmented. The algorithm is conceptually simple since there is no need to define initial level set seeds, or extract markers as in watershed. The space time segmentation approach uses extra temporal information that makes it easier to resolve difficult cases like touching cells, dividing cells and cells that temporarily overlap. Nevertheless, it seems that extensive further research is required to handle these difficulties properly. The feature vector for each resulting segment contains multilevel information about the segment that can be exploited in various ways. It can be used to classify segments as background, dividing cell, or regular cell, and it can potentially be used (in 3D) to separate tracks of dividing cells from tracks of regular cells, to aid the proper handling of cell division events. It can also be used to study cell morphology and its dynamics.

The multilevel segmentation approach provides a single, unified technique that produces accurate cell trajectory segments (for moderately complex cases so far) and gives a lot of additional useful information. It is clear that it will have to be combined with other advanced methods of geometric, modeling and statistical nature in order to obtain a complete and robust segmentation and tracking system, along the lines of the sophisticated and comprehensive segmentation and tracking methods that recently have been described (see [

Output: an

Define global segmentation parameters:

Initialize level 1 variables

Current level

Current number of nodes

Label the pixels

Let coupling matrix

Let variance matrix

Let weighted boundary length matrix

Let area matrix

Let boundary length matrix

Let saliency vector

Determine overlapping segments by calling the recursive graph segmentation function:

Assign pixels uniquely to segments: set

Output:

If

Coarsen the current graph: set

Let

Let

If

Let interpolation matrix

Let column-scaled interpolation matrix

Let coarse-level intensity vector

For each block on the current coarse level,

Define coarse-level coupling matrix

Let

Rescale using coarse-level intensity:

If

Let coarse-level weighted area matrix

Let coarse-level area matrix

Define coarse-level weighted boundary length matrix

Let

Define coarse-level boundary length matrix

Let

Let coarse-level saliency vector

If

Recursively segment the coarse graph: Let

Find the current segmentation matrix from the coarse-level segmentation matrix: Let

Sharpen overlapping segments: For all

Return current segmentation matrix

Input:

Output:

Let

Let

Let

Let

For

While not all

Let

Set

Let

For all

Set

Let

For all

Let the vector of

Return

Let

Read in the

(The global segmentation parameters are defined as before.)

Initialize variables

Set the current number of nodes

Obtain the

The other variables are defined as before.

Call the recursive function as before.

Everything else in the algorithm, including the recursive part and the coarsening, remains unchanged.