Most Active Contour Models (ACMs) deal with the image segmentation problem as a functional optimization problem, as they work on dividing an image into several regions by optimizing a suitable functional. Among ACMs, variational level set methods have been used to build an active contour with the aim of modeling arbitrarily complex shapes. Moreover, they can handle also topological changes of the contours. Self-Organizing Maps (SOMs) have attracted the attention of many computer vision scientists, particularly in modeling an active contour based on the idea of utilizing the prototypes (weights) of a SOM to control the evolution of the contour. SOM-based models have been proposed in general with the aim of exploiting the specific ability of SOMs to learn the edge-map information via their topology preservation property and overcoming some drawbacks of other ACMs, such as trapping into local minima of the image energy functional to be minimized in such models. In this survey, we illustrate the main concepts of variational level set-based ACMs, SOM-based ACMs, and their relationship and review in a comprehensive fashion the development of their state-of-the-art models from a machine learning perspective, with a focus on their strengths and weaknesses.
Image segmentation is the problem of partitioning the domain
Traditionally, image segmentation methods can be classified into five categories. The first category is made up of threshold-based segmentation methods [
The second category of methods is called boundary-based segmentation [
The third category of methods is called region-based segmentation [
The fourth category of methods is learning-based segmentation [
The last category of methods is energy-based segmentation [
More specifically, ACMs usually deal with the segmentation problem as an optimization problem, formulated in terms of a suitable “energy” functional, constructed in such a way that its minimum is achieved in correspondence with a contour that is a close approximation of the actual object boundary. Starting from an initial contour, the optimization is performed iteratively, evolving the current contour with the aim of approximating better and better the actual object boundary (hence the denomination “active contour” models, which are used also for models that evolve the contour but are not based on the explicit minimization of a functional [
Although ACMs often provide an effective and efficient means to extract smooth and well-defined contours, trapping into local minima of the energy functional may still occur, because such a functional may be constructed on the basis of simplified assumptions on properties of the images to be segmented (e.g., the assumption of Gaussian intensity distributions for the sets
A summary of the Active Contour Models (ACMs) reviewed in the paper.
ACM | Reference | Regional information | Main strengths/advantages | Main limitations/disadvantages | |
---|---|---|---|---|---|
Local | Global | ||||
GAC |
[ |
No | No | Makes use of boundary information. | Hardly converges in the presence of ill-defined boundaries. |
Identifies accurately well-defined boundaries. | Very sensitive to the contour initialization. | ||||
|
|||||
CV |
[ |
No | Yes | Can handle objects with blurred boundaries in a global way. | Makes strong statistical assumptions. |
Can handle noisy objects. | Only suitable for Gaussian intensity distributions of the subsets. | ||||
|
|||||
SBGFRLS |
[ |
No | Yes | Very efficient computationally, and robust to the contour initialization. | Makes strong statistical assumptions. |
Gives efficient and effective solutions compared to CV and GAC. | It is hard to adjust its parameters. | ||||
|
|||||
LBF |
[ |
Yes | No | Can handle complex distributions with inhomogeneities. | Computationally expensive. |
Can handle foreground/background intensity overlap. | Very sensitive to the contour initialization. | ||||
|
|||||
LIF | [ |
Yes | No | Behaves likewise LBF, but is computationally more efficient. | Very sensitive to noise and contour initialization. |
|
|||||
LRCV | [ |
Yes | No | Computationally very efficient compared to LBF and LIF. | Very sensitive to noise and contour initialization. |
|
|||||
LSACM |
[ |
Yes | No | Robust to the initial contour. | Computationally expensive. |
Can handle complex distributions with inhomogeneities. | Relies on a probabilistic model. | ||||
|
|||||
GMM-AC |
[ |
No | Yes | Exploits prior knowledge. | Makes strong statistical assumptions. |
Very efficient and effective. | Requires a huge amount of supervised information. | ||||
|
|||||
SISOM |
[ |
No | No | Localizes the salient contours using a SOM. | Topological changes cannot be handled. |
No statistical assumptions are required. | Computationally expensive and sensitive to parameters. | ||||
|
|||||
TASOM |
[ |
No | No | Adjusts automatically the number of SOM neurons. | No topological changes can be handled. |
Less sensitive to the model parameters compared to SISOM. | Sensitive to noise and blurred boundaries. | ||||
|
|||||
BSOM |
[ |
No | Yes | Exploits regional information. | Topological changes cannot be handled. |
Deals better with ill-defined boundaries compared to SISOM and TASOM. | Computationally expensive and produces discontinuities. | ||||
|
|||||
eBSOM |
[ |
No | Yes | Produces smooth contours. | Topological changes cannot be handled. |
Controls the smoothness of the detected contour better than BSOM. | Computationally expensive. | ||||
|
|||||
FTA-SOM |
[ |
No | Yes | Converges quickly. | Topological changes cannot be handled. |
Is more efficient than SISOM, TASOM, and eBSOM. | Sensitive to noise. | ||||
|
|||||
CFBL-SOM |
[ |
No | Yes | Exploits prior knowledge. | Topological changes cannot be handled. |
Deals well with supervised information. | Sensitive to the contour initialization. | ||||
|
|||||
CAM-SOM |
[ |
No | Yes | Can handle objects with concavities, small computational cost. | Topological changes cannot be handled. |
More efficient than FTA-SOM. | High computational cost compared to level set-based ACMs. | ||||
|
|||||
CSOM-CV |
[ |
No | Yes | Very robust to the noise. | Supervised information is required. |
Requires a small amount of supervised information. | Suitable only for handling images in a global way. | ||||
|
|||||
SOAC |
[ |
Yes | No | Can handle complex images in a local and supervised way. | Supervised information is required. |
Can handle inhomogeneities and foreground/background intensity overlap. | Sensitive to the contour initialization. | ||||
|
|||||
SOMCV |
[ |
No | Yes | Reduces the intervention of the user. | Is easily trapped into local minima. |
Can handle multimodal intensity distributions. | Deals with images in a global way. | ||||
|
|||||
SOM-RAC |
[ |
Yes | Yes | Robust to noise, scene changes, and inhomogeneities. | Very expensive computationally. |
Robust to the contour initialization. |
The paper is organized as follows. Section
To build an active contour, there are mainly two methods. The first one is an explicit or Lagrangian method, which results in parametric active contours, also called “Snakes” from the name of one of the models that use such a kind of parametrization [
In parametric ACMs, the contour
The parametric representation of a contour.
The main drawbacks of parametric ACMs are the frequent occurrence of local minima in the image energy functional to be optimized (which is mainly due to the presence of a gradient energy term inside such a functional), and the fact that topological changes of the objects (e.g., merging and splitting) cannot be handled during the evolution of the contour.
The difference between parametric active contour and geometric (or variational level set-based) Active Contour Models is that in geometric active contours, the contour is implemented via a variational level set method. Such a representation was first proposed by Osher and Sethian [
The geometric representation of a contour.
A common and simple expression for
In the variational level set method, expressing the contour
Accordingly, the evolution of the level set function
In order to guide efficiently the evolution of the current contour, ACMs allow to integrate various kinds of information inside the energy functional, such as local information (e.g., features based on spatial dependencies among pixels), global information (e.g., features that are not influenced by such spatial dependencies), shape information, prior information, and also a-posteriori information learned from examples. As a consequence, depending on the kind of information used, one can further divide ACMs into several subcategories, for example, edge-based ACMs [
In particular, edge-based ACMs make use of an edge-detector (in general, the gradient of the image intensity) to try to stop the evolution of the active contour on the true boundaries of the objects of interest. One of the most popular edge-based active contours is the Geodesic Active Contour (GAC) model [
For images with a high level of noise, the presence of the Edge Stopping Function may not be enough to stop the contour evolution at the right boundaries. Motivated by this issue, a novel edge-based ACM has been proposed in [
Since edge-based models make use of an edge-detector to stop the evolution of the initial guess of the contour on the actual object boundaries, they can handle only images with well-defined edge information. Indeed, when images have ill-defined edges, the evolution of the contour typically does not converge to the true object boundaries.
An alternative solution consists in using statistical information about a region (e.g., intensity, texture, and color) to construct a stopping functional that is able to stop the contour evolution on the boundary between two different regions, as it happens in region-based models (see also the survey paper [
Following [
After replacing
The
Compared to edge-based models, region-based models usually perform better in images with blurred edges and are less sensitive to the contour initialization.
Hybrid models that combine the advantages of both edge and regional information are able to control better the direction of evolution of the contour than the previously mentioned models. For instance, the Geodesic-Aided Chan-Vese (GACV) model [
Overall this model is faster, is computationally more efficient, and performs better than the conventional
In order to deal with images with intensity inhomogeneity, several authors have introduced in the SPF function terms that relate to local and global intensity information [
In general, global models cannot segment successfully objects that are constituted by more than one intensity class. On the other hand, sometimes this is possible by using local models, which rely on local information as their main component in the associated variational level set framework. However, such models are still sensitive to the contour initialization and may lead to object leaking. Some examples of such local region-based ACMs are illustrated in the following.
In general, the LBF model can produce good segmentations of objects with intensity inhomogeneities. Furthermore, it has a better performance than the well-known Piecewise Smooth (
The main idea of this model is to use the local image information to construct an energy functional, which takes into account the difference between the fitted image and the original one to segment an image with intensity inhomogeneities. The complexity analysis and experimental results showed that the LIF model is more efficient than the LBF model, while yielding similar results.
However, the models above are still sensitive to the contour initialization, and to high levels of additive noise. Compared to the two above-mentioned models, a model that has shown higher accuracy when handling images with intensity inhomogeneity is the following one.
The objective functional
The evolution of the contour in the
Equation (
A drawback of the
The evolution of the level set function in the LSACM model is controlled by the following gradient descent formulation:
From a machine learning perspective, ACMs for image segmentation can use both supervised and unsupervised information. Both kinds of ACMs rely on parametric and/or nonparametric density estimation methods to approximate the intensity distributions of the subsets to be segmented (e.g., foreground/background). Often, in such models one makes statistical assumptions on the image intensity distribution, and the segmentation problem is solved by a Maximum Likelihood (
Now, we briefly discuss some supervised ACMs, which take advantage of the availability of labeled training data. As an example, Lee et al. proposed in [
The two terms
The inclusion of supervised examples in ACMs can improve significantly their performance by constructing a Knowledge Base (KB), to be used as a guide in the evolution of the contour. However, state-of-the-art supervised ACMs often make strong statistical assumptions on the image intensity distribution of each subset to be modeled. So, the evolution of the contour is driven by probability models constructed based on given reference distributions. Therefore, the applicability of such models is limited by how accurate the probability models are.
Before discussing SOM-based ACMs, we shortly review the use of SOMs as a tool in pattern recognition (hence, in image segmentation as a particular case).
The SOM [ Initialize randomly the weights of the neurons in the output layer, and select suitable learning rate and neighborhood size around a “winner” neuron. For each training input vector, find the winner neuron, also called Best Matching Unit ( Update the weights on the selected neighborhood of the winner neuron. Repeat Steps (2)-(3) above selecting another training input vector, until learning is accomplished (i.e., a suitable stopping criterion is satisfied).
More precisely, after its random initialization, the weight
SOMs have been used extensively for image segmentation, but often not in combination with ACMs [
In [
Although SOMs are traditionally associated with unsupervised learning, in the literature there exist also supervised SOMs. A representative model of a supervised SOM is the Concurrent Self-Organizing Map (CSOM) [
We conclude mentioning that, when SOMs are used as supervised/unsupervised image segmentation techniques, the application of the resulting model usually produces segmented objects characterized by disconnected boundaries, and the segmentation result is often sensitive to the noise.
In order to improve the robustness of edge-based ACMs to the blur and to ill-defined edge information, SOMs have been also used in combination with ACMs, with the explicit aim of modelling the active contour and controlling its evolution, adopting a learning scheme similar to Kohonen’s learning algorithm [
The basic idea of existing SOM-based ACMs belonging to the class of edge-based ACMs is to model and implement the active contour using a SOM, relying in the training phase on the edge map of the image to update the weights of the neurons of the SOM, and consequently to control the evolution of the active contour. The points of the edge map act as inputs to the network, which is trained in an unsupervised way (in the sense that no supervised examples belonging to the foreground/background, resp., are provided). As a result, during training the weights associated with the neurons in the output map move toward points belonging to the nearest salient contour. In the following, we illustrate the general ideas of using a SOM in modelling the active contour, by describing a classical example of a SOM-based ACM belonging to the class of edge-based ACMs, which was proposed in [
Construct the edge map of the image to be segmented. Initialize the contour to enclose the object of interest in the image. Obtain the horizontal and vertical coordinates of the edge points to be presented as inputs to the network. Construct a SOM with a number of neurons equal to the number of the edge points of the initial contour and two scalar weights associated with each neuron; the points on the initial contour are used to initialize the SOM weights. Repeat the following steps for a fixed number of iterations: Select randomly an edge point and feed its coordinates to the network. Determine the best-matching neuron. Update the weights of the neurons in the network by the classical unsupervised learning scheme of the SOM [ Compute a neighborhood parameter for the contour according to the updated weights and a threshold.
Figure
The architecture of the SISOM-based ACM proposed in [
We conclude by mentioning that, in order to produce good segmentations, the SISOM-based ACM requires the initial contour (which is used to initialize the prototypes of the neurons) to be very close to the true boundary of the object to be extracted, and the points of the initial contour have to be assigned to the neurons of the SOM in a suitable order: if such assumptions are satisfied, then the contour extraction process performed by the model is generally robust to the noise. Moreover, differently from other ACMs, this model does not require a particular energy functional to be optimized.
In this subsection, we describe other SOM-based ACMs belonging also to the class of edge-based ACMs, and highlight their advantages and disadvantages.
The TASOM-based ACM can overcome one of the main limitations of the SISOM-based ACM, that is, its sensitivity to the contour initialization, in the sense that, for a successful segmentation, the initial guess of the contour in the TASOM-based ACM can be even far from the actual object boundary. Likewise in the case of the SISOM-based ACM, topological changes of the objects (e.g., splitting and merging) cannot be handled by the TASOM-based ACM, since both models rely completely on the edge information (instead than on regional information) to drive the contour evolution.
Figure
The architecture of the CFBL-SOM-based ACM proposed in [
Recently, a new class of SOM-based ACMs combined with variational level set methods has been proposed in [
Figure
The architecture of the CSOM-CV ACM proposed in [
In this paper, a survey has been provided about the current state of the art of Active Contour Models (ACMs), with an emphasis on variational level set-based ACM, Self-Organizing Map (SOM-) based ACMs, and their relationships (see Figure
Some relationships between variational level set-based ACMs and Self-Organizing Map (SOM-) based ACMs.
Variational level set-based ACMs have been proposed in the literature with the aim of handling implicitly topological changes of the objects to be segmented. However, such methods are usually trapped into local minima of the energy functional to be minimized. Then, SOM-based ACMs have been proposed with the aim of exploiting the specific ability of SOMs to learn the edge-map information via their topology preservation property, and reducing the occurrence of local minima of the functional to be minimized, which is also typical of parametric ACMs such as Snakes. This is partly due to the fact that such SOM-based ACMs do not rely on an explicit gradient energy term. Although SOM-based ACMs belonging to the class of edge-based ACMs can effectively outperform other ACM models in handling complex images, most of such SOM-based ACMs are still sensitive to the contour initialization compared to variational level set-based ACMs, especially when handling complex images with ill-defined edges. Moreover, such SOM-based ACMs have not usually the ability to handle topological changes of the objects. For this reason, we have concluded the paper presenting a recently proposed class of SOM-based ACMs, which takes advantage of both SOMs and variational level set methods, with the aims of preserving topologically the intensity distribution of the foreground and background in a supervised/unsupervised way and, at the same time, of allowing topological changes of the objects to be handled implicitly.
Among future research directions, we mention:
The authors declare that there is no conflict of interests regarding the publication of this paper.