Image Segmentation by Edge Partitioning over a Nonsubmodular Markov Random Field

Edge weight-based segmentation methods, such as normalized cut or minimum cut, require a partition number specification for their energy formulation.The number of partitions plays an important role in the segmentation overall quality. However, finding a suitable partition number is a nontrivial problem, and the numbers are ordinarilymanually assigned.This is an aspect of the general partition problem, where finding the partition number is an important and difficult issue. In this paper, the edge weights instead of the pixels are partitioned to segment the images. By partitioning the edge weights into two disjoints sets, that is, cut and connect, an image can be partitioned into all possible disjointed segments. The proposed energy function is independent of the number of segments. The energy is minimized by iterating the QPBO-α-expansion algorithm over the pairwise Markov random field and the mean estimation of the cut and connected edges. Experiments using the Berkeley database show that the proposed segmentation method can obtain equivalently accurate segmentation results without designating the segmentation numbers.


Introduction
There are numerous approaches and applications for unsupervised image segmentation in computer vision.Many different theories are proposed for varying the roles of the unsupervised segmentation.As a low level vision problem, an image can be simplified by oversegmentation using a number of different approaches, such as mode-seeking mean shift, multilevel thresholding, histogram-based neural networks, superpixel algorithms, and various graph-based methods [1][2][3][4].Conversely, semantic segmentation is attempted for simultaneous detection, recognition, and segmentation [5].
Generally, the role of unsupervised segmentation falls between image simplification and full semantic segmentation, where semantically meaningful segments are expected to be found but not necessarily recognized.Segmentation is posed as an image-coloring problem that minimizes specific energy functions.Energy functions can be optimized using stochastic methods such as deterministic annealing and stochastic clustering [6][7][8][9][10].For graph theoretic segmentation approaches, the spectral method and graph cut are efficient deterministic optimization methods [11][12][13].Another traditional segmentation method is the variational method, which evolves boundary contours in a level set framework [14,15].
The edge weight-based segmentation methods have evolved together with graph partition problems.When edge weights are all positive, the minimum cut can be found; however, the minimum cut has bias toward smaller cuts.Adding negative edge weights can prevent the problem so the graph becomes nonsubmodular; however, the problem becomes NP-hard [16].Different algorithms have been introduced to estimate the correlation in clustering problem [17,18].In contrast, Shi and Malik normalized nonnegative edge weights so the bias toward smaller cuts was eliminated [11].
For the graph theoretic segmentation and level set methods, the number of segments must be predefined.The segment number choice greatly influences the quality of segmentation, especially for a normalized cut.Nonetheless, there have been attempts to solve this problem.The number of segments can be controlled by setting the threshold value to the recursive normalized cut [11].For level set approaches, a four-color theorem was used to segment images with an arbitrary number of phases with one or two level set functions [19].However, these methods are still functions of , the number of segments.
In this paper, transforming the pixel clustering problem into an edge partition problem circumvents the segment number selection problem.Edges among adjacent pixels can represent dissimilarity or similarity weights.Two edge partitions are always sufficient for pixel-partitioning problems.An edge can be in a cut set or connected set, which can then be translated into a unique segmentation, as in Figure 1(c).The cut edges indicate that the two node labels are different, whereas the connected edges indicate that two nodes have the same labels.In most cases, however, the cut or connect assignments on the edges are not enough to define a specific segmentation configuration, as in Figure 1(d).Random cut and connect assignments on the edges may result in contradiction of the node labels.However, under the pixel coloring framework, cut and connect assignments on the edges are defined concurrently with pixel labels, and inconsistencies, such as those in Figure 1(d), are prevented.
Under the pixel-labeling framework, a label number selection problem arises.Although the label number selection might seem similar to the segment number selection problem, there are subtle differences.First, pixels do not need to use all label assignments; thus, low numbers of segments are possible with large numbers of labels.Second, under the four-color map theorem, the maximum number of labels for two-dimensional (2D) segmentation can be as low as four.The four-color map theorem states that any 2D map can be colored with intact borders using a maximum of four colors [26].This theorem can be translated directly to the segmentation problem; any 2D image segmentation can be represented using four labels [19].
In the following sections, a new energy function is introduced for image segmentation through the edge partition.The edge partitions can uniquely define the image segmentation with the hard constraints enforced by the imagelabeling framework.Next, an energy minimization algorithm is proposed for the edge partitioning.The experimental section discusses tests of the proposed algorithm using the Berkeley image segmentation database.

Pixel Clustering
Image segmentation can be viewed as a pixel-partitioning problem.Many image segmentation methods borrow their ideas from the general partitioning techniques.The -means algorithm minimizes the following function and segments the image into  regions.Consider where   is the pixel feature value and   is the mean values of the respective partition   .The -means algorithm minimizes the sum of the squared distance from the mean of each partition.The energy function must have a fixed segmentation number.However, estimating the number of segments is a difficult task, and the number of partitions is often designated by human discretion.

Edge Partitions
An image can be represented as a set of nodes and edges by a graph  = (V, E).An edge

Energy Function.
The segmentation problem is formulated in terms of edge partitions.The edges can be partitioned into two sets  off (cut) and  on (connect), such that  off ∪  on = E and  off ∩  on = 0.If an edge is in  on , the pixel nodes connected by the edge have the same label.Otherwise, if an edge is in  off , the pixel nodes connected by the edge have a different label In (2), (, V) is an edge between pixel nodes  and V.The pixel labels for pixels  and V are denoted by   and  V , respectively.(, V) is a positive edge weight between pixel nodes  and V. (, V) can be a similar or dissimilar measure between the two pixel nodes.A simple example of (, V) is the absolute difference between two pixel colors.Thus, if the colors between the two pixels have a large difference, the edge will likely be in  off .If the two pixel colors have a small difference, the edge should be in  on .The mean edge weight values of the  on and  off edge sets are found in the following equation: and then the energy function associated with the edge partitions can be defined by following equation: | ⋅ | is the cardinality of the set.The energy function ( 4) is the same as the -means algorithm in (1) except that the number of partitions is set to  = 2.The proposed energy function has two mean centers, but it also has hard constraints in (2).Regardless of the segmentation number, there can only be two partitions for the edges, cut  off and connected edges  on .The proposed energy function breaks down into an image-labeling problem in order to maintain the label consistency conditions of (2).The image label state x = ( 1 ,  2 , . . .,  |V| ) that minimizes (4) under (3) and (2) constraints is the proposed segmentation state.The number of labels must be at least two to avoid division by zero in (3).Under the well-known four-color map theorem, four labels {1, 2, 3, 4} are sufficient to define all possible segment configurations for 2D images [19].

Optimization.
Given the image label state x, the mean values   off ,   on can be estimated as in (3).Otherwise, if   off and   on are kept constant, the image label state x can be found by optimizing the following pairwise energy function: If the labels between edges are not the same, the edge is considered to be in the  off cut set; otherwise, it belongs to the  on connected set.With   off and   on constants, minimizing ( 5) is equivalent to minimizing the edge partition function (4).
The multilabel pairwise energy function ( 5) can be solved by QPBO--expansion.QPBO--expansion optimizes the multilabel MRFs by iteratively expanding a single label using graph cut [27].Graph cut can find the optimal expansion if the expansion is submodular.In this problem, the expansions are nonsubmodular.The pairwise potentials for QPBO-expansion, where   ,  V ∈ x is the current label state, can be defined as follows: This nonsubmodular binary labeling problem can be approached using the QPBO algorithm [28] with the possibility of a large number of unlabeled nodes.Recently introduced, QPBO improve (QPBOI) algorithm can cope with unlabeled regions [28]; however, this algorithm is not as efficient as the graph cut which minimizes the submodular potentials.The QPBOI algorithm can randomly improve the solution, but iterations of the improved steps can be time-consuming for large numbers of nodes.Similar to the original -means algorithm, good initialization is helpful to the optimization.The initial estimation of the means,   off and   on , can be found by a -means algorithm minimization of edge partitions (4) without the labeling constraint of (2).To estimate the initial state x, the pixel clustering -means algorithm (1) can be used.The general framework is illustrated in Algorithm 1.

Edge Weights.
Various examples of the edge partition segmentation results using the color distance edge weights are shown in Figure 2 for the MSRC image database [29].The color distance from the neighboring pixels is sufficient for some image segmentation problems, but more rigorous weight calculations are often suited for semantic segmentation.Instead of proposing new edge weight calculations, an existing state-of-the-art contour detection algorithm is incorporated.
The global probability of the boundary (GPB) edge detection method [25,30], which scored best for the Berkeley database (http://www.cs.berkeley.edu/projects/vision/bsds), is employed as the edge weights.The edge weights can be connected between the pixel nodes, and the proposed edge partitioning algorithm can be implemented.Figure 3 shows the other segmentation results under the pixel-to-pixel edge connections.Although Figures 3(a   a good segmentation in Figures 3(c) and 3(d).The QPBOI algorithm often fails in the presence of a large number of nodes.Thus, to reduce both the computational time and the chance of failure in the QPBOI algorithm, the oversegmentation process is adopted from [25] in this segmentation.The edges are connected between the superpixels instead of the pixels.The number of oversegments is between 400 and 1000.
The edge partitioning algorithm segments a BSDS image average in under 5 seconds.

Evaluation
The proposed edge partition approach is evaluated using the popular Berkeley image database.The set contains 300  images with at least four human segment annotations per image.The three quantitative evaluation methods used are as follows: Probabilistic Rand Index (PRI) [31], Variation of Information (VoI) [32], and Boundary Displacement Error (BDE) [33].Global Consistency Error (GCE) [34] is not included in this evaluation.GCE measures the extent to which one segmentation can be viewed as a refinement of another.However, one pixel per segment and one segment for an entire image can give zero error for GCE [31].GCE favors extremely oversegmented or undersegmented results, and both cases are unwanted for a semantic segmentation.GCE is deemed to be an inconsistent evaluation method.
The evaluation methods used in this study are PRI, VoI, and BDE.PRI counts the number of consistent labels between the segmentation and the ground truth.VoI measures the segmentation randomness that cannot be explained by the ground truth.BDE is the average displacement error or the boundary pixels between two segmentation results.PRI counts the correctness in segmentation, while VoI and BDE measure the errors between the segmentation and ground truth.In the first subsection, the proposed method is evaluated against various segmentation methods.In the second subsection, the comparison between the proposed and the merge-threshold methods is demonstrated using the same edge weights.
For PRI measurements, the merge-threshold method of CtoR ranks first.The proposed segmentation ranks first for VoI and BDE.The CtoR method is available to the public by the authors.The threshold value for the CtoR method was chosen to be 80 for its highest average ranking.A number of segmentation results of CtoR and of the proposed EPartition are shown in Figure 4.For the normalized cut and fully connected spectral segmentation, the segmentation number is chosen for each image and is excluded from the rankings.
CtoR and EPartition use the same edge weights; thus, their performances are similar.However, in CtoR, a mergethreshold algorithm is used for segmentation.Different thresholds among integer intervals {1, . . ., 255} are shown for the PRI, VoI, and BDE evaluation methods in Figure 5. Generally, PRI and BDE favor oversegmentation and VoI favors undersegmentation.The optimal threshold value is generally smaller for PRI and BDE than VoI.
In contrast, the edge partitioning segmentation is independent of a threshold value.Figure 5 shows the performance of the CtoR merge-threshold method in terms of threshold optimal score for each segmentation evaluation approach.In contrast, the proposed EPartition is independent of the threshold values and finds the approximate optimal segmentation for all evaluation approaches.values.The proposed EPartition segmentation evaluation scores for PRI, BDE, and VoI are very close to the highest evaluation score of CtoR.However, the merge-threshold method in CtoR requires a specific threshold value for each segmentation evaluation method.The advantage of EPartition is that correct segmentation is possible without the designation of segmentation number or a threshold value.

Comparison to Trained Threshold.
In previous experiments, EPartition was shown to have competitive performance with CtoR when the optimal threshold value is handpicked for CtoR.In this section, the threshold value is trained from the Berkeley 300 set and the segmentation performances are compared to the Weizmann segmentation set [35].The Weizmann set contains 100 images with three human segmentation annotations.
In Table 2, the segmentation evaluations of the CtoR and EPartition methods are compared.There is a minuscule difference for PRI and small differences in the VoI evaluation methods.For BDE evaluation, EPartition clearly outperforms CtoR method.The trained threshold value was not robust for different segmentation evaluation approaches.By partitioning the edges through minimizing the mean squared distance, the proposed EPartition shows adaptive performance among the three evaluation methods.Various comparative segmentation results are shown in Figure 6.

Conclusion and Future Works
In this paper, image segmentation by edge partitioning is proposed.In contrast with previous edge weight-based segmentation methods, such as normalized cut, the proposed  method is independent of the number of segments.Furthermore, compared with the previous segmentation techniques, edge partitioning remains competitive without the need for the segmentation number selection.Segmentation by edge partitioning has shown to be competitive with previous segmentation techniques in the Berkeley database.The advantage of the proposed method lies in its adaptive nature for handling edge weights without threshold values or segment number assignments.
The proposed algorithm can be extended to general partitioning problems.Four labels are sufficient when segmenting 2D images.However, for fully connected graphs, the number of labels can be arbitrarily large.If a maximum number of labels are chosen, the edge partitioning method can be incorporated into a general partition problem without designating the specific number of partitions among nodes.

Figure 1 :
Figure 1: An image can be segmented by partitioning edges into two sets.Cut (dotted red) and connected (solid black) edge sets can be translated into a unique segmentation as in (c).However, it is also possible to have edge partitions that contradict the label assignments as in (d).By finding the image labeling that minimizes the edge partition energy, edge partitions like (d) are prevented, and a consistent image segmentation becomes possible as shown in (a) and (b).
) and 3(b) show a good segmentation result, the QPBOI algorithm cannot obtain

( 1 )
Estimate image label state x with the -means algorithm on pixels,   = 4.(2) Estimate   off and   on by the -means algorithm minimization of(4) without the labeling constraint of (2).(3)Estimate the image label state x using the QPBOI--expansion.(keep  off and   on constant) (4)Estimate   off and   on from the image label state x.(5) If   off and   on are unchanged, terminate.Else, repeat steps 3 and 4.Algorithm 1: Minimizing Edge Partition.

Figure 2 :Figure 3 :
Figure 2: Even with the simple color distance weight, the edge partitions can produce adequate segmentation results.Some of the segmentation results from the MSRC database are shown.

Figure 4 :
Figure 4: The segmentation results from Table 1 are shown for CtoR and the proposed EPartition.

Figure 5 :
Figure5: The segmentation evaluation (PRI, VoI, and BDE) versus threshold values is plotted for CtoR.Different threshold values give an optimal score for each segmentation evaluation approach.In contrast, the proposed EPartition is independent of the threshold values and finds the approximate optimal segmentation for all evaluation approaches.

Figure 6 :
Figure 6: The segmentation results from Table 2 are shown for CtoR and the proposed EPartition.

Table 1 :
Constant parameters are maintained for the Berkeley image set and the test methods.The top ranking results are written in bold, and the rankings are in parenthesis.For segmentation methods with a * , a different segmentation number is assigned optimally for each image.

Table 2 :
The segmentation results for the Weizmann image set are summarized.The threshold value of CtoR is trained from the Berkeley set.Superior results are written in bold.Integrative Diagnosis-Treatment Support Software System for Cardiovascular Diseases).This work was also supported by Hankuk University of Foreign Studies Research Fund.