Visual Sensor Based Image Segmentation by Fuzzy Classification and Subregion Merge

. The extraction and tracking of targets in an image shot by visual sensors have been studied extensively. The technology of image segmentation plays an important role in such tracking systems. This paper presents a new approach to color image segmentation based on fuzzy color extractor (FCE). Different from many existing methods, the proposed approach provides a new classification of pixels in a source color image which usually classifies an individual pixel into several subimages by fuzzy sets. This approach shows two unique features: the spatial proximity and color similarity, and it mainly consists of two algorithms: CreateSubImage and MergeSubImage. We apply the FCE to segment colors of the test images from the database at UC Berkeley in the RGB, HSV, and YUV, the three different color spaces. The comparative studies show that the FCE applied in the RGB space is superior to the HSV and YUV spaces. Finally, we compare the segmentation effect with Canny edge detection and Log edge detection algorithms. The results show that the FCE-based approach performs best in the color image segmentation.


Introduction
Image segmentation is a fundamental issue in image processing and computer vision as it is a preliminary process in many applications that need to segment an image into meaningful regions.Applications of segmenting gray-level images are very limited so studies on color image segmentation become interesting because color images provide much more information than gray-level ones.A number of image segmentation techniques have been proposed in past decades [1][2][3].A physical model in [4] is proposed for color image understanding.A widely used technique for color image segmentation is based on the idea of region growing.A color segmentation algorithm combining region growing and region merging processes is discussed in [5].The fuzzy growing and expanding approach presented in [6] uses histogram tables for fine segmentation.A method for segmentation of color texture region using the defined class maps is presented in [7].An approach to contrast analysis using fuzzy region growing is documented in [8].Color image segmentation using seeded region growing is presented in [9][10][11].Most pixel clustering algorithms work in the RGB color space in which the pixels are represented by red, green, and blue components.Clustering methods [12,13], for example, means fuzzy classification [14], are often considered as an unsupervised classification of pixels as a prior knowledge of the image.Both supervised and unsupervised segmentation of textured color images are presented in [15,16].The level setting approaches to process a color image for segmentation are discussed in [17,18].All of the proposed techniques above provide a crisp segmentation of images, where each pixel is classified into a unique subset.This classification may not reflect an understanding of images by humans very well, as the work [1] stated "the image segmentation problem is basically one of psychophysical perception, and therefore not susceptible to a purely analytical solution."Probably, one of the difficulties in color image segmentation is how to define color conception, since an understanding of colors in an image varies by different observers.
Considering uncertainty and ambiguity in many real applications, this paper presents an approach to color image segmentation based on a fuzzy color extractor (FCE).Its preliminary version appears in [19].The FCE is directly extended from the fuzzy gray-level extractor, which was applied to recognize landmarks on roads for autonomous vehicle navigation in [20][21][22]; that is, the fuzzy rules for extracting a road from a gray-level image are as follows: If the gray level of a pixel is close to the gray-level of the road,

Then one can make it black, or
Else one can make it to white.
If these are modified to the rules for extracting an object from a color image, then if the color components of a pixel closely matches the color components of an object, then one can extract this pixel as a Matched pixel of the object, or else one can extract this pixel as an Unmatched pixel.
The FCE viewed as a mapping operator of measuring colors is able to extract colors that are similar to a color pattern (seed).If a pixel is extracted it is defined as a Matched pixel; otherwise it is defined as an Unmatched pixel.In doing this, we develop procedures IniSeedList and UpdateSeedList for automatic selection of seeds.Our approach mainly consists of two algorithms: CreateSubImage and MergeSubImage.The CreateSubImage algorithm always extracts the seed as the first Matched pixel into a subimage and continually uses the FCE to check neighbors of each Matched pixel to grow the subimage.The subimage is created when neighbors' colors of all Matched pixels are "not close to" the seed's color.The CreateSubImage algorithm iteratively invokes UpdateSeedList to select new seeds to create next subimages until all the pixels in the image are processed.Different from all of the color segmentation techniques discussed above, the CreateSubImage algorithm most likely classifies an individual pixel into several subimages by fuzzy sets.As a result, two created subimages may share a common area, which can be considered as a color similarity metric for merging the subimages.The MergeSubImage algorithm first removes insignificant subimages from the set of subimages and merges significant subimages in order to extract meaningful segmented images.The insignificant subimages usually have a "too small" or a "too large" ratio of their area sizes over the source image.A "too small" insignificant subimage is mainly caused by a tiny fragment or a noise pixel in the source image, while a "too large" insignificant image is created by a "bad" seed.Then, the algorithm merges two subimages together according to their color similarity described by the size of their common pixels.We tested the proposed approach using exemplar color images from the database at UC Berkeley [23] in three different color spaces and discussed its effectiveness and robustness by comparing segmentation effect with two edge detection algorithms, namely, Canny edge detection and Log edge detection.

Three Color Spaces
In many color-based computer vision applications, it is very important to segment colors in an image, but it is very difficult due to the fact that colors are ambiguous in uncertain environments.In this study, we apply the FCE to segment colors in the image represented in the RGB, HSV, and YUV spaces.
2.1.RGB Color Space.RGB is the most traditional color space and is widely used in image segmentation because of its simplicity and fast processing [24,25].Colors consist of red (R), green (G), and blue (B) components, as shown in Figure 1(a).

HSV Color Space.
HSV is another color space widely used for image processing, where a color is decomposed into hue (H), saturation (S), and value (V).The distribution of color in the HSV color space is shown in Figure 1(b).The color components in the RGB space can be transformed into the HSV space by max = max (R, G, B) , Usually, the RGB values are between 0 and 255, so the HSV values are also between 0 and 255 through the transformation by (1).(

YUV Color
The YUV values are between 0 and 255 through the transformation by (2).The advantage of using the YUV color space for color segmentation is that the luminance Y is separated from the colors U and V.

Fuzzy Color Extractor
The fuzzy color extractor proposed here enables us to deal with such uncertainty.We will only discuss how to use the FCE-based algorithm defined in the RGB color space to segment color images, because the algorithm can directly process the color images in the HSV and YUV spaces, which are converted from the RGB space.Figure 1(a) shows the RGB space, where colors are represented by their red, green, and blue components in an orthogonal Cartesian space.The upper-left corner in the image is the origin.The color of each pixel (, ) denoted by (, ) RGB is processed to separate its red, green, and blue components ((, ) R , (, ) G , (, ) B ).The FCE extracts a cluster of colors based on a defined color pattern (seed) denoted by seed RGB , seed RGB and is either directly defined by its RGB components (seed R , seed G , seed B ) or determined by a pixel in the image.The color components' differences between a pixel (, ) RGB and seed RGB are calculated as follows: and  represent the size of an array which holds an image.The corresponding fuzzy rules can be obtained as follows: Both the rules indicate that (, ) belongs to the object to be extracted, if the Euclidean distances between (, ) RGB and seed RGB along the three axes in RGB coordinate system are small enough; otherwise, (, ) does not belong to the object.
Figure 2(a) shows the membership functions ( N (),  Z (),  P ()) for the input fuzzy variables (Negative, Zero, Positive) defined by and Figure 2(b) shows the membership functions ( M (),  U ()) for the output fuzzy variables (Matched, Unmatched) defined by Figure 2(b) shows the produced areas in the output domain, while   and   cut  M () and  U ().A crisp output value, Δ F , is calculated by the centroid defuzzification method: where  out () represents the envelope function of the areas cut by   and   in fuzzy output domain.If Δ F < , (, ) is extracted, (, ) is not extracted, where  is a threshold.The FCE can be understood as a mapping operator between Euclidean distances {dif(, ) R , dif(, ) G , dif(, ) B } in the RGB space and a difference Δ F in the intensity space under a fuzzy metric.The example in Figure 3 shows how to create some subimages with interesting colors from the source image  FLOWER by the FCE.First, we define three seeds: seed RED = (255, 0, 0), seed GREEN = (0, 255, 0), and seed BLUE = (0, 0, 255) to create three subimages { RED ,  GREEN ,  BLUE } that hold the words "red," "green," and "blue" from the source image  FLOWER , as shown in Figure 3(b).Also, we select a seed (0, 0) RGB = (6, 6, 8) at the origin to remove the background from the source image  FLOWER as shown in Figure 3(c).Figure 3(d) shows  PETAL =  FLOWER −( RED ∪ GREEN ∪ BLUE ∪  BACKGROUND ) that holds the color components of the flower petal.

Creating Subimages Using FCE
This section discusses the CreateSubImage algorithm for creating a set of subimages from a color image using the FCE in the RGB space.A critical step in this algorithm is to select seeds for the FCE.For this purpose, we evenly divide the RGB space into the eight subspaces {] black, ] red, ] green, ] blue, ] yellow, ] pink, ] cyan, ] white}, which are defined by the well-defined colors: {black, red, green, blue, yellow, pink, cyan, white}, at their corresponding vertices of the RGB space, as shown in Figure 1(a).Table 1 lists their definitions; for example, ] black is defined by a cubic [0, 127] × [0, 127] × [0, 127] on the red-axis, green-axis, and blue-axis.
We calculate the distances of each pixel's RGB components to the eight well-defined colors by where  and  are the index of a pixel in the image, (, ) RGB is the color components of the pixel located at (, ), V  RGB is the color components of vertex  in the RGB space, and   (, ) is the distance between the pixel's color and vertex .
We developed two procedures, IniSeedList and Update-SeedList, to initialize a seed list and to update it.For an input image, IniSeedList classifies each pixel into one of the eight subspaces according to its minimum distance to vertex  that is a well-defined color in the RGB space.In each subspace, IniSeedList selects such a pixel as a seed candidate, seed cand () , that has the shortest distance to its vertex .Then, IniSeedList inserts seed cand () into the seed list in order of the number of pixels classified in the subspaces and returns it.UpdateSeedList checks if seed cand () is popped out or is extracted by the FCE.If so, UpdateSeedList updates seed cand () in its corresponding subspace.The pseudocode of the IniSeedList procedures is shown in Pseudocode 1 and the pseudocode of the Update-SeedList is shown in Pseudocode 2. Table 2 shows an example of a seed list generated by IniSeedList from the color image,  BIRD .Columns 1 and 2 in Table 2 indicate the defined subspaces and the number of the classified pixels in the subspaces, column 3 shows the order of seed cand () in the list, and columns 4 and 5 list the RGB color components and locations of seed cand () in  IMG , respectively.The top seed in Table 2 is seed cand (1) = (244, 251, 243) with location (269, 271) from the subspace ] white holding 100,369 pixels.After the algorithm pops out this seed for creating  M () , UpdateSeedList produces a new seed candidate in the subspace ] white from  U (1) and inserts it on the top of the list.The new seed cand (1) = (240, 251, 247) is located at (283, 267) in  U (1) and is also from the subspace ] white with 105,045 pixels.Please note that the input image to IniSeedList is  IMG =  BIRD −  CENTER , instead of  BIRD itself. CENTER is an image extracted by seed CENTER = (127, 127, 127) that is the center of the RGB space.It is difficult to determine in which subspace the center color is classified, so any seed close to the center may not be able to differentiate colors very well, as shown in Figure 4(a).In order to guarantee to get a "good" seed, our approach does not select any seed falling in  CENTER .It is also possible that there is no pixel in a subspace; that is, the seed candidates are less than eight.For example, the seed candidates for  BIRD are not available in the subspaces ] blue, ] yellow, ] green, and ] pink.
The pseudocode of the CreateSubImage procedures is listed as shown in Pseudocode 3.
CreateSubImage creates a set of subimages from a color source image.CreateSubImage invokes mainly three procedures: IniSeedList, UpdateSeedList, and FuzExtractor.The pseudocode of the FuzExtractor is shown in Pseudocode 4.
CreateSubImage has two arguments:  SRC and center, where  SRC is a source image and center is a Boolean variable.If center = true,  CENTER is created by seed CENTER , otherwise  CENTER is assigned by the empty image ø.FuzExtractor is mainly implemented based on (3) and extracts a subimage  TEMP .The FuzExtractor procedure starts with seed RGB to generate  MATCH with region growing.Because seed RGB belongs to its subimage  MATCH , FuzExtractor pushes its four neighbor pixels into a queue and checks the first-in element, pixel FIRST , in the queue to see if it belongs to the subimage  MATCH .If pixel FIRST belongs to  MATCH , its new four neighbors are pushed into the queue.The FuzExtractor procedure repeats this process until the queue is empty, which indicates that there are no more pixels in the source image  SRC matching the seed RGB color.After FuzExtractor returns  MATCH , CreateSubImage assigns  MATCH to  TEMP as described.
CreateSubImage uses two conditions to check if  TEMP is significant or not.The first condition is to check  TEMP / SRC that it is a ratio of  TEMP area over  SRC one.If this ratio is too small, CreateSubImage ignores  TEMP as  TEMP holds too few pixels extracted from the source image  SRC . TEMP is most likely affected by noise and is not significant for any object in the source image.CreateSubImage also uses the condition, ( TEMP / SRC ) & var ( TEMP ), to check  TEMP , where var ( TEMP ) is a variance defined by var A big  TEMP / SRC indicates a large area size of  TEMP and a big var ( TEMP ) implies that  TEMP contains the color components   () until  U () = ø; that is, each pixel in the image is processed.
There may exist tiny holes in the subimage  M () .Because these tiny fragments may be insignificant for object pattern

Discussions and Conclusions
In this section, we present our selection of the membership functions' parameters of the FCE defined in (4)-( 7) and settings of the CreateSubImage and MergeSubImage procedures used for our tests.We empirically selected the parameters of the membership functions as  1 = 60,   ) .We select the parameters in CreateSubImage and MergeSubImage as  = 0.01,  1 = 0.25,  2 = 500, and  = 0.6.The function of  is to remove insignificant subimages with a "too small" area that may be caused by noise.Both parameters  1 and  2 are defined to remove another type of insignificant subimages, for example,  TEMP shown in Figure 4(b).This subimage is not extracted very well due to a "bad seed," seed RGB = (64, 28, 28).Such a subimage is featured by its large area size and a large variance of its color components.The large variance indicates that the subimage holds a wide range of color distribution in the RGB space, so this subimage should be removed. is the threshold to check the overlap area over the subimage with a smaller area.In this paper,  = 0.6 herein means that the overlap area is 60% of the subimage with the smaller area.The bigger  is, the more similar the color components of the two subimages are.Changing  affects the number of partitioned objects generated by MergeSubImage.As an extreme example,  = 1.0 means that only if the larger subimage fully overlaps the smaller subimage, the two subimages will be merged together.
In our implementation, the algorithm invokes the Cre-ateSubImage procedure twice.For the first invocation, an input image,  IMG , to IniSeedList and UpdateSeedList, is  SRC −  CENTER , where  CENTER is extracted by seed CENTER = (127, 127, 127).Because the first call is to process a source image  SRC , the algorithm does not select a seed close to the RGB space center.As discussed in Section 4, the seed seed CENTER cannot differentiate color components which are far from the well-defined eight colors.For example, the extracted image  CENTER shown in Figure 4(a) holds many color components of six corn cobs.Because the algorithm does not select any seed from  CENTER , it is very possible that some of the pixels in the source image remain unclassified.In       role in achieving a good segmentation performance.Another issue we will address in our study is how to automatically determine the parameters of fuzzy sets based on an input color image.

Figure 1 :
Figure 1: Color distribution in the RGB and HSV color spaces.
Space.YUV is the color space which is used in an analog color TV system, which translates color image components obtained in the RGB space into the luminance Y and the colors B -Y (U) and R -Y (V) by Y = 0.299R + 0.587G + 0.114B, U = (−0.1687R− 0.3313G + 0.500B) + 128, V = (0.500R − 0.4187G − 0.0813B) + 128.
Membership functions for defuzzification

Figure 2 :
Figure 2: Definition of membership functions for the FCE.

Figure 3 :
Figure 3: Extraction of colors in  FLOWER image by FCE.

Figure 5 (
b) shows the segmented image  SEG by a union operation of  OBJ (a)  LAKE (b)  SEG (c)   SEG (d)   SEG (e)  BENCHMARK (f)  Canny (g)  Log

Figure 8 :
Figure 8: Source image,  LAKE ; segmented images in the RGB, HSV, and YUV space,  SEG ,   SEG , and   SEG ; manually made benchmark,  BECHMARK ; segmented images with Canny edge detection and Log edge detection,  Canny ,  Log .

Figure 9 :
Figure 9: Source image,  MAN ; segmented images in the RGB, HSV, and YUV space,  SEG ,   SEG , and   SEG ; manually made benchmark,  BECHMARK ; segmented images with Canny edge detection and Log edge detection,  Canny ,  Log .

Figure 10 :
Figure 10: Source image  BABOON and its segmented image  SEG .

Figure 11 :
Figure 11: Source image,  HORSE ; segmented images in the RGB, HSV, and YUV space,  SEG ,   SEG , and   SEG ; manually made benchmark,  BECHMARK ; segmented images with Canny edge detection and Log edge detection,  Canny ,  Log .

Figure 12 :
Figure 12: Source image,  VEGGI ; segmented images in the RGB, HSV, and YUV space  SEG ,   SEG , and   SEG ; manually made benchmark,  BECHMARK ; segmented images with Canny edge detection and Log edge detection,  Canny ,  Log .

Table 1 :
Definitions of the eight subspaces shown in Figure1(a).Find seed cand()in each subspace; // with the shortest distance to its vertex .Psh seed cand()into SeedList; Sort seed cand(); // according to pixel number classified in the subspaces.

Table 2 :
The seed list generated from the color image,  BIRD .
Such a subimage is also considered as insignificant.If  TEMP is significant, CreateSubImage assigns  TEMP to a ℎ subimage  M ( +1 ) and creates an ℎ  U ( +1 ) , where  M ( +1 ) holds the color components matching the color of seed RGB and  U ( +1 ) holds unextracted pixels in the source image.CreateSubImage iteratively processes  U