Subpixel Mapping Algorithms Based on Block Structural Self-Similarity Learning

Subpixel mapping (SPM) algorithms effectively estimate the spatial distribution of different land cover classes within mixed pixels. This paper proposed a new subpixel mapping method based on image structural self-similarity learning. Image structure selfsimilarity refers to similar structures within the same scale or different scales in image itself or its downsampled image, which widely exists in remote sensing images. Based on the similarity of image block structure, the proposed method estimates higher spatial distribution of coarse-resolution fraction images and realizes subpixel mapping. The experimental results show that the proposed method is more accurate than existing fast subpixel mapping algorithms.


Introduction
Mixed pixels widely exist in remote sensing imageries.Subpixel unmixing, a sort of soft classification technique, solves the problem of mixed pixels.This method obtains the relative abundance (i.e., the proportion or the component) of each land cover class within each pixel and gets fraction images of each class of hyperspectral remote sensing image.While it is the defect of subpixel unmixing that this technique can only obtain the proportion of each land cover class, it is hard to specify spatial distribution of different land cover classes within pixel, which means that many specific spatial details are still missing.To address this issue, subpixel mapping (SPM) proposed firstly by Atkinson et al. [1] estimates the specific spatial distribution of different land cover classes within mixed pixels.Subpixel mapping converts soft classification into hard classification [2] on higher spatial scale.This technique segments mixed pixels into subpixels by appropriate scale in order to predict the class of each subpixel and obtain specific land cover information on higher spatial resolution.
At present, most existing SPM algorithms take spatial correlation hypothesis as theoretical basis, which declares that the closer the subpixels are, the more possibilities they belong to the same class.This theory stands in most cases, by which subpixel mapping can be carried out [3].Similar to pixel swapping, Hopfield neural network (HNN), linear optimization, genetic algorithm, subpixel spatial attraction (SPA), and interpolation-based [4], almost all the existing SRM algorithms adopt spatial correlation hypothesis.But some of them consume too much time because large numbers of iterations are required to meet the satisfactory results, such as genetic algorithms, Hopfield neural network, and Markov random field, while the others need prior spatial information which is hard to be obtained (e.g., traditional BPNN and indicator cokriging).
A series of interpolation-based fast algorithms [4] emerge and show their advantages for they do not need prior spatial structure information on classes and consume far less time without too many parameters and iterations.However, most of them fail to perfectly reflect the characteristics of complex landscapes only relying on spatial correlation hypothesis, mainly because not enough prior spatial structure information is taken into consideration.Such fast SPM methods trend to reproduce the details defectively when dealing with division, tiny point features, and narrow linear, strip features such  as rivers and highways.In view of this phenomenon, existing fast SPM results differ significantly from the actual situation, which limits the accuracy and performance of SPM and imposes restrictions on the further development of fast SPM algorithms.
Image super-resolution reconstruction (SR) is widely applied to image processing field.This technique reconstructs coarse-resolution image to fine-resolution image through a series of signal processing methods.Not only coarseresolution images but also additional information should be involved in the super-resolution reconstruction process in order to make up the lack of detailed information.A superresolution reconstruction algorithm based on multiscale similarity learning proposed by Pan et al. [5] improves the accuracy of SR.Multiscale structure self-similarity refers to similar structures which exist in the same image of same or different scales.Structure self-similarity exists widely in remote sensing images.For super-resolution reconstruction, this algorithm takes advantage of the similarity of image blocks of same scale as well as different scales within the image itself as additional information to learn from.Zhang et al. [6] introduced multiscale self-similarity into subpixel mapping field.This algorithm takes multiscale selfsimilarity redundancy as a new regularization term and improves the accuracy.However, with too complex iterations, this method consumes much time.Another new algorithm, self-similarity pixel swapping (SSPS) proposed by Su [7], combines self-similarity with spatial continuity and does well in terms of accuracy.Inheriting from pixel swapping, this time-consuming algorithm also needs massive iterations.
Inspired by those reasons above, this paper proposes a novel algorithm, which directly applies self-similarity to fraction images.Learning from the corresponding relationship between fraction image blocks with similar structure of different scales by BPNN, this algorithm converts coarse-resolution fraction images into fine-resolution fraction images without too much time-consuming iterative process and estimates the specific spatial distribution on higher resolution scale.Structural self-similarity of fraction image block provides additional spatial information which is easy to obtain, because the prior spatial information comes from coarse-resolution fraction image itself and its downsampled image.

BPNN SPM Algorithm Based on Structure
Self-Similarity Learning 2.1.Spatial Correlation Hypothesis.Most existing subpixel mapping algorithm predicts the specific spatial distribution of each land cover class in mixed pixels according to the theory of spatial correlation hypothesis [8]. Figure 1 shows a simple schematic diagram of the spatial distribution of simulated pixels, which is assumed to contain two kinds of land classes connoted by black and white, respectively.The scale factor is set as 5,  = 5.In a sense, spatial correlation hypothesis decreases the uncertainty of SPM.When facing division, tiny point features, and narrow linear, strip features, this method exposes its drawback.

Self-Similarity Learning of Image.
Glasner et al. [9] proposed that similar areas (i.e., similar block) of same scale and different scales generally exist within the same neighborhood, between different neighborhoods no matter in same image or different images, which is the specific performance of image multiscale structure self-similarity.Structure self-similarity occurs so often in remote sensing imageries.They usually exist widespread in the form of roads, houses, and natural landscape significantly or latently, which provides useful additional information [10] for finer spatial resolution.As a result, image structure self-similarity learning can be applied to SPM as prior spatial information.
The principle of the SR on similar blocks with the same scale is shown in Figure 2(a).In Figure 2(b), fineresolution image is represented by HR, while LR represents the corresponding coarse-resolution image [11].The size of HR image is  2 times the size of the LR image.and LR completely the same as that of Ω 2 HR and Ω 1 HR , Ω 2

LR
provides precise additional information for Ω 1 LR reconstruction in LR.A clear conclusion can be drawn that small size image blocks tend to contain spatial patterns information, which recur themselves in higher resolution image through the relationship between fine-resolution image and coarseresolution image [12].
Therefore, it is the key point of image self-similarity learning algorithm to conduct block processing on image itself as well as its degradation coarse-resolution image and seek the association or corresponding relationship of block and its downsampled block.
Self-similarity has already been applied to SPM by [6].In this paper, self-similarity is used as regularization term to improve accuracy with massive iterations, which consumes too much time and limit SPM efficiency.Besides, [7] combines self-similarity with pixel swapping.This article has the same shortcomings as it contains iterative process and consumes much time.
Here is the specific process of the learning methods to seek this corresponding relationship: select training image set, and then degenerate the training set to get coarseresolution image set and conduct block process.Obtain the feature of the image block to be reconstructed as well as the feature of its downsampled image block and express them in the form of vector.A dictionary entry is composed of two corresponding vectors.The feature of coarse-resolution image block functions as search index and the feature of fine-resolution block works as dictionary entry content to be searched.Once the relationship is obtained, it can be used on image to estimate itself on higher resolution.

SPM Based on BPNN.
SPM based on BPNN should take coarse-resolution fraction image blocks and their downsampled fraction image blocks to set up training sample.Downsample the coarse-resolution fraction images and "super-coarse-resolution fraction images" can be obtained, on which a  ×  local window is used to operate to describe the super-coarse-resolution fraction block.The vector composed of soft attribution values within a  ×  block is taken as input of training sample and neurons of input layer amount to  2 correspondingly.Then the vector which consists of the probability values of this kind of class within block (containing  ×  ×  ×  subpixels) is set as output, so the output layer is composed of  2 •  2 neurons.
Afterwards, the train sample is used to train the BPNN to obtain the link weights between neurons of different network layers.As a result, the nonlinear mapping relationship between input and output is learned.During the progress of network training, link weights will be gradually modified by feedback, which lets the output of the network approaches the expected output.

SPM Method Based on Block Structure Self-Similarity
Learning.Image self-similarity learning can be applied to subpixel mapping, for this algorithm can solve the shortcomings of existing SPM method, such as low efficiency on linear, strip object as well as complex landscapes mapping.
The specific method of subpixel mapping based on selfsimilar learning is as follows.

Downsampled Coarse Fraction
Images.Suppose original coarse-resolution fraction images   ( = 1, 2, . . ., ); K refers to the number of land cover classes.Let zoom factor be .And then conduct downsampling on all the coarse fraction images   to obtain "super-coarse-resolution fraction images,"   according to .The size of   is  2 times the size of   .This algorithm aims to estimate   's spatial distribution on finer spatial resolution.The relationship between   and   should be learned to estimate the spatial distribution on finer spatial scale.In addition,   and   are threshold values.These two threshold values directly influence the accuracy and consuming time of SPM.However, this paper tries to validate proposed method.So these two threshold values are not under discussion.Both values are set as 0.15, since either accuracy or consuming time is taken into consideration.

Train BP Network. The vector is composed of soft attribution values within a 𝐶 × 𝐶 block, 𝑥 𝑘
is taken as input of training sample, and neurons of input layer amount to  2 correspondingly.Then the vector which consists of    is set as output, so the output layer has  2 •  2 neurons.
Then BPNN should be trained to obtain the link weights between neurons of different network layers.As a result, the nonlinear mapping relationship between input and output is learned.During the progress of network training, link weights will be gradually changed by feedback to approach expected output.

Estimate Based on BPNN.
The trained network can be used on coarse-resolution fraction block to estimate its finer resolution faction block.The vector of soft attribution values of each land class of coarse-resolution block to be mapped should be extracted as input, and the vector of estimated soft attribution values of this land cover class within fineresolution block will be obtained as output.

Stitch Fraction
Blocks.Stitch all estimated fineresolution block sequentially to get fine-resolution fraction images.[13] for each subpixel in turn.

Experiments and Analysis
To validate the proposed SPM method, experiments on two images were carried out.Traditional BPNN, interpolationbased fast algorithm with MSIs, and proposed method were tested.To fully assess the method, fine spatial resolution images were degraded to simulate coarse image.PCC (percent correctly classified) and Kappa coefficient were introduced into this paper to evaluate the accuracy.Besides, two indicators PCC  and Kappa  [14] are introduced to evaluate mixed pixels mapping results.The scale factor  had been fully discussed in other papers where it usually varies between 2 and 5.In this paper, the scale factor is set as 3.No more discussion about  was included in this paper, while the influence of the size of blocks towards accuracy was discussed.

Test Image 1.
An image acquired by ROSIS works as the first set of experimental data.There are 4 kinds of land cover classes in this image.Firstly, downsampling is conducted on the original image to obtain fraction images of different classes.The size of original image is 900 × 900 pixels, and the scale factor is set as 3,  = 3, and the block size is set as 2 × 2. So the size of fraction image is 270 × 270 pixels, which means that each pixel of coarse image corresponds to 3 × 3 pixels of original image.The flowchart of SPM based on self-similarity learning is shown in Figure 3.
Furthermore, downsampling is conducted on the fraction images to obtain lower coarse fraction images, whose sizes are 90 × 90.Traditional BP method, interpolation-based fast algorithm with MSIs, and the BSSL method are carried out for SPM experiments.The results are shown in Figures 4(c) and 4(d).

Test Image 2.
Select an aerial image as the experimental data for test 2; this image still contains 4 kinds of land cover classes: road, water, corn, and vegetables.The image size is 540 × 720.Suppose the scale factor as 3,  = 3, and set block size as 2 × 2. After being treated in the same manner, test image is downsampled to obtain "super-coarse-fraction images."Afterwards, interpolation-based fast algorithm with MSIs, traditional BPNN method, and the proposed method are, respectively, conducted on coarse fraction image blocks to test SPM, and the results obtained are shown in Figures 5(b), 5(c), and 5(d).

Influence of Block Size on SPM.
Discussion about the influence of block size on accuracy was conducted on test image 2. Three block sizes were tested on proposed method (scale factor is set as 3).And the results are shown in Tables 1 and 2. As the size increases, the PCC shows significant decline.The bigger the block size is, the more complex structure information it contains, and thus, the harder SPM works.It consumes more time with smaller block size.However, the influence on time is not as obvious and critical as that on accuracy.To pursue better accuracy, sacrifice of such little time is acceptable.As a result, small size block should be utilized for better accuracy when the size of image to be estimated is relatively small.

Experiment Results
Contrast.From Figures 4 and 5, a conclusion can be drawn visually where BSSL method has better spatial distribution.In the mapping results of BPNN and interpolation-based method, the size and shape of land cover like elongated river become deformed seriously, and serious distortion exists in most isolated and scattered points like land cover.There are obvious burrs at the edge of land cover.The results of BSSL overcome the defects of the traditional methods to a large extent.Significant promotion can be found for slender river restoration, and isolated, scattered point feature mapping results have been obviously improved compared with traditional methods.Thus, the SPM that this paper proposed utilizes the structure features of the land cover relatively and remains more in line with the actual situation.
For further comparison of the experimental results, quantitative analysis should be carried out by adopting three accuracy indexes including confusion matrix, PCC (percent correctly classified), and Kappa coefficient to evaluate the accuracy of the results.In addition, two new indicators PCC  and Kappa  are introduced.These two indicators conduct calculations only on mixed pixels, which exclude the impact of nonmixed pixels to evaluate SPM better.Pixels of the result under evaluation and reference image are compared pixel by pixel, and it comes to the final result after statistical calculation.Tables 3 and 4 show the comparison of accuracy evaluation coefficients of the two methods in two sets of test images.As shown above, the PCC index of this method in image 1 has an increase of 2.269% and 5.728% and the kappa coefficient is, respectively, high being 0.0230 and 0.0556, compared with interpolation-based method with MSIs and traditional BPNN method, respectively.PPC  index rises to 3.9% and 8.005%, respectively, and kappa  coefficient increases to 0.0303 and 0.0976.And the PCC index of this method in image 2 has an increase of 3.097% and 7.207% and the kappa coefficient is, respectively, high being 0.0344 and 0.0764, compared with MSI and traditional BPNN.PPC  index rises to 2.929% and 7.031% respectively and kappa  increases to 0.0372 and 0.1096.
In addition, computation time comparison is listed in Table 5.The proposed method consumes moderate computation time with better accuracy compared with fast algorithms.
In general, each accuracy index of BSSL algorithm has been improved in various degrees in comparison with traditional algorithms.Computation time of proposed method is moderate.It can be concluded that the proposed method takes the structural information of the image into account and is superior to traditional fast algorithms on accuracy.

Conclusion
Owing to the shortcoming of existing fast SPM algorithms based on spatial correlation hypothesis that they are not able to reconstruct such specific spatial distribution as narrow linear, strip and division, tiny point land cover features perfectly, the results of existing SPM tend to meet the technical requirements badly.
The SPM based on BSSL that this paper proposed takes land cover structure features into account and improves the accuracy of SPM.As experimental results show, this method lives up to practical distribution of land cover and promotes the effect and accuracy of SPM and consumes moderate computation time, compared with existing fast SPM algorithms.The size of image block is of vital importance in this method.In most cases, 2 × 2 or 3 × 3 small image block provides relatively good SPM result considering both time and accuracy.
The proposed method shows its potential in real-time applications, while the computation time can be further shortened in future research.

Figures 1 (
b) and 1(c) represent two different space distribution conditions and Figure1(b) represents random subpixel spatial distribution, while Figure1(c) is the space distribution with higher spatial correlation.Clear conclusion can be drawn through observation that the distribution of Figure1(c) is relatively reasonable.

Figure 2 :
Figure 2: Schematic diagram of the similarity principle of block diagram of the same and different scale.
Dictionary.Divide each   into  blocks image by image.Each image block contains  ×  pixels.For each "super-coarse-resolution block"      (each containing  ×  ×  ×  pixels), can be found at the same location in the corresponding coarse-resolution fraction image, the size of which is  2 times the size of    .Both    and    are vectors composed of the soft attribute values of all the pixels of this class in corresponding block.And calculate (  ), the average soft attribute value of    .Set dictionary entry, {(   ),    ,    }. (   ) and    function as the index of dictionary entries.(   ) is set as the first index, while    is the second index, and    functions as the contents of the dictionary items., . . ., ), image blocks to be trained, traverse all the dictionary entries in order and calculate , and keep  1 items which makes  <   .Moreover, let    traverse the second index of  1 items and calculate .Keep  2 items which makes  <   , search the corresponding    , and conduct weighted sum on    to get    to estimate the fine-resolution block corresponding to low-resolution block, which matches similar image blocks.

Table 1 :
Different accuracy with different sizes of image block.

Table 2 :
Different computation times with different sizes of image block.

Table 3 :
Accuracy of test image 1.

Table 4 :
Accuracy of test image 2.

Table 5 :
Computation times of different methods.