Deep Convolutional Neural Networks for Hyperspectral Image Classification

,


Introduction
The hyperspectral imagery (HSI) [1] is acquired by remote sensers, which are characterized in hundreds of observation channels with high spectral resolution.Taking advantages of the rich spectral information, numerous traditional classification methods, such as -nearest-neighbors (-nn), minimum distance, and logistic regression [2], have been developed.Recently, some more effective feature extraction methods as well as advanced classifiers were proposed, such as spectral-spatial classification [3] and local Fisher discriminant analysis [4].In the current literatures, support vector machine (SVM) [5,6] has been viewed as an efficient and stable method for hyperspectral classification tasks, especially for the small training sample sizes.SVM seeks to separate two-class data by learning an optimal decision hyperplane which best separates the training samples in a kernelincluded high-dimensional feature space.Some extensions of SVM in hyperspectral image classification were presented to improve the classification performance [3,7,8].
Neural networks (NN), such as multilayer perceptron (MLP) [9] and radial basis function (RBF) [10] neural networks, have already been investigated for classification of remote sensing data.In [11], the authors proposed a semisupervised neural network framework for large-scale HSI classification.Actually, in remote sensing classification tasks, SVM is superior to the traditional NN in terms of classification accuracy as well as computational cost.In [12], a deeper architecture of NN has been considered a powerful model for classification, whose classification performance is competitive to SVM.
Deep learning-based methods achieve promising performance in many fields.In deep learning, the convolutional neural networks (CNNs) [12] play a dominant role for processing visual-related problems.CNNs are biologicallyinspired and multilayer classes of deep learning models that use a single neural network trained end to end from raw image pixel values to classifier outputs.The idea of CNNs was firstly introduced in [13], improved in [14], and refined and simplified in [15,16].With the large-scale sources 2 Journal of Sensors of training data and efficient implementation on GPUs, CNNs have recently outperformed some other conventional methods, even human performance [17], on many visionrelated tasks, including image classification [18,19], object detection [20], scene labeling [21], house number digit classification [22], and face recognition [23].Besides vision tasks, CNNs have been also applied to other areas, such as speech recognition [24,25].The technique has been verified as an effective class of models for understanding visual image content, giving some state-of-the-art results on visual image classification and other visual-related problems.In [26], the authors presented DNN for HSI classification, in which stacked autoencoders (SAEs) were employed to extract discriminative features.
CNNs have been demonstrated to provide even better classification performance than the traditional SVM classifiers [27] and the conventional deep neural networks (DNNs) [18] in visual-related area.However, since CNNs have been only considered on visual-related problems, there are rare literatures on the technique with multiple layers for HSI classification.In this paper, we have found that CNNs can be effectively employed to classify hyperspectral data after building appropriate layer architecture.According to our experiments, we observe that the typical CNNs, such as LeNet-5 [14] with two convolutional layers, are actually not applicable for hyperspectral data.Alternatively, we present a simple but effective CNN architecture containing five layers with weights for supervised HSI classification.Several experiments demonstrate excellent performance of our proposed method compared to the classic SVM and the conventional deep learning architecture.As far as we know, it is the first time to employ the CNN with multiple layers for HSI classification.
The paper is organized as follows.In Section 2, we give a brief introduction to CNNs.In Section 3, the typical CNN architecture and the corresponding training process are presented.In Section 4, we experimentally compare the performance of our method with SVM and some neural networks with different architectures.Finally, we conclude by summarizing our results in Section 5.

CNNs
CNNs represent feed-forward neural networks which consist of various combinations of the convolutional layers, max pooling layers, and fully connected layers and exploit spatially local correlation by enforcing a local connectivity pattern between neurons of adjacent layers.Convolutional layers alternate with max pooling layers mimicking the nature of complex and simple cells in mammalian visual cortex [28].
A CNN consists of one or more pairs of convolution and max pooling layers and finally ends with a fully connected neural networks.A typical convolutional network architecture is shown in Figure 1 [24].
In ordinary deep neural networks, a neuron is connected to all neurons in the next layer.CNNs are different from ordinary NN in that neurons in convolutional layer are only sparsely connected to the neurons in the next layer, based on their relative location.That is to say, in a fully connected

Fully connected layer
Figure 1: A typical CNN architecture consisting of a convolutional layer, a max pooling layer, and a fully connected layer.
DNNs, each hidden activation ℎ  is computed by multiplying the entire input V by weights W in that layer.However, in CNNs, each hidden activation is computed by multiplying a small local input against the weights W. The weights W are then shared across the entire input space, as shown in Figure 1.Neurons that belong to the same layer share the same weights.Weight sharing is a critical principle in CNNs since it helps reduce the total number of trainable parameters and leads to more efficient training and more effective model.A convolutional layer is usually followed by a max pooling layer.Due to the replication of weights in a CNN, a feature may be detected across the input data.If an input image is shifted, the neuron detecting the feature is shifted as much.Pooling is used to make the features invariant from the location, and it summarizes the output of multiple neurons in convolutional layers through a pooling function.Typical pooling function is maximum.A max pooling function basically returns the maximum value from the input.Max pooling partitions the input data into a set of nonoverlapping windows and outputs the maximum value for each subregion and reduces the computational complexity for upper layers and provides a form of translation invariance.To be used for classification, the computation chain of a CNN ends in a fully connected network that integrates information across all locations in all the feature maps of the layer below.
Most of CNNs working in image recognition have the lower layers composed to alternate convolutional and max pooling layers, while the upper layers are fully connected traditional MLP NNs.For example, LeNet-5 is such a CNN architecture presented for handwritten digit recognition [14] firstly and then it is successfully used for solving other visualrelated problems.However, LeNet-5 might not be directly employed for HSI classification, especially for small-size data sets, according to our experiments in Section 4. In this paper, we will explore what is the suitable architecture and strategy for CNN-based HSI classification.

CNN-Based HSI Classification
3.1.Applying CNNs to HSI Classification.The hierarchical architecture of CNNs is gradually proved to be the most efficient and successful way to learn visual representations.The fundamental challenge in such visual tasks is to model the intraclass appearance and shape variation of objects.The hyperspectral data with hundreds of spectral channels can be illustrated as 2D curves (1D array) as shown in Figure 2 (9 classes are selected from the University of Pavia data set).We can see that the curve of each class has its own visual shape which is different from other classes, although it is relatively difficult to distinguish some classes with human eye (e.g., gravel and self-blocking bricks).We know that CNNs can achieve competitive and even better performance than human being in some visual problems, and its capability inspires us to study the possibility of applying CNNs for HSI classification using the spectral signatures.Figure 3, the net contains five layers with weights, including the input layer, the convolutional layer C1, the max pooling layer M2, the full connection layer F3, and the output layer.

Architecture of the
Assuming  represents all the trainable parameters (weight values),  = {  } and  = 1, 2, 3, 4, where   is the parameter set between the ( − 1)th and the th layer.
Classifying a specified HSI pixel requires the corresponding CNN with the aforementioned parameters, where  1 and  5 are the spectral channel size and the number of output classes of the data set, respectively.In our experiments,  1 is better to be ⌈ 1 /9⌉, and  2 =  1 − 1 +1. 3 can be any number between 30 and 40, and  2 = ⌈ 2 / 3 ⌉. 4 is set to be 100.These choices might not be the best but are effective for general HSI data.
In our architecture, layer C1 and M2 can be viewed as a trainable feature extractor to the input HSI data, and layer F3 is a trainable classifier to the feature extractor.The output of subsampling is the actual feature of the original data.In our proposed CNN structure, 20 features can be extracted from each original hyperspectral, and each feature has  3 dimensions.
Our architecture has some similarities to architectures that applied CNN for frequency domain signal in speech recognition [24,25].We think it is caused by the similarity between 1D input of speech spectrum and hyperspectral data.Different from [24,25], our network varies according to the spectral channel size and the number of output classes of input HSI data.

Training Strategies.
Here, we introduce how to learn the parameter space of the proposed CNN classifier.All the trainable parameters in our CNN should be initialized to be a random value between −0.05 and 0.05.The training process contains two steps: forward propagation and back propagation.The forward propagation aims to compute the actual classification result of the input data with current parameters.The back propagation is employed to update the trainable parameters in order to make the discrepancy between the actual classification output and the desired classification output as small as possible.
3.3.1.Forward Propagation.Our ( + 1)-layer CNN network ( = 4 in this work) consists of  1 input units in layer INPUT,  5 output units in layer OUTPUT, and several socalled hidden units in layers C2, M3, and F4.Assuming x  is the input of the th layer and the output of the ( − 1)th layer, then we can compute x +1 as where and W   is a weight matrix of the th layer acting on the input data, and b  is an additive bias vector for the th layer.  (⋅) is the activation function of the th layer.In our designed architecture, we choose the hyperbolic tangent function tanh(u) as the activation function in layer C1 and layer F3.The maximum function max(u) is used in layer M2.Since the proposed CNN classifier is a multiclass classifier, the output of layer F3 is fed to  5 way softmax function which produces a distribution over the  5 class labels, and the softmax regression model is defined as The output vector y = x +1 of the layer OUTPUT denotes the final probability of all the classes in the current iteration.

Back Propagation.
In the back propagation stage, the trainable parameters are updated by using the gradient descent method.It is realized by minimizing a cost function and computing the partial derivative of the cost function with respect to each trainable parameter [29].The loss function used in this work is defined as where  is the number of training samples.Y is the desired output.y ()  is the th value of the actual output y () (see (3)) of the th training sample and is a vector whose size is  5 .In the desired output  () of the th sample, the probability value of the labeled class is 1, and the probability values of other classes are 0. 1{ = Y () } means, if  is equal to the desired label of the th training sample, its value is 1; otherwise, its value is 0. We add a minus sign to the front of () in order to make the computation more convenient.
The derivative of the loss function with respect to u  is where ∘ denotes element-wise multiplication.  (u  ) can be easily represented as Therefore, on each iteration, we would perform the update for adjusting the trainable parameters, where  is the learning factor ( = 0.01 in our implementation), and We know that   contains W  and b  , and where With an increasing number of training iteration, the return of the cost function is smaller, which indicates that the actual output is closer to the desired output.The iteration stops when the discrepancy between them is small enough.We use average sum of squares to represent the discrepancy.Finally, the trained CNN is ready for HSI classification.The summary of the proposed algorithm is shown in Algorithm 1.

Classification.
Since the architecture and all corresponding trainable parameters are specified, we can build the CNN classifier and reload saved parameters for classifying HSI data.The classification process is just like the forward propagation step, in which we can compute the classification result as (3).

Experiments
All the programs are implemented using Python language and Theano [30] library.Theano is a Python library that makes us easily define, optimize, and evaluate mathematical expressions involving multidimensional arrays efficiently and conveniently on GPUs.The results are generated on a PC The Indian Pines data set was gathered by Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) sensor in northwestern Indiana.There are 220 spectral channels in the 0.4 to 2.45 m region of the visible and infrared spectrum with a spatial resolution of 20 m.From the statistical viewpoint, we discard some classes which only have few labeled samples and select 8 classes for which the numbers of training and testing samples are listed in Table 1.The layer parameters of this data set in the proposed CNN classifier are set as follows:  1 = 220,  1 = 24,  2 = 197,  2 = 5,  3 = 40,  4 = 100, and  5 = 8, and the number of total trainable parameters in the data set is 81408.
The second data employed was also collected by the AVIRIS sensor, capturing an area over Salinas Valley, California, with a spatial resolution of 3.7 m.The image comprises 512 × 217 pixels with 220 bands.It mainly contains vegetables, bare soils, and vineyard fields (http://www.ehu.es/ccwintco/index.php/HyperspectralRemote Sensing Scenes).There are also 16 different classes, and the numbers of training and testing samples are listed in Table 2.The layer parameters of this data set in our CNN are set to be  1 = 224,  1 = 24,  2 = 201,  2 = 5,  3 = 40,  4 = 100, and  5 = 16, and the number of total trainable parameters in the data set is 82216.
The University of Pavia data set was collected by the Reflective Optics System Imaging Spectrometer (ROSIS) sensor.The image scene, with a spatial coverage of 610 × 340 pixels covering the city of Pavia, Italy, was collected under the HySens project managed by DLR (the German Aerospace Agency).The data set has 103 spectral bands prior to water  3.

Results and Comparisons
. Table 4 provides the comparison of classification performance between the proposed method and the traditional SVM classifier.SVM with RBF kernel is implemented using the libsvm package (http://www.csie.ntu.edu.tw/∼cjlin/libsvm);cross validation is also employed to determine the related parameters, and all optimal ones are used in following experiments.It is obvious that our proposed method has better performance  (approximate 2% gain) than SVM classifier using all the three data sets.Figures 4, 5, and 6 illustrate the corresponding classification maps obtained with our proposed method and RBF-SVM classifier.Furthermore, compared with RBF-SVM, the proposed CNN classifier has higher classification accuracy not only for the overall data set but also for almost all the specific classes as shown in Figure 7.   and test time.According to our experiments, it takes only 5 minutes to achieve 90% accuracy on MNIST dataset [32] by using Caffe compared to more than 120 minutes by using our implemented framework.To further verify that the proposed classifier is suitable for classifying data sets with limited training samples, we also compare our CNN with RBF-SVM under different training sizes on the University of Pavia data set as shown in Figure 10.It is obvious that our proposed CNN consistently provides higher accuracy than SVM.However, although the conventional deep learning-based method [26] can outperform the SVM classifier, it requires plenty of training samples for constructing autoencoders.
To demonstrate the relationship between classification accuracies and the visual differences of curve shapes (see Figure 2), we present the detailed accuracies of our proposed CNN classifiers for the University of Pavia data set in Table 6.In the table, the cell in the th row, th column means the percentage of the th class samples (according to ground  truth) which is classified to the th class.For example, 87.34% of class Asphalt samples are classified correctly, but 6.25% of class Asphalt samples are wrongly classified to class Bitumen.The percentages on diagonal line are just the classification accuracies of corresponding classes.As for one class, the more unique the corresponding curve shape is, the higher accuracy the proposed CNN classifier can achieve (check the class Shadow and class Sheets in Figure 2 and Table 6).The more similar two curves are, the higher opportunity they are wrongly classified to each other (check the class Gravel and class Bricks in Figure 2 and Table 6).Furthermore, the excellent performance verifies that the proposed CNN classifier has discriminative capability to extract subtle visual features, which is even superior to human vision for classifying complex curve shapes.Finally, we also implement three other types of neural network architectures for the Indian Pines data set using the same training and test samples.The first one is a simple architecture with only two fully connected layers beside the input layer.The second one is LeNet-5 which is a classic CNN architecture with two convolutional layers.The third one is a conventional deep neural networks (DNNs) with 3 hidden fully connected layers (a 220-60-40-20-8 architecture as suggested in [26]).The classification performance is summarized in Table 5.From Table 5, we can see that our CNN classifier achieves the highest accuracy with competitive training and testing computational cost.LeNet-5 and DNNs cost more time to train models due to their complex architecture, but limited training samples restrict their capabilities of classification (only 20% samples selected for testing in [26] compared with 95% in our experiment).Another reason for the difficulty that deeper CNNs and DNNs face to achieve higher accuracies could be that the HSI lacks the type of high frequency signal commonly seen in the computer vision domain (see Figure 2).

Conclusion and Future Work
In this paper, we proposed a novel CNN-based method for HSI classification, inspired by our observation that HSI classification can be implemented via human vision.
Compared with SVM-based classifier and conventional DNN-based classifier, the proposed method could achieve higher accuracy using all the experimental data sets, even with a small number of training samples.Our work is an exploration of using CNNs for HSI classification and has excellent performance.The architecture of our proposed CNN classifier only contains one convolutional layer and one fully connected layer, due to the small number of training samples.In the future, a network architecture called a Siamese Network [33] might be used, which has been proved to be robust in the situation where the number of training samples per category is small.Some techniques, such as Dropout [34], can also be used to alleviate the overfitting problem caused by limited training samples.Furthermore, recent researches in deep learning have indicated that unsupervised learning can be employed to train CNNs, reducing the requirement of labeled samples significantly.Deep learning, especially deep CNNs, should have great potentiality for HSI classification in the future.Moreover, in the current work, we do not consider the spatial correlation and only concentrate on the spectral signatures.We believe that some spatial-spectral techniques also can be applied to further improve the CNN-based classification.At last, we plan to employ efficient deep CNN frameworks, such as Caffe, to improve our computing performance.

Figure 2 :
Figure 2: Spectral signatures of the 9 classes selected from University of Pavia data set with 103 channels/spectral bands.
Proposed CNN Classifier.The CNN varies in how the convolutional and max pooling layers are realized and how the nets are trained.As illustrated in C1: feature maps M2: feature maps Output F3 Convolution Max pooling Full connection Full connection Input Sharing same weights

Figure 3 :
Figure 3: The architecture of the proposed CNN classifier.The input represents a pixel spectral vector, followed by a convolution layer and a max pooling layer in turns to compute a set of 20 feature maps classified with a fully connected network.

Figure 4 :
Figure 4: RGB composition maps resulting from classification for the Indian Pines data set.From left to right: ground truth, RBF-SVM, and the proposed method.

Figure 5 :Figure 6 :
Figure 5: RGB composition maps resulting from classification for the Salinas data set.From left to right: ground truth, RBF-SVM, and the proposed method.

Figure 8
further illustrates the relationship between classification accuracies and the training time (the test time is

Figure 7 :
Figure 7: Classification accuracies of all the classes for experimental data sets.From (a) to (c): Indian Pines, Salinas, and University of Pavia.The class number is corresponding to the first column in Tables 1, 2, and 3.

Figure 9 Figure 8 :
Figure 9 illustrates the relationship between cost value (see (4)) and the training time for the University of Pavia data set.The value of the loss function is reduced with an increasing number of training iteration, which demonstrates

Figure 9 :
Figure 9: Cost value versus the training time for University of Pavia data sets.

Figure 10 :
Figure 10: Classification accuracies versus numbers of training samples (each class) for the University of Pavia data sets.

Table 1 :
Number of training and test samples used in the Indian Pines data set.
4.1.The Data Sets.Three hyperspectral data, including Indian Pines, Salinas, and University of Pavia scenes, are employed to evaluate the effectiveness of the proposed method.For all the data, we randomly select 200 labeled pixels per class for training and all other pixels in the ground truth map for test.Development data are derived from the available training data by further dividing them into training and testing samples for tuning the parameters of the proposed CNN classifier.Furthermore, each pixel is scaled to [−1.0, +1.0] uniformly.

Table 2 :
Number of training and test samples used in the Salinas data set.

Table 3 :
Number of training and test samples used in University of Pavia data set.It has a spectral coverage from 0.43 to 0.86 m and a spatial resolution of 1.3 m.Approximately 42776 labeled pixels with 9 classes are from the ground truth map, and the numbers of training and testing samples are shown in Table

Table 4 :
Comparison of results between the proposed CNN and RBF-SVM using three data sets.

Table 5 :
Results of comparison with different neural networks on the Indian Pines data set.
[31] data can reach over 90%.We must admit that the training process is relatively time-consuming to achieve good performance; however, the proposed CNN classifier shares the same advantages (e.g., fast on testing) of deep learning algorithms (see Table5).Moreover, our implementation of CNN could be improved greatly on efficiency, or we can use other CNN frameworks, such as Caffe[31], to reduce training

Table 6 :
The detailed classification accuracies of all the classes for University of Pavia data set.