Medical images play an important role in medical diagnosis and research. In this paper, a transfer learning- and deep learning-based super resolution reconstruction method is introduced. The proposed method contains one bicubic interpolation template layer and two convolutional layers. The bicubic interpolation template layer is prefixed by mathematics deduction, and two convolutional layers learn from training samples. For saving training medical images, a SIFT feature-based transfer learning method is proposed. Not only can medical images be used to train the proposed method, but also other types of images can be added into training dataset selectively. In empirical experiments, results of eight distinctive medical images show improvement of image quality and time reduction. Further, the proposed method also produces slightly sharper edges than other deep learning approaches in less time and it is projected that the hybrid architecture of prefixed template layer and unfixed hidden layers has potentials in other applications.
Medical imaging [
For medical images, many SRR methods were proposed. Those methods fuse some LR images from the same scene to one high-resolution (HR) image. Many corresponding machine learning [
Those two conditions raise the following problems in reality: (1) a set of low-resolution medical images cannot be obtained for potential reasons, such as costs to patients, and (2) to machine learning approach, a big training dataset of LR and HR medical images is a giant cost in commercial project. For solving those problems, three conventional methods have been widely used. Nearest neighbor interpolation, bilinear interpolation, and cubic convolution interpolation are widely used [
In this paper, we adapt deep learning [ Using deep learning to achieve better SRR result than conventional methods Using transfer learning to enlarge training dataset for DCNN Transfer learning can reduce costs of preparing medical images in reality SIFT feature-based transfer learning and DCNN can offer sharper edge Proposing a hybrid DCNN structure which contains a prefixed template layer.
The rest of this paper is organized as follows: Section
There are three conventional super resolution reconstruction methods: nearest neighbor (NN) interpolation [
Three conventional SRR methods (MRI: brain).
Low-resolution image (118
Nearest neighbor interpolation (236
Bilinear interpolation (236
Bicubic interpolation (236
Figure
Advantages and disadvantages of three conventional SRR methods.
Interpolation method | Advantages | Disadvantages |
---|---|---|
Nearest neighbor | Easy to implement | Problem of image aliasing |
Very fast | ||
|
||
Bilinear | Antialiasing | Blur edges |
Considering with 4 nearest pixels | ||
|
||
Bicubic | Antialiasing | Slightly blur edges |
Considering with 16 nearest pixels | Relatively slow |
Plenge et al. [
Many effective SRR methods are proposed, but most of them are based on a group of LR medical images. Furthermore, nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation can achieve SRR result for single medical image task now, but a better SRR method is needed in medical research and clinical diagnosis.
The proposed method includes three distinctive parts of algorithms/techniques: (1) SIFT feature-based transfer learning, (2) image scaling-down algorithm, and (3) deep learning (deep convolutional neural network (DCNN)). Image scaling-down algorithm is a conventional algorithm. Using SIFT feature-based transfer learning and a hybrid DCNN structure is the major contribution in this paper. As Figure
Overview: deep learning- and transfer learning-based SRR for single medical image.
We improve the SRCNN [
Deep convolutional neural network for medical image SRR.
Steps of Figure
Bicubic interpolation methods use extra points to fit the sampling functions, a critical problem is time cost. As Figure
Bicubic interpretation.
The pixel
Then, the bicubic interpretation templates can be deduced as follows:
The
If
Therefore,
Bicubic interpretation template parameters.
Template | Conditions | Discretized parameters |
---|---|---|
|
|
Let |
|
|
Let |
|
|
Let |
|
|
Let |
|
|
Let |
|
|
Let |
|
|
Let |
|
|
Let |
|
|
Let |
|
|
Let |
|
|
Let |
|
|
Let |
|
|
Let |
|
|
Let |
|
|
Let |
|
|
Let |
Table
Bicubic interpretation template solutions.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Therefore, 16 templates in hidden layer 1 can fulfill the bicubic interpretation method as Figure
Hidden layer 1: fast bicubic interpretation.
This step extracts patch from the results of fast bicubic interpolation layer and maps patch into high-dimensional space as vectors. The dimensionality of these vectors equals a set of feature mapping.
Hidden layer 2 contains a group of rectified linear unit (ReLU, max (0,
Plot of ReLU and classic activation function.
This step fulfills nonlinear mapping. The vectors are mapped in another high-dimensional space for patch extraction and representation layer. This mapping is for representing another set of features.
Like layer 2, the operation of the hidden layer 3 is
Here,
This step generates the result of HR image by aggregating the above patch representations.
Here,
In the learning procedure, the mapping function
Many machine learning approaches hypothesize that the training and test dataset are drawn from the same feature space and they are in the same distribution. Once the distribution changes, lots of modules have to be rebuilt. In real-world applications, to collect sufficient training data is expensive or impossible. Therefore, it would be nice if reducing the effort of collecting the training data and the transfer learning would be desirable.
One typical example is Web document classification; this is an instance-based transfer learning example. Once a document in the area of Web document classification is offered with manual labeling, it would be helpful if the classification knowledge could be transferred into new Web pages with that manual labeled document. As Figure
Instance-based transfer learning example: Web document classification.
Inspired by the transfer learning as mentioned above, SRR for medical images can also employ transfer learning methodology. As Figure
Transfer learning: SRR for medical images.
For obtaining sharper outlines, we use scale-invariant feature transform (SIFT) feature as the base which provides the distinctiveness, the robustness, and the generality. SIFT feature descriptor can capture structural properties robustly, and its points dominantly distribute among regions even color and texture change.
To a given image
In scale space, each pixel is compared to its surrounding 8 adjacent points and 18 neighboring points which are corresponding positions of two images adjacent scale of up and down in pyramid. If the pixel value is different to any of the 26 points, then this pixel is the candidate feature point.
It can be sampled in the neighborhood window and centered at the candidate feature points. Then, histogram is used to count the gradient direction of neighborhood pixels. The range of gradient histogram is from 0 to 360 degrees, and each column represents a direction in histogram.
Rotate the axis to the direction of the feature point which can ensure rotation invariance. Then, select an 8
We traverse all SIFT features in the given training image set. For each subregion of candidate images, we calculate the Euclidean distances to all the SIFT features, then sum up all of the distance, and define a mean of the sum as the distance among training images. It can be calculated as
Figure
SIFT feature-based transfer learning example (vessels and light).
Involved tools contain CUDA, Python language, and Open CV. The method selects candidate images from Image-Net dataset [
As Figure
Preparation of training images.
We use peak signal-to-noise ratio (PSNR) for quantitatively image restoration quality. PSNR can be calculated by
For a fair comparison with conventional methods and SRCNN [
How to get PSNR and comparative figures.
Figure
Ground-truth medical image.
Number 1 DR image
Number 2 Ebola virus
Number 3 MRI of knee
Number 4 CT of liver
Number 5 PET of brain
Number 6 Mammography
Number 7 Cardiac angiography of the heart
Number 8 Angiography
Public medical images for comparison.
Number | File name | Description | Provider | Download weblink |
---|---|---|---|---|
(1) | image008.png | Size: 1500 |
Lappeenranta University of |
|
(2) | 7058_lores.jpg | Size: 700 |
Centers for Disease Control and |
|
(3) | MRIofKnee.jpg | Size: 693 |
National Institute of Health |
|
(4) | CaseKS11-CT-liverSOL-3.JPG | Size: 1114 |
Department of |
|
(5) | 300px-PET-image.jpg | Size: 300 |
|
|
(6) | ht_141204_senoclaire_3d_mammography_800 × 600.jpg | Size: 800 |
|
|
(7) | heart-385 × 330.jpg | Size: 385 |
|
|
(8) | panoramic-cnv-octa.jpg | Size: 1360 |
|
|
Diabetic retinopathy image.
Nearest neighbor
Bilinear
Bicubic
The proposed method
Ebola virus.
Nearest neighbor
Bilinear
Bicubic
The proposed method
MRI knee.
Nearest neighbor
Bilinear
Bicubic
The proposed method
CT liver.
Nearest neighbor
Bilinear
Bicubic
The proposed method
PET brain.
Nearest neighbor
Bilinear
Bicubic
The proposed method
Mammography.
Nearest neighbor
Bilinear
Bicubic
The proposed method
Cardiac angiography of the heart.
Nearest neighbor
Bilinear
Bicubic
The proposed method
Angiography.
Nearest neighbor
Bilinear
Bicubic
The proposed method
As an overview of Table
The results of PSNR (dB).
Number and image name | Bicubic | The proposed method | Bilinear | NN | SRCNN |
---|---|---|---|---|---|
Number 1 DR | 46.35 | 46.74 | 45.77 | 44.35 |
|
Number 2 Ebola | 25.61 |
|
24.31 | 22.79 | 27.25 |
Number 3 MRI knee | 36.67 | 37.74 | 35.33 | 33.04 |
|
Number 4 CT liver | 39.27 |
|
36.96 | 32.49 | 27.74 |
Number 5 PET brain | 37.71 | 37.21 | 34.61 | 30.11 |
|
Number 6 Mammography | 30.64 |
|
30.13 | 29.49 | 31.60 |
Number 7 Cardiac angiography | 36.28 |
|
35.56 | 34.47 | 37.19 |
Number 8 Angiography | 25.96 |
|
24.98 | 24.13 | 27.21 |
In Figure
In Figure
SRR result of MRI image (knee) is shown in Figure
Figure
Figures
Figure
Comparison of different regions (PSNR).
Plainly region
Corners and edges region
Plainly region
Corners and edges region
Comparison of edge and noncorner region in PSNR index (dB).
Image | Region | The proposed method | SRCNN |
---|---|---|---|
Number 4 MRI knee | Figure |
|
34.12 |
Figure |
37.51 |
|
|
|
|||
Number 7 Cardiac angiography | Figure |
|
36.13 |
Figure |
42.79 |
|
Another comparison is running time. The proposed convolutional hidden layer 1 saves approximately half time costs than classic bicubic interpretation; therefore, it can save much time. As (
Running costs of bicubic interpretation and hidden layer 1 (integer and floating-point arithmetic).
Computational operation | Hidden layer 1 | Bicubic interpretation |
---|---|---|
Integer addition | 15 times | 0 times |
Integer division | 1 time | 0 times |
Integer multiplications | 16 times | 0 times |
Floating-point additions | 4.6 times | 41 times |
Floating-point multiplications | 0 times | 28 times |
Table
Overall comparisons of running time (in milliseconds).
Image | Methods | ||||
---|---|---|---|---|---|
NN | Bicubic | Bilinear | SRCNN | The proposed method | |
Number 1 DR | 8.075 | 14.790 | 14.655 | 63579.328 | 63565.847 |
Number 2 Ebola | 8.000 | 10.518 | 11.613 | 11244.818 | 11234.735 |
Number 3 MRI knee | 7.003 | 12.180 | 12.029 | 19231.110 | 19217.921 |
Number 4 CT liver | 7.362 | 11.270 | 12.214 | 36315.960 | 36304.727 |
Number 5 PET brain | 8.993 | 9.067 | 12.954 | 1986.092 | 1977.153 |
Number 6 Mammography | 6.762 | 12.769 | 11.296 | 17342.371 | 17330.239 |
Number 7 Cardiac angiography | 6.123 | 8.940 | 9.965 | 3336.666 | 3328.433 |
Number 8 Angiography | 8.929 | 13.580 | 13.626 | 66651.142 | 66638.937 |
The SRCNN is a novel super resolution reconstruction method, and we tried to improve three parts:
Prefixed template layer: A prefixed template layer saves costs by using mathematic deduction. On the other hand, training a convolutional layer with given training samples to fulfill bicubic interpretation is feasible. However, to train a convolutional layer requires various training images in pair. Therefore, we suggest to prefix the bicubic interpretation layer by mathematics deduction. Moreover, the proposed fixed templates may help other researchers and engineers to use them in real application easily, and they can deduce and verify those templates by their own. Hybrid DCNN structure: Most researches focus on training deep neural networks (NN), and the whole NN is composed of unfixed parameters. The structure of our method combines fixed and unfixed parameters. Maybe, the combination of fixed and unfixed NN structure has undiscovered potentials in other applications. Reducing costs and enhanced edge: The prefixed template layer can save more time than SRCNN, and the proposed SIFT feature-based transfer learning method guarantees the proposed DCNN can produce enlarged medical images with sharper edges.
To conclude, nearest neighbor method may be suitable for some occasions, but it definitely cannot satisfy SRR needs of medical images, such as microscope, CT, MRI, mammography, cardiac angiography, and angiography. Compared with the other conventional methods, bicubic gives better results than bilinear result; however, bicubic method still yields to the proposed method. The SRCNN is an effective and efficient DCNN architecture, but it lacks a faster convolutional interpretation layer. The bicubic interpretation hidden layer in the proposed method ensures a faster running speed than SRCNN, and the SIFT feature-based transfer learning provides sharper edges and corners region than SRCNN by selectively choosing training samples. Moreover, the bicubic interpretation hidden layer can provide an enlarged image which has continuous first and second derivative. This novel bicubic interpretation hidden layer has the potential to solve other image enhancement problems.
In this paper, a deep learning- and transfer learning-based super resolution reconstruction method has been presented. The proposed method aims to reconstruct a high-resolution image form one single low-resolution image. We propose a fast bicubic interpretation layer and SIFT feature-based transfer learning to speed up DCNN and to obtain sharper outlines; therefore, the proposed method can avoid collecting a great number of various medical images. Empirical experiments show that the proposed method can achieve better performance than other conventional methods. We suggest that this enhancement method is meaningful for clinical diagnosis, medical research, and automatic image analysis.
The authors declare that they have no conflicts of interest.