DiaMole: Mole Detection and Segmentation Software for Mobile Phone Skin Images

Results This study developed mole detection and segmentation software DiaMole using mobile phone images. DiaMole utilized multiple deep learning algorithms for the object detection problem and mole segmentation problem. An object detection algorithm generated a rectangle tightly surrounding a mole in the mobile phone image. Moreover, the segmentation algorithm detected the precise boundary of that mole. Three deep learning algorithms were evaluated for their object detection performance. The popular performance metric mean average precision (mAP) was used to evaluate the algorithms. Among the utilized algorithms, the Faster R-CNN could achieve the best mAP = 0.835, and the integrated algorithm could achieve the mAP = 0.4228. Although the integrated algorithm could not achieve the best mAP, it can avoid the missing of detecting the moles. A popular Unet model was utilized to find the precise mole boundary. Clinical users may annotate the detected moles based on their experiences. Conclusions DiaMole is user-friendly software for researchers focusing on skin lesions. DiaMole may automatically detect and segment the moles from the mobile phone skin images. The users may also annotate each candidate mole according to their own experiences. The automatically calculated mole image masks and the annotations may be saved for further investigations.


Introduction
Skin cancer is a major public health problem, and melanoma accounted for 75% of deaths associated with skin cancers [1]. Screening of all skin cancer types before metastasis may provide the patients with more effective treatment options [2]. Noninvasive dermoscopy is a popular technology to capture high-resolution images of skin lesions [2].
Melanoma detection based on mobile phone images recently caught the dermatologic researchers' attention [3,4]. However, the existing studies did not demonstrate sufficient detection accuracies, and they may even identify melanomas as normal cases with a high likelihood [3]. e MoleMapper project collected the skin moles captured by the iPhone ResearchKit and released the data for the research community [5]. However, this dataset did not provide the detailed masks of the lesion sites and the disease annotations.
Segmentation and classification were the two tasks for the dermoscopy image-based melanoma prediction study [6]. A dermoscopy image usually zoomed in to the skin mole, and segmentation served as the first task to get the skin mole's precise boundary. Zhang et al. coupled a deep fully convolutional network and a shallow network with textons to automatically detect the skin lesions' detailed segmentations [7]. Fully convolutional residual networks also demonstrated promising performances in segmenting the skin lesions on the dermoscopy images [8]. e second task of lesion classification may utilize deep neural networks [8] or regular classifiers like support vector machines (SVMs) [9].
is study hypothesized that locating the lesion site with a rectangle may improve the mole detection using mobile phone images. is was based on the observation that the skin lesion usually occupied the majority area of a dermoscopy image, and usually there was only one lesion per dermoscopy image. e experienced dermatologist may ensure the high quality of a dermoscopy image. However, all these assumptions may not always be correct for a mobile phone image captured by a nonprofessional user. e software DiaMole was proposed to provide both object detection and segmentation functions using popular algorithms. e user may also annotate the moles using their experiences within the software DiaMole.

Summary of the Datasets Used in is Study.
Two datasets were used to evaluate the performances of the proposed software DiaMole.
e MoleMapper project generously authorized us to evaluate DiaMole using their mole images captured by the Apple ResearchKit [5]. e project recruited 2,069 participants, who contributed 1,920 demographic surveys, 3,274 mole measurements, and 2,422 curated mole images. Each image is 320 × 568 pixels in size. But the current version of the dataset did not provide the annotations of the bounding boxes and mole masks. is study annotated the bounding boxes of the MoleMapper images using the open-source software labelImg [10]. Most of the images in this dataset put coins as the size meters of the moles. is study only keeps the images with both moles and coins for further analysis. ere are 2,415 images reserved to train and evaluate our models. We use 70% of the dataset to train our models and 30% to test the trained models. ere are 1,691 and 724 images in the training and testing datasets, respectively.
Another public dataset from the ISIC 2018 challenge was retrieved to train our segmentation model [11,12]. is dataset provides 2,594 images, and each is 224 × 224 pixels in size.
is study was approved by the institutional review board (IRB) of the College of Public Health of the Jilin University. All the recruited local participants signed the informed consent forms. Ten local participants were recruited to collect the mole images captured by mobile phones. ere are five male and five female participants in this local cohort. Seven mole images were captured by iPhone 6S, and the operating system is IOS 12.3.1. Each image is 1080 × 1440 pixels in size. To verify the influence by different mobile operating systems, we chose three local participants to collect the mole images by three different mobile operating systems. In this study, we use iPhone XR to collect the images, and the operating system is IOS 14.2, and each image is 4032 × 3024. We use HLK-AL00, and the operating system is Android 10.1.0, and each image is 3000 × 4000. We use OPPO Find X2, and the operating system is Android 10, and each image is 3000 × 4000. is dataset serves as an independent validation and demonstrative dataset for DiaMole. e images of the local cohort were resized to 320 × 568 in pixels, the same size in the MoleMapper cohort.
Although there were over two thousand images in the public datasets, we were concerned about whether the deep learning models could be well trained using these images. So we only split the dataset into 70% training and 30% test datasets. e training dataset was used to train the model and to tune the hyperparameters. e 30% images in the test dataset were not involved in the model training step. is may cause the model overfitting problem. So both the 30% test dataset and the locally collected dataset were used to independently evaluate the trained model. e conventional training/validation/test splitting strategy will be used if a larger dataset is available.

Summary of the DiaMole Implementation.
is study developed the mole detection and segmentation software DiaMole using the Python programming language version 3.6.2. Two major Python packages TensorFlow version 1.12.0 and OpenCV version 4.1.1.26 were used to support the object detection and lesion segmentation modules. DiaMole was developed and tested on the two mainstream operating systems Windows 7 and Windows 10.
DiaMole has a few prerequisite Python packages, including Numpy and Pillow. To provide an easy installation procedure, we used the software Anaconda Distribution version 4.3.25 to manage the virtual environments. e virtual environment was exported as the configuration file "diamole.yaml." e Anaconda Distribution is an opensource Python system that supports Linux, Windows, and Mac OS X. If the user wants to install the running environment, use the command line "conda env create -f diamole.yaml." en the environment may be started by the command line "activate diamole." e graphical user interface (GUI) of DiaMole may be started in the above environment by "python gui\main.py."

Implementation Details of Object Detection Algorithms.
e first main task of DiaMole was object detection, and three popular object detection algorithms were integrated. e popular object detection algorithm YOLOv3 (You Only Look Once) [13] calculated the bounding box of an object at three scales, as similar to the feature pyramid networks [14]. Each bounding box was evaluated using logistic regression, and the class label of each detected box was predicted by a series of 3 × 3 and 1 × 1 convolutional layer duets. YOLOv3 ran very fast to accurately predict the objects and their class labels in the given image.
Another widely used two-stage object detection algorithm, Faster R-CNN (faster region convolutional neural network, abbreviated as FR-CNN), was also integrated into DiaMole [13]. FR-CNN utilized a region proposal network (RPN) to generate high-quality region proposals. Its object detection performance was better than the previous version, the fast region convolutional neural network [15].
Lin et al. proposed RetinaNet, a simple dense detector [16]. RetinaNet matched the running speed of the previous one-stage object detection networks but achieved a better object detection accuracy [16]. e main contribution of RetinaNet was a new focal loss function that reduced the training bias caused by the dataset imbalance and rapidly focused the model training process on the difficult samples.
e detection results of all the above three object detection algorithms might be integrated using the Non-Maximum Suppression (NMS) strategy to generate the consensus object bounding boxes [17].
e integration procedure was denoted as the algorithm ALL.

Implementation Details of Mole Segmentation.
e dataset MoleMapper did not provide the mole masks and labels, and this study transferred a publicly available segmentation model from the ISIC 2018 challenge [18]. e ISIC 2018 challenge released a number of segmentation source codes, and the team Opsins released their convolutional neural networks (CNNs) with the UNet architecture.
e pretrained VGG-16 model was chosen as the base model in this study [19].
A mole usually occupied the major part of a dermoscopic image, but a mobile phone skin image may cover the large neighboring regions of a few moles. So this study firstly got the bounding boxes of the moles using the above object detection algorithms. en each bounding box was regarded as a dermoscopy-like image and fed to the segmentation model.
is model was implemented using the Python programming language version 3.6.2. e following Python packages were used in the above-mentioned UNet model:

Workflow of DiaMole.
e graphical user interface (GUI) of DiaMole may be loaded by running the command line "python gui/main.py" in the directory of DiaMole, as shown in Figure 1. e default workflow is to detect the moles in a given human skin image.
is image may be captured by a mobile phone. e bounding boxes of the moles are detected using the three popular object detection algorithms, that is, YOLOv3, FR-CNN, and RetinaNet. ese algorithms' results may also be integrated using the algorithm NMS.
In the second step, the UNet model will get the precise boundary of each mole, whose bounding box is cropped by the detection algorithms. Besides, the user can draw the bounding boxes based on their own experiences for the moles which are not detected by the algorithms. All the annotations could be saved to local database.
All the intermediate data are stored in the local database. e user may export the masks and annotations of the detected moles for further analysis. e user can export the masks, bounding boxes, and annotations of the detected moles to a word file.

Data Analysis and Annotation Protocol Using DiaMole.
is section described the data analysis and annotation protocol using the proposed software DiaMole. e experimental data of DiaMole on multiple datasets were described and discussed in the following sections.
Step 1. Run the software DiaMole and its GUI, as shown in Figure 2(a). e left region consists of two buttons and one text box. e user may click the button "Input an Image" to load one image into DiaMole or click the other button "Choose a Folder" to load all the images into DiaMole. DiaMole can automatically load the images in the specified folder if these images are in the formats of JPG, PNG, BMP, and TIFF. e middle region has four buttons, and each is to provide one of the four object detection algorithms, YOLOv3, RetinaNet, FR-CNN, and ALL. e below part is the visualization region, and there are three pages for the raw image (raw), images with bounding boxes (detect), and images for manual curation (draw).
e right region has three buttons, that is, "About," "Export," and "Save." e button "About" gives a popup window with the information about DiaMole. e After that, each bounding box was cropped out and segmented by the UNet. e users may annotate these moles. e annotation information and the mask images of the detected moles will be assembled as the final mole report.
button "Save" is to save the results of detection, segmentation, and annotation to the local temporary database. e button "Export" is to export the information from the database into a word file. e detected moles are enumerated in the text box in the below part so that the user may click to choose each detected mole.
Step 2. Choose an image to analyze, as shown in Figure 2(b). e user may click the button "Input an Image" to load an image. Or the user may load all the images in a specified file folder, by clicking the button "Choose a Folder." After loading all the images files in the supported formats, the user may choose an image file by clicking the file name in the file list box in the left region of DiaMole.
Step 3. Detect and segment the moles in the given image, as shown in Figure 3. e bounding boxes of the moles in the given images may be detected using four algorithms, that is, YOLOv3, RetinaNet, FR-CNN, and ALL. Click the button to choose the object detection algorithm.
ere is only one mole segmentation algorithm UNet, and the mole in each bounding box is automatically segmented. e visualization region under the four buttons has three pages, that is, "raw," "detect," and "draw." e page "raw" gives the original image. e bounding boxes and the masks of the detected moles are illustrated in the page "detect." e detected moles are listed in the below part of the right region. e page "draw" gives the canvas for the user to draw a bounding box to cover a user-detected candidate mole.
Step 4. Define a bounding box for a candidate mole missed by DiaMole, as shown in Figure 4. Firstly, the user may switch to the page "draw" in the middle region of DiaMole, as shown in Figure 4(a). e user may then use the mouse to draw a rectangle to cover a candidate mole in the image, as shown in Figure 4(b). DiaMole asks whether the user wants to save this rectangle as a candidate mole. e user may click "Yes," if satisfied with the drawn bounding box. Otherwise, the user may just click "No" to cancel this draw bounding box. e newly drawn bounding box is shown in the image after the user clicks "Yes," as shown in Figure 4(c). e user can annotate a candidate mole with any text after clicking the mole's name in the right region of DiaMole, as shown in Figure 4(d).
Step 5. Click the button "Save" in the right region of DiaMole to save the current data. e masks and annotations of all the candidate moles will be saved into the local database.
Step 6. Click the button "Export" in the right region of DiaMole to export the current data. e masks and  annotations of all the candidate moles will be exported as an annotation report, and the data may be analyzed by other tools.
Step 7. Click the button "Help" in the left region of DiaMole to get the instruction. e usage instruction of DiaMole will be popped up if you click the "Help" button.

Performance Evaluation of Mole Detection Algorithms.
DiaMole used three popular object detection algorithms to detect the two types of objects in the MoleMapper dataset. e integration algorithm ALL used the NMS strategy to combine the detection results of the above three algorithms. An object detection algorithm's performance is evaluated by mean average precision (mAP). e mAP is widely used in measuring the object detection algorithms [13,15,16]. In the detection of moles and coins, curves can be drawn for every class according to recall and precision. e average precision (AP) is the area under the corresponding curve, and mAP is the average AP of multiple targets. e higher the mAP is, the better the algorithm's performance is.
In this study, we used three object detection algorithms, YOLOv3, RetinaNet, and FR-CNN. We can get the positions of the coins and moles in the images by these algorithms. During the training phase of YOLOv3, we used the momentum to optimize the weights. e learning rate was 0.0001, and the cost function was optimized using 10705 epochs. When training the Faster R-CNN, we used the momentum to optimize the weights, set the learning rate to 0.0001, and used 150000 epochs to optimize the cost function. When training the RetinaNet, we used the momentum to optimize the weights, set the learning rate 0.0005, and used 150000 epochs to optimize the cost function. e four algorithms were evaluated for their detection performances of the two classes "Coin" and "Mole," as shown in Figure 5.
is study randomly split the Mole-Mapper dataset into 70% training dataset and 30% test dataset. Both FR-CNN and RetinaNet performed very well on detecting the objects in the class "Coin" but only achieved 0.644 in AP for detecting moles. YOLOv3 did not perform well on detecting the moles. FR-CNN achieved the best mAP � 0.835. When we used NMS to integrate the above algorithms, the integration algorithm ALL achieved mAP � 0.4228. Although the integration performance was not good, ALL avoided missing the detection of moles by integrating the three object detection algorithms.

Demonstration of the Segmented Moles.
is section demonstrated that object detection was necessary for detecting moles in the mobile phone images, as shown in Figure 6. e segmentation algorithm UNet did not find any moles in the mobile phone image, as shown in Figures 6(a) and 6(b). If the object detection algorithm ALL was used to find the bounding boxes of the candidate moles in the image, the UNet model accurately detected the two moles' precise boundaries from the mobile phone image.

Performance of DiaMole on the Locally Collected Dataset.
e mole images from six other locally recruited participants were collected, and their information was summarized in Figure 7. ese data were used to evaluate how DiaMole performed on the mobile phone images captured in everyday life, as shown in Figure 8. RetinaNet detected a light-colored mole, while the other two algorithms FR-CNN and YOLOv3 missed it, as shown in Figure 8(a). All three algorithms detected the right mole, but FR-CNN and RetinaNet recognized the scar as a mole, as shown in Figure 8 e user may also choose the integrated algorithm ALL or the other two algorithms.

Performance of DiaMole on Mole Images of Different
Qualities. To verify the software's performance on mole images obtained using different mobile operating systems, this study took additional mole images by three different mobile phones for three participants, as shown in Figure 9. e mole images were obtained at different image qualities, and the mobile operating systems were iOS 14.2, Android 10, and Android 10.1.0, respectively. As Figure 9 illustrated, the mole images were taken by different mobile operating systems and had complicated backgrounds. But DiaMole detected all the eye-observable moles. So the image quality showed neglectable impacts on DiaMole's mole detection performance.

Evaluation of Each Algorithm's Running Time.
is study used three object detection algorithms to detect the moles, and then the algorithm UNet was used to get the precise border of each detected mole's bounding box. In order to evaluate how fast each object detection algorithm ran on an image, this study chose the nine images from Figure 9 and ran YOLOv3, RetinaNet, and FR-CNN on each image. e average running time of each algorithm on each mobile operating system was illustrated in Figure 10. So FR-CNN ran the fastest to detect moles on the mobile-taken mole images, and it completed its task no more than 15.95 seconds for the mole images taken by both iOS and Android. YOLOv3 was the slowest mole detection algorithm. e data in the above sections suggested that a combined result of all the three algorithms (the mode "ALL") may be chosen to avoid any missed moles.

Discussion
is study developed easy-to-use integrated software, Dia-Mole, to help dermatological researchers collect and annotate the moles in the mobile phone-captured images.
ree popular object detection algorithms YOLOv3, FR-CNN, and RetinaNet were integrated to find the bounding boxes of candidate moles. eir results may be further integrated using the NMS strategy. e candidate mole in a given bounding box will be automatically segmented using the UNet model. e users may manually draw a bounding box to cover a mole missed by DiaMole and annotate these candidate moles based on their experiences.
All the detected moles and their annotations may be exported for further investigations.

Conclusions
e proposed software DiaMole may help the dermatological researchers investigate the moles in mobile phonecaptured images. Melanoma is a major lethal skin cancer, and its early diagnosis will greatly increase the survival rate of the patients. Melanoma may develop from moles, but the patients may ignore the changes of moles in many cases. is study demonstrated that the previous mole or melanoma detection algorithms were usually trained over the Time (seconds) Figure 10: Running time of the three mole detection algorithms. e horizontal axis gave the algorithm names. e vertical axis gave the running time of each algorithm on each mobile operating system. ree participants were tested, and the running time was averaged over the three images taken by each mobile operating system. dermoscopic images and did not perform well on the mobile phone-captured skin images. DiaMole utilized the object detection algorithms to ensure that the segmentation algorithm was not distracted from the background objects. But the DiaMole's running time is a little long, and this will affect the users' efficiency. In future studies, we plan to test and integrate more lightweight neural network models into DiaMole. e neural network Python libraries with support for parallel computing will be preferred to reduce the detection time.
Data Availability e program and the phone-captured mole images of seven local participants are available at http://www. healthinformaticslab.org/supp/.

Conflicts of Interest
e authors declare that they have no conflicts of interest.