A Collaborative Deep and Shallow Semisupervised Learning Framework for Mobile App Classification

,


Introduction
e proliferation of mobile devices in recent years has led to a rapid growth of mobile applications (represented as "App" in the rest of the paper), which play a more and more important role in mobile users' daily lives. For example, in 2017, there are about 64 billion and 28 billion App downloads at Apple's App Store [1] and Android's Google Play [2], respectively. With such a huge number of Apps, it is necessary to classify them into predefined semantic categories to support various intelligent services, such as App search, App recommendation, and user profiling [3][4][5]. Take user profiling as an example, if user A often uses "Angry Bird" and user B usually uses "Fruit Ninja," we could know that they both like to play casual games based on an appropriate App classification.
Someone may argue that most App delivery platforms (e.g., App Store and Google Play) have already predefined the taxonomy of Apps and classified each App as one of the categories. However, the App classification provided by App delivery platforms has the following limitations. First, the classification results usually cannot be obtained by third party services. For example, the Android development platform can only access to the App information such as name and version, not including category from delivery platforms. Second, the predefined App taxonomy from delivery platforms is fixed, while users may usually need customized App taxonomy. For example, the App " e Palace Museum" is classified as "Life" in the App Store, which is sometimes too general for understanding its real usage, e.g., some specific applications might prefer to tag it as "Tour." us, it is necessary to design a method that can classify Apps into customized categories based only on the most easily accessed data, i.e., App names. However, it is a challenging task, because the App names are usually too short, and the contained words are too sparse to reflect the relevance between Apps and semantic categories [6]. For example, it is almost impossible to tell the category of App "YouTube" by only looking at its name. Aiming at this challenge, most existing works enrich the semantic information of App names using external knowledge (e.g., Web search engine) by borrowing the idea from short text classification [7][8][9] and then train App classifiers based on supervised machine learning techniques. However, the existing App classification methods based on supervised machine learning techniques still have the limitation that they require a large number of labeled samples to train the classifiers, especially the deep learning techniques. However, it is an expensive task to collect adequate labeled samples in practice, when a customized App taxonomy is required.
Semisupervised learning is one of the prominent ways to address the problem of limited labeled samples by learning from both labeled and unlabeled samples without human intervention [10]. In semisupervised learning, an effective paradigm is referred to as disagreement-based semisupervised learning [11], where multiple base learners are trained and they learn from each other collaboratively to improve the performance by exploiting the disagreement among them. In disagreement-based semisupervised learning, diversity of base learners is the key requirement [11]. Most existing methods ensure the diversity based on feature split (i.e., using different features to train the base learners). However, this scheme heavily relies on the strategy of feature split. When the features are not properly split, the performance would be greatly degraded [11]. In this paper, we exploit totally different machine learning paradigms (i.e., shallow learning and deep learning) to ensure a greater degree of diversity without feature split. Different from the existing disagreement-based semisupervised learning algorithms, we found that the greatest problem of fusing shallow learning and deep learning in a collaborative framework is that they have highly unbalanced and dynamically varied performance. For example, deep learning learners usually have stronger performance when there are adequate labeled samples. is problem would cause the stronger base learner to easily learn noises from the weaker base learner.
To this end, this paper proposes Co-DSL, a collaborative deep and shallow semisupervised learning method for App classification. First, in order to enrich the textual features of an App, we take advantage of a Web search engine to obtain snippets for the App by using its name as the query keyword. Second, we apply Co-DSL, which uses only a few labeled samples and a large number of unlabeled samples, by exploiting the complementary discriminative power of totally different machine learning paradigms (i.e., deep learning and shallow learning) in a cotraining style. Specifically, the main contributions of this paper are summarized as follows.
(1) We propose a method for automatic App classification. First, it uses Web knowledge to enrich the textual features of mobile Apps. Second, it could train the App classifier using a very limited number of labeled samples. Based on the experiment results, by using only two labeled samples from each App category, the proposed method could achieve over 85% classification accuracy. (2) We propose a semisupervised learning framework called Co-DSL, which has the following novelties. First, instead of using feature split, we exploit totally different machine learning paradigms (i.e., shallow learning and deep learning) to train the base classifiers. Second, to address the problem of unbalanced performance of base classifiers, we design a biased sampling strategy to choose the unlabeled samples (Section 3.3.1) and a base classifier accuracy estimation strategy to evaluate the base classifier with few labeled samples (Section 3.3.2) in each round of training. e remainder of this paper is organized as follows. Section 2 reviews related work. Section 3 details the proposed method, including the base classifiers and the semisupervised learning framework. Section 4 reports the experiment results. Section 5 concludes the paper.

App Classification.
A major challenge of automatic App classification is that there are not many effective and explicit features due to the limited information of Apps available for analysis. In many cases, only the names of the Apps are available.
erefore, the existing works try to enrich the information of Apps by using various strategies. e most fundamental strategy is the textual enrichment based on Web knowledge. For example, Ma et al. [12] used search engines to obtain several textual snippets to enrich the textual information of the Apps and extracted word frequency vectors as features to represent the Apps. Li et al. [13] also used search engines to enrich the textual information of the Apps and then extracted various features using vector space model and topic model. Garg et al. [14] crawled App descriptions from Google Play and adopted two topic models to create the feature vectors.
In addition to the textual information, some researchers exploit various types of extra data to enrich the information of Apps. e first type is metadata. For example, Berardi et al. [15] gathered metadata (e.g., file size, user rating, and rating count) from the App delivery platforms and selected effective features to build the App classifier. Chen et al. [16] defined multiple kernels based on various metadata (e.g., developer, update, and reviews) to learn a similarity metric of different Apps. Olabenjo [17] used a crawler to extract various attributes (e.g., is free or not and have in App purchases or not) of the Apps from Google Play to create the App classifier. e second type is contextual data. For example, Zhu et al. [6] found that different types of Apps are usually relevant to real-world contexts (e.g., business Apps are likely used in the work place), which could be leveraged for App classification. Radosavljevic et al. [18] considered both the temporal contexts of the sequences of App installation and the description of Apps and integrated these data into an embedding framework. According to the existing works, no matter what types of extra data are considered, the textual information from Web knowledge is always the most effective for distinguishing Apps.

Text Classification Based on Semisupervised Learning.
Text classification is a well-studied problem [19,20]. However, only a few works consider the problem that the labeled samples are inadequate. e mostly exploited approach for this problem is semisupervised learning, which utilizes a large number of unlabeled samples in addition to the labeled samples to automatically improve the performance. For example, Nigam et al. [21] proposed a semisupervised learning framework for text classification. It uses an iterative EM (expectation maximization) technique to exploit both the labeled and unlabeled samples. Jiang and Li [22] proposed IWNB, a semisupervised learning framework based on Naïve Bayes, for text classification. It uses the classifier trained on labeled samples to label and weight the unlabeled samples and then retrains the classifier on all samples. Johnson and Zhang [23] proposed a semisupervised learning framework with convolutional neural networks (CNNs) for text classification. It learns the embeddings of text regions from unlabeled samples for integration into a supervised CNN.
In semisupervised learning, an effective paradigm, is referred to as disagreement-based semisupervised learning [11], where multiple base learners are trained, and the disagreement among these base learners is exploited. For example, Nigam [24] found that disagreement-based semisupervised learning algorithms (e.g., Co-Training) outperform other semisupervised learning algorithms when a natural split of features exists. Wan [25] used features of the English documents and features of the translated Chinese documents as the two independent views of the Co-Training algorithm to exploit the unlabeled Chinese documents. Shi et al. [26] constructed an ensemble of classifiers to progressively improve the performance of text classification. In disagreement-based semisupervised learning, diversity of base learners is the key requirement for improving the performance [11]. Different from the existing methods that exploit diversity based on feature split, Co-DSL ensures a greater degree of diversity using two totally different learning paradigms (i.e., shallow learning and deep learning). Since we explore semisupervised learning technique for App classification in this paper, we summarize the existing semisupervised learning techniques in Table 1.

Overview.
e proposed method for App classification consists of two main stages. In the first stage, we leverage Web knowledge to enrich the textual information of Apps. To be specific, as shown in Figure 1, given an App a, we firstly submit a's name to a Web search engine (e.g., Google API in our experiments). en, we obtain the top M search snippets and integrate them into a document D a (we call it as the App document of a). A search snippet is an abstract of the Web page that is returned as relevant to the submitted App's name. In order to ensure the quality of search snippets, we only keep the search snippets from some professional mobile App websites (e.g., Google Play, App Store, and Wikipedia). Finally, we preprocess D a by removing all the stop words (e.g., "of" and "the") and normalizing verbs, nouns, and adjectives (e.g., "playing⟶ play," "games ⟶ game," and "better ⟶ good").
In the second stage, we use Co-DSL to train an App classifier. Specifically, as shown in Figure 2, we firstly train two initial base App classifiers using only a few labeled samples based on shallow learning and deep learning techniques, respectively. en, we retrain and upgrade the two base App classifiers by exploiting a large number of unlabeled samples in a cotraining style. For example, assume that we had 10,000 App samples, and only 100 of them were properly labeled; we firstly train two base App classifiers based on the 100 labeled samples. en, the base App classifiers could learn from each other in a cotraining style using the 9,900 unlabeled samples based on Co-DSL.

e Base Classifier of Shallow Learning.
e shallow learning technique trains the initial base classifier based on a variety of textual features extracted from the App documents. e textual features are extracted from two aspects, i.e., the vector space model (VSM) features and the topic model features.
e VSM features are extracted as follows. In the first step, we extract the top N keywords for each App category. Specifically, we firstly build a dictionary W that contains all the unique words in all App documents. Second, we calculate the weight of each word in W for each App category. e weight of word w i for App category c j is calculated based on the idea of TF-IDF [34] as equation (1), where n (i, j) is the count of w i in all App documents with App category c j and C is the set of App categories. ird, we choose N unique words with the highest weights for each App category as keywords: In the second step, we transform each App document D a into a word vector wv (a). e ith element in wv (a) indicates the weight of the ith keyword in D a (1 ≤ i ≤ N × |C|), and it is also calculated based on the idea of TF-IDF as equation (2), where m (i, a) is the count of the ith keyword in D a and D is the set of App documents. Here, wv (a) is regarded as the VSM feature vector of D a : (2) Although the VSM features can capture the relevance between the App document and App category in terms of the word distribution, it does not consider the latent semantic meanings behind the words. For example, "song," "playlist," and "artist" are treated as totally different elements in VSM feature vector, but they implicitly reflect the same latent semantic topic "Music." Previous work also found that the latent semantic topics are useful for short text classification [35]. us, we use LDA (Latent Dirichlet Allocation) topic model [36] to learn the latent semantic topics and extract topic model features. e objective of LDA is to determine the word distribution for each topic and the topic distribution for each document by analyzing a large corpus of documents in an unsupervised way. After the LDA model is learnt, an App document D a could be represented as a distribution of topics Mobile Information Systems tv(a) � 〈p 1 a , p 2 a , . . . , p T a 〉, where p k a is the probability that the kth topic is present in D a . Here, tv (a) is regarded as the topic model feature vector of D a .
For each App document D a , we concatenate wv (a) and tv (a) as the final feature vector. en, we train a base classifier to map the feature vectors to the App categories based on a specific machine learning algorithm (e.g., SVM).

e Base Classifier of Deep Learning.
In this paper, we apply convolutional neural network (CNN) proposed in [37] to train the initial base classifier since it shows a good balance between classification accuracy and training efficiency for the text classification tasks. Note that the training efficiency is especially important in our method since the training of base classifiers would be performed multiple times in Co-DSL. e architecture of the CNN is shown in Figure 3, including an embedding layer, a convolutional layer, a pooling layer, and an output layer. First, the embedding layer initially uses pretrained word vectors [38], and fine-tunes them through training via back propagation. Let the dimension of   e convolutional layer would produce 300 feature maps. ird, the pooling layer applies a max-over-time pooling scheme to process the feature maps (i.e., it only takes the maximum value of a feature map).
us, the pooling result of feature map f is max{f}. By concatenating the pooling results of all filters, we could obtain a feature vector with a uniform length (i.e., 300) for different App documents. Fourth, the feature vector from the pooling layer is passed to a fully connected layer to output the probability distribution over App categories. Dropout scheme is employed on the fully connected layer to prevent overfitting.

e Semisupervised Learning Framework.
When a new taxonomy of Apps is defined, the training samples have to be relabeled manually. It is an extremely expensive task. erefore, we could only obtain a small number of labeled samples and a large number of unlabeled samples in practice.
Co-DSL is designed by extending the classical disagreement-based semisupervised learning algorithm Co-Training [32], which trains two base classifiers based on a feature split. Each base classifier is initialized using a few labeled samples on hand. At each round of training, each base classifier chooses one unlabeled sample per class with the highest classification confidence to add to the labeled sample set. en, each base classifier is retrained from the augmented labeled sample set, and the process is repeated.

e Biased Sampling Strategy.
Co-Training treats each base classifier equally, which means that each base classifier chooses the same number of unlabeled samples to add to the labeled sample set in each round of training. However, we found that the two base classifiers in this paper have highly unbalanced discriminative power. is problem would cause the stronger base classifier to easily learn noises from the weaker base classifier. e worst part is the advantage of the base classifiers is variable with the training iteration, e.g., the base classifier of shallow learning has higher performance when the labeled samples are extremely limited, while the base classifier of deep learning gains its advantage when the labeled sample set becomes larger.
Aiming at this problem, we design a biased sampling strategy for choosing unlabeled samples. e idea is that in each round of training, the number of unlabeled samples chosen by each base classifier is proportional to its performance.
us, a stronger base classifier would choose more samples with higher quality than a weaker base classifier did. Figure 4 shows the pseudo-code of Co-DSL. In each round of training, Co-DSL works in two steps, i.e., retraining and selection. In the retraining step, it applies the shallow and deep learning techniques in Section 3.2 to train two base classifiers based on the current labeled sample set and then evaluates the accuracy of the two base classifiers (lines 3-6). In the selection step, it firstly applies each base classifier to the unlabeled sample set (line 7). Second, each base classifier chooses multiple unlabeled samples per App category with the highest classification confidence to augment the labeled sample set of another base classifier (lines [8][9][10][11]. Here, the number of chosen unlabeled samples is proportional to the accuracy of the base classifier, i.e., min {n SC (j), ceil (δ × P SC )} for base classifier SC and min{n DC (j), ceil (δ × P DC )} for base classifier DC, where δ is a basic number, ceil (x) is a function that returns the smallest integer greater than or equal to x, n SC (j) is the number of unlabeled samples in U 2 that SC classifies as c j , and n DC (j) is the number of unlabeled samples in U 1 that DC classifies as c j .  Mobile Information Systems e ceil ( * ) function is used to ensure that at least one unlabeled sample would be picked for each App category in each iteration.
After the two base classifiers are trained, we fuse them to output the final classification result based on the Stacking framework [39].

e Accuracy Estimation Strategy.
e biased sampling strategy is designed based on the evaluation of the base classifiers (lines 5-6), which is not an easy task here. A classifier is normally evaluated by using a testing set segmented from the original labeled sample set (e.g., by performing cross-validation), but the original labeled samples are too few to support the testing set segmentation in our problem. Aiming at this problem, we design a base classifier accuracy estimation strategy as follows.
First, we construct an undirected graph SG over all the original unlabeled samples. Each vertex in the graph v i � (z i , y i ) corresponds to a sample, where z i is the feature vector of the sample and y i is the predicted label of the sample. In this paper, z i is represented as the VSM feature vector in Section 3.2, and y i is predicted by the base classifier to be evaluated at each round of training. ere is an edge connecting two vertices v i and v j if v i is close to v j in the feature space. In this paper, we employ the k-nearest neighbour criterion to generate the edges (i.e., v i is close to v j if v i is among the knearest neighbours of v j or v j is among the k-nearest neighbours of v i ). Here, we use the cosine similarity of z i and z j to measure the similarity of v i and v j , which is also associated as the weight w ij if there is an edge between v i and v j .
Second, we exploit the structure of SG to estimate the accuracy of SC and DC at each round of training. Our evaluation strategy is designed based on a basic assumption that a correctly labeled sample should possess the same label to most of its neighbours (i.e., those having an edge with it in SG), which coincides with the manifold assumption that samples with high similarity in the input space would also have high similarity in the output space [40]. At each round of training, we apply the retrained SC to predict the label of each sample in SG, and an edge in SG is defined as a cut edge if the connected two vertices have different predicted labels. en, we estimate the accuracy of SC based on equation (3), where SG.V represents all the vertices in SG, N (v i ) denotes the set of samples that are connected with v i in SG, and I ij is a function that returns 1 if the edge between v i and v j is not a cut edge and returns 0 if the edge is a cut edge. We use the same procedure to estimate the accuracy of DC at each round of training: (3)

Experiment Setup and Datasets.
We collected 4364 Apps from AppBrain. We used part of the App categories in AppBrain to define the App taxonomy in the experiments. e App taxonomy contains 11 App categories, and the details are summarized as follows. 8.5% Apps belong to "Social," 7.3% Apps belong to "Navigation," 8.6% Apps Co-DSL Input: The labeled sample set, L The unlabeled sample set, U The shallow learning technique in Section 3.2.1, Learner 1 The deep learning technique in Section 3.2.2, Learner 2 The basic number of the chosen unlabeled samples in each iteration, δ Output: The two base classifiers, SC and DC 1.
Repeat for T rounds: 3.
SC use Learner 1 to train a base classifier based on L 1 4.
DC use Learner 2 to train a base classifier based on L 2 5.
P SC evaluate the accuracy of SC 6.
P DC evaluate the accuracy of DC 7.
Apply SC to each sample in U 2 and DC to each sample in U 1 8.
for each App category c j do 9.
Pick min {n SC (j), ceil(δ × P SC )} samples that SC mostly confidently classifies as c j , and add it to L 2 10.
Pick min {n DC (j), ceil(δ × P DC )} samples that DC mostly confidently classifies as c j , and add it to L 1 11. end for 12.
if U 1 and U 2 are both empty then 13. Break L, U 1 U Figure 4: e pseudo-code of Co-DSL.
belong to "Music," 10.3% Apps belong to "News," 11.4% Apps belong to "Photograph," 10.8% Apps belong to "Shopping," 9.7% Apps belong to "Communication," 9.3% Apps belong to "Education," 9.0% Apps belong to "Finance," 5.8% Apps belong to "Food," and 9.3% Apps belong to "Health." It can be seen that the distribution is relatively even. e default values of the parameters are set as follows. For the base classifier of shallow learning, we set the number of keywords for each App category N � 1000 and the number of topics T �11. For the base classifier of deep learning, we set the dimension of the word vectors d � 200 and the length of App documents l � 3000. For the semisupervised learning framework, we set the number of nearest neighbours in the k-nearest neighbour criterion k � 5, and the basic number of the chosen unlabeled samples in each iteration δ is tuned in the experiment.

Experiment 1: e Evaluation of the Base Classifiers.
In the first experiment, we evaluate the performance of different base classifiers in a supervised learning manner. We adopt 5fold cross-validation strategy for evaluation. Specifically, the base classifiers are trained on 80% of the labeled samples and tested on the remaining 20% labeled samples. en, the procedure is repeated five times, and the average performance is reported. ree base classifiers of shallow learning (i.e., SVM, Naive Bayes, and C4.5) and two base classifiers of deep learning (i.e., MLP and CNN) are evaluated. e base classifiers of shallow learning uses features discussed in Section 3.2.1, CNN is the base classifier discussed in Section 3.2.2, and MLP is a deep neural network that replaces the convolutional layer and pooling layer of CNN with a multilayer perceptron layer.
e experiment results are shown in Figure 5(a). It can be clearly found that SVM has the best performance among the base classifiers of shallow learning, and CNN has the best performance among the base classifiers of deep learning. us, we use SVM as the base classifier of shallow learning and CNN as the base classifier of deep learning in the following experiments.
In the second experiment, we evaluate the performance of the base classifiers by varying the number of the labeled samples. As shown in Figure 5(b), SC denotes the base classifier of shallow learning, and DC denotes the base classifier of deep learning. First, the performance of both SC and DC improves with increasing the number of labeled samples. Second, when the labeled samples are extremely scarce (e.g., only one or two labeled samples for each App category), DC has a far worse performance than that of SC.
ird, when there are relatively adequate labeled samples, DC starts to outperform SC. It shows that deep learning technique relies more on the number of labeled samples, and it gains advantage when more labeled samples are available.

Experiment 2:
e Evaluation of the Semisupervised Learning Framework. In the first experiment, we evaluate the effect of δ (the basic number of the chosen unlabeled samples in each iteration in Section 3.3.1). e results are shown in Figure 6. First, the performance of both base classifiers generally increases along with the iterations of the semisupervised learning procedure, while DC shows a far more significant increase. It might be because that DC has a lower initial accuracy and deep learning techniques benefit more from the number of labeled samples. Second, a larger δ leads to a faster improvement of performance, especially for DC. However, a larger δ tends to pick more unlabeled samples when the base classifiers are relatively weak, and thus the chosen samples might contain more noise. As a result, it is more difficult to reach the global optimum. In this experiment, the global optimum is reached when δ � 3 for SC and δ � 5 for DC. To this end, we set δ � 5 in the rest of the experiments.
In the second experiment, we evaluate the effectiveness of the base classifier accuracy estimation strategy. We compare the real accuracy and the estimated accuracy by varying the number of the labeled samples. As shown in Figure 7, although the proposed strategy is only a heuristic way to estimate the accuracy of the base classifiers, the experiment results validate its usefulness in capturing the variation trend of the accuracy of the base classifiers. e mean absolute errors between the real accuracy and the estimated accuracy of SC and DC are 0.045 and 0.090, respectively.
In the third experiment, we evaluate the necessity of the biased sampling strategy, where "original sampling strategy" means that Co-DSL in Figure 4 picks δ � 5 unlabeled samples for each App category in each iteration (lines 9 and 10) without considering the unbalanced performance of the two base classifiers. e results are shown in Figure 8. First, "biased sampling strategy" outperforms "original sampling strategy" in almost all the iterations for both SC and DC. It demonstrates the capability of "biased sampling strategy" for choosing unlabeled samples with higher quality. Second, as more iterations are performed, "biased sampling strategy" gains more advantage over "original sampling strategy." It shows that "biased sampling strategy" could effectively reduce the accumulated noises.

Experiment 3: e Comparison with Baselines.
To evaluate the effectiveness and show its competitive performance of Co-DSL, we compare it with the following eight baseline methods, including five supervised learning baseline methods and three semisupervised learning baseline methods. e evaluation procedure is conducted as follows.
e App document samples are divided into a training set and a testing set. We randomly choose two samples from each App category to form the training set, and the rest samples are all put in the testing set. e supervised learning baseline methods are trained using the training set and tested on the testing set. e semisupervised learning baseline methods are trained using both the training set (treated as labeled samples) and the testing set (treated as unlabeled samples), and tested on the testing set. e evaluation procedure is repeated ten times, and the average performance is reported. e supervised learning baseline methods are as follows.
e results are shown in Figure 9(a). e following tendencies could be discerned from the results. First, the baselines using only deep learning techniques (i.e., DC-CNN, DC-LSTM, and CNN-LSTM) have far worse performance than those using only shallow learning techniques (i.e., SC-NB, SC-SVM, LDA-SVM, and SVM-NB), in both supervised learning and semisupervised learning cases. It is an interesting phenomenon. It shows that deep learning techniques are usually weak at learning knowledge from extremely small data (only two labeled samples for each App category are available). is conclusion could also be verified by the result that DC-LSTM has the worst performance since LSTM is the most complex model here, and it requires more labeled samples to train. Second, SVM-NB and LabelProp outperform SC-NB and SC-SVM, while CNN-LSTM outperforms DC-CNN and DC-LSTM. It indicates that the disagreement-based semisupervised learning techniques are effective at exploiting unlabeled samples to improve the classification accuracy.
ird, SC-NB and SC-SVM outperform LDA-SVM. It indicates that the VSM features are more effective at representing App samples than topic model features do. Fourth, Co-DSL has better performance than SVM-NB and CNN-LSTM. It shows that combining shallow learning and deep learning techniques could achieve better performance than combining different shallow learning techniques or combining different deep learning techniques. Since the diversity between base classifiers is the key for the success of disagreement-based semisupervised learning, the result shows that different machine learning paradigms (i.e., shallow learning and deep learning) could provide diversity to a greater extent.
At last, we plot the classification confusion matrix of Co-DSL. e value in the ith row and the jth column is the proportion of App samples with the ith category that are classified as the jth category. As shown in Figure 9(b), it has relatively low classification accuracy on "Social" (it tends to be misclassified as "Education" or "Food"). It might be because that the definition of "Social" is relatively vague.

Conclusions
In this paper, we studied the problem of automatic App classification, which is a challenging task. First, name is the most easily accessed information of an App, but App names are usually too short to represent the semantic meanings. Second, it is difficult to collect adequate labeled samples to train a good classifier when a customized taxonomy of Apps is needed. Aiming at these challenges, we propose a collaborative deep and shallow semisupervised learning framework. It firstly leverages Web knowledge to enrich the textual features of the Apps. en, it trains two base classifiers based on totally different machine learning paradigms (i.e., shallow learning and deep learning) to maximize the model diversity by using only a few labeled samples. Finally, it fuses the two base classifiers with unbalanced performance by exploiting a large number of unlabeled samples. e experiment results show that the proposed method outperforms the existing deep learning methods and semisupervised learning methods.
In the future, we will extend our work from the following directions. First, the method in this paper is designed based on natural language process (NLP) technique, which is computationally expensive. erefore, we will try to make more efficient usage of the Web knowledge by preprocessing it based on the knowledge graph technique. Second, it is necessary to design a more efficient framework that could run on resource limited mobile devices.

Data Availability
e data used to support the findings of this study are available from the authors upon reasonable request. And the authors will publish their dataset in GitHub in the future.

Conflicts of Interest
e authors declare that they have no conflicts of interest.