Exploiting Syntactic and Semantic Information for Textual Similarity Estimation

The textual similarity task, which measures the similarity between two text pieces, has recently received much attention in the natural language processing (NLP) domain. However, due to the vagueness and diversity of language expression, only considering semantic or syntactic features, respectively, may cause the loss of critical textual knowledge. This paper proposes a new type of structure tree for sentence representation, which exploits both syntactic (structural) and semantic information known as the weight vector dependency tree (WVD-tree). WVD-tree comprises structure trees with syntactic information along with word vectors representing semantic information of the sentences. Further, Gaussian attention weight is proposed for better capturing important semantic features of sentences. Meanwhile, we design an enhanced tree kernel to calculate the common parts between two structures for similarity judgment. Finally, WVD-tree is tested on widely used semantic textual similarity tasks. The experimental results prove that WVD-tree can eﬀectively improve the accuracy of sentence similarity judgments.


Introduction
e measurement of the similarity between a pair of sentences is a fundamental and essential task in a wide range of NLP applications, such as question and answer system (Q&A system) [1,2], information retrieval [3], and text classification [4]. Because of the vagueness and diversity of language expression, the measurement faces a certain amount of technical challenges. For instance, a pair of sentences with many same words may represent different meanings, respectively.
Recently, the emergence of word embedding techniques [5], which map a word into a numerical vector, results in many methods achieving success via sentence embeddings in textual similarity tasks [6,7]. For example, Tien et al. [8] encoded many features from various sets of word embeddings into one embedding and secondly learned similarity between sentences via the new embedding. Xing et al. [9] made use of a word embedding method to find high-frequency phenotypes used as input in a sentence embedding method. e core idea behind these works is to identify semantically related terms in both sentences and to bring these similarities together to aggregate an overall similarity. But the disadvantage is that the process of calculating sentence similarity is superficial and does not delve into the relationships between words in sentence. Gradually, works have turned to model with distributed representations and neural network architectures. Yao et al. [10] took advantage of a convolutional neural network (CNN) and word embedding to achieve sentence embeddings. Palangi et al. [3] designed a model utilizing recurrent neural networks (RNN) with long short-term memory (LSTM) cells to perform document retrieval. ese methods of neural network have truly improved performance in the textual similarity task. However, the training processes of these methods are often time-consuming, and the syntactic features of sentences often cannot be fully exploited. Our research follows some typical features with theirs: (1) it also employs word embedding technology to capture semantic information and (2) the method also performs well on the issues related to sentence similarity.
Moreover, some studies have added the attention weight mechanism to further improve performance [11,12]. Since there are many successful applications of attention mechanism in machine translation [13,14], attention mechanism has been widely applied in the natural language processing domain. Chen et al. [15] proposed a new network to pay attention to the production of the hidden state of one sentence with the help of the other sentence's hidden states and attention information. In this paper, we design a novel attention weight that is more sensitive to "distance" to improve the performance of the attention mechanism.
Another line of works that discussed tree kernels is also associated with our work because our model encodes syntactic-semantic information represented by tree structures. Some attempts have proven the advantages of tree structures that capture syntactic features of sentences [16,17]. Li et al. [18] proposed an algorithm based on a syntactic structure for textual similarity, which analyzed the sentence element and then transformed sentence similarity into word similarity through analysis. Gao et al. [19] presented a tree-to-tree method of Chinese machine translation based on subtree alignment. However, these proposed tree structures lacked semantic features consideration. As for the study of the tree kernel, Aiolli et al. [20] described an efficient algorithm for injecting positional information into a tree kernel and presented ways to enlarge its feature space. Rieck et al. [21] proposed an effective approximation technique for parse tree kernels. Nevertheless, few tree kernel methods are applied to textual similarity tasks.
Based on the inheritance of existing tree structure, tree kernel calculation method, and attention mechanism, this paper develops a novel sentence modeling approach that incorporates semantic and syntactic information. To obtain a sentence representation with multiple features, we propose a structure called WVD-tree, which treats the sentence pairs as input objects. WVD-tree integrates the word vector information into the tree structure. Moreover, Gaussian attention weight is proposed to assign a weight value to each word for better highlighting the important semantic features of sentences. To estimate the similarity between structure trees, we design an enhanced tree kernel tailored for WVD-tree. Our method is tested on STS tasks and gets satisfactory performances. Our method's advantages are the following: (1) it can be used as a general architecture because more powerful techniques can replace the integration techniques; (2) it avoids time-consuming training, different from some neural network methods.
In summary, this paper's main contributions are as follows: (1) In this paper, the WVD-tree is proposed to connect semantic and syntactic information, which can be viewed as sentence representation. (2) Considering that the features of different words in a sentence should be treated differently, we design a novel Gaussian attention weight for better capturing semantic information. (3) For similarity judgment, the enhanced tree kernel is proposed based on the traditional tree kernel and calculates the tree structures' common fragments.
In the remainder of this paper, Section 2 describes several main techniques used in the architecture. Section 3 introduces the proposed model. Section 4 shows the experimental results. Section 5 analyzes the experiments. Finally, Section 6 summarizes the conclusions.

Background
is section briefly reviews several main techniques to facilitate understanding of this paper's model, including word embedding, attention mechanism, dependency tree, and tree kernel.

Word
Embedding. At present, continuous vectors are often used to capture the hidden semantic features of sentences. Generally, this technique is known as word embedding [22]. We can map a word into a numerical vector called word vector. If words have a similar meaning, they will be mapped to a similar vector space position. For example, "good" and "great" are close to each other, whereas "good" and "apple" are distant. e above phenomenon means that the word embedding technique can be directly applied to measure the semantic similarity for word pairs or find the word that is most similar to the target word. However, people focus more on studies of the similarity between two sentences or passages. e sentences are embedded via word embedding technique to determine the similarity between texts. Precisely, we can compute the cosine similarity between numerical vectors of words. en we add up the results and take the average. Finally, we get the similarity between the sentences. e process of semantic similarity computation is shown in Figure 1. In the correlation matrix graph, a word pair has a high correlation if the color between them is dark. It means the words are closer to each other. For example, the similarity score between "boy" and "child" is 0.6, while that between "boy" and "hitting" is 0.083. is phenomenon illustrates that "boy" and "child" have a relatively similar vector space position. e final similarity score can be calculated as follows: where s 1 and s 2 are sentence one and sentence two, respectively; w 1i and w 2j are the i th and j th word vectors of s 1 and s 2 , respectively; and l(s 1 ) and l(s 2 ) are lengths of s 1 and s 2 , respectively.

Attention
Mechanism. e aim of introducing the attention mechanisms [11] for the sentence similarity measurement is to pay more attention to some critical words instead of each word being treated equally in the processing of sentence features. So far, much research has proven that the attributes of words affect human reading efficiency [23]. erefore, researchers believe that different words should be assigned different weights. To propose a new attention weight based on traditional attention weight value, we introduce TF-IDF [24] to obtain the original attention weights of words. e original weight value can be calculated as follows: where tf i,j means the number of occurrences of the word i in document j; df i means the number of documents containing the word i; N means the total number of documents in a corpus.

Dependency
Tree. e foundational feature of the dependency tree [25,26] is that we view sentence structure in terms of the dependency relation. A sentence is represented as a tree with words at leaf nodes and their dependencyrelation tags at the nonterminal nodes. Internal nodes are labeled by the nonterminal category, while leaf nodes are labeled by the terminal category. A simple explanation is that a sentence can be represented as a dependency tree. Each leaf node represents a word in a sentence, while each nonterminal node represents a dependency relation between two words. Every node in the dependency tree can be classified as a root node, a branch node, a nonbranch node, or a leaf node. Within a dependency tree, there is only one root node. A branch node links two or more other nodes. A nonbranch node links only one node. A leaf node is a terminal node that does not control other nodes. For example, in the sentence "A boy is hitting a baseball," firstly, Figure 2(a) displays the dependency-relation of a sentence; subsequently, Figure 2(b) shows the traditional dependency tree constructed by the Stanford parser tool. Some abbreviations are listed in Table 1.

Tree Kernel.
A representative tree kernel, known as a partial tree kernel (PTK) [27], is used to fully exploit dependency trees. e kernel that can be considered represents trees in terms of their substructures (fragments). Tree kernel aims to measure the number of common substructures between a pair of trees T 1 and T 2 without taking into account the whole fragment space. e tree kernel can effectively and automatically extract meaningful fragments. A general tree kernel function framework has been designed to compute the common fragments as follows: where N T 1 and N T 2 are the sets of nodes in the tree T 1 and tree T 2 ; n 1 and n 2 are two nodes from two trees, respectively. Δ(n 1 , n 2 ) function enriches the diversity of kernel space because it can be designed to achieve different performance. e kernel detects whether a tree subset (common to both trees) belongs to the feature space that PTK intends to produce. For this purpose, the desired substructures need to be described. For syntactic parse trees, each node with its children is related to a grammar production rule. e tree kernel in turn generates a tree subset composed of each node and its child nodes according to grammar production rule. en tree kernel function requires to calculate the similarity between two subsets from different trees. e evaluation of the common substructures rooted in nodes n 1 and n 2 needs the selection of the shared child subsets of the two nodes; for example,

[VP [V DT NP]] and [VP [DT NP NP]] have [VP [NP]] (2 times) and [VP [DT NP]] in common.
e PTK computes common fragments that are shared between the two trees. e considered fragments are called partial trees (PTs) in the calculation of structure tree similarity. PTs refer to a node with its partial children (subtree structure can be incomplete, meaning that partial children are allowed). For example, Figure 3 displays a tree with its 10 PTs. PTs can be generated by the partial production rule of the grammar. Owing to the partial production rule, PTK processes more fragments than other tree kernel methods when calculating similarity. Specifically, some common fragments rooted at the n 1 and n 2 nodes, and the Δ(n 1 , n 2 ) function on the partial tree can be calculated as follows.
If the node labels of n 1 and n 2 are different, then Δ(n 1 , n 2 ) � 0. Else, where μ and λ are two decay factors; c n 1 and c n 2 refer to list of children nodes of n 1 and n 2 ; l m � min length(c n 1 ), length(c n 2 )}; Δ p (·) refers to the number of common subsequences whose length is p; and n 1 � n 2 means that the labels of n 1 and n 2 are the same.

Model Description
e WVD-tree of a sentence is constructed based on knowledge integration of word vector, attention mechanism, and dependency-relations. is type of tree can be viewed as semantic-syntactic sentence representation, which contains information about the weight distribution of words in a sentence. For a sentence pair, they are firstly transformed into two WVD-trees. Subsequently, an enhanced tree kernel (improves based on the existing partial tree kernel) is designed to calculate the similarity between two structure trees. e process of the WVD-tree similarity calculation is shown in Figure 4.

Gaussian Attention Weight.
e traditional attention mechanism calculates the original attention weight value. However, it cannot capture the local structure of texts. is problem is improved with a Gaussian probability. Our method is inspired by the following observation: the neighboring words tend to have a higher semantic contribution to the central word than distant words. However, the traditional attention mechanism is not very sensitive to "distance." is phenomenon means that the same words at different distances are treated almost equally. As shown in Figure 5, there are two "new" words in the sentence "I bought a new book with a new friend," but only the first "new" is meaningful for the current word "book." However, the traditional attention mechanism assigns the same weight to the two "new" words, as shown in Figure 5(a). is paper's idea is that the attention mechanism should be encouraged to provide greater weight to neighboring words around the central word. For example, in Figure 5, the central word is "book" because it has the highest weight value in a sentence. erefore, the original weight value calculated by the traditional attention mechanism should multiply Gaussian distribution according to the adjacent position, as shown in Figure 5(b). e weight distribution of the original attention weight is changed to capture the sentence's semantic information more effectively, as  shown in Figure 5(c). is new attention mechanism is called Gaussian attention weight. Experiments prove that Gaussian attention weight works better than the traditional attention mechanism.
For simplicity, this paper uses the standard normal distribution whose probability density function is G(d) � e −πd 2 , where d is the random variable. d is 0 when the word has the highest weight value. is paper introduces a scalar variable ω to Gaussian probability function to loosen the restriction, calculated as equation (5).
is paper also presents a parameter b to the weight of the central word attending itself, which can be calculated as in equation (6).
us, the corrected Gaussian attention weight value G weight can be calculated as in equation (7): where | · | represents the absolute value; ω > 0 and b ≤ 0 are scalar parameters; TF-IDF can refer to equation (2); softmax(·) is a normalized function.

WVD-Tree.
e new way of sentence representation is similar to the traditional dependency tree in some ways; for example, (1) it also has only one root node; (2) the internal nodes are labeled by some abbreviations, for instance, "DT" and "NN"; and (3) every leaf node includes a word from the sentence. e change is that all leaf nodes in the WVD-tree also include two other factors: one is a word vector and the other is the Gaussian attention weight value. e process of constructing a WVD-tree has several essential steps. Firstly, every word in the sentence is assigned a "part of speech"; for example, the word "boy" is classified as "noun," and the word "hitting" is classified as "verb." Secondly, the word vector of the corresponding word is obtained from the pretrained word embedding; meanwhile, each word is given a Gaussian attention weight to distinguish its importance with others. irdly, it is also essential to analyze the relationships among nodes. For example, in the sentence "A boy is hitting a baseball," "direct object" is the relationship between "hitting" and "baseball"; "determiner" is the relationship between "a" and "baseball." Finally, all relations among nodes are connected to form a complete structure. As such, we can obtain the WVD-tree, as shown in Figure 6. 3.3. Enhanced Tree Kernel. As mentioned before, the tree kernel is used to compute the similarity between structured trees. e newly designed tree kernel is inspired by the traditional partial tree kernel. Like the previous tree kernel, a generic function framework is used (refer to equation (1)). e change is that vec 1 and vec 2 (word vectors of nodes n 1 and n 2 ) and wt 1 and wt 2 (Gaussian attention weights of nodes n 1 and n 2 ) are added to the Δ(n 1 , n 2 ) function of tree kernel. e traditional tree kernel computation method considers the syntactic information of nodes but does not explore semantic information. Unlike the traditional tree e score of similarity  Mathematical Problems in Engineering kernel, we calculate the semantic similarity of leave nodes separately, instead of using the same function for all nodes, just like equation (4). e recursive function Δ ETK (n 1 , n 2 ) of enhanced partial tree kernel is defined as follows: Δ ETK n 1 , n 2 � 0, n 1 or n 2 is not leaf node, and n 1 ≠ n 2 , Att weight × SIM vec 1 , vec 2 , n 1 , n 2 are leaf nodes, where c n 1 , c n 2 l(m), μ, and λ have the same meaning mentioned in equation (4); SIM(·) function is designed to measure the cosine similarity between vectors; n 1 ≠ n 2 means that the labels of n 1 and n 2 are different. For example, a label with a node of "NN" and a label with a "VP" do not match.
It remains to account for the way of calculating Δ p (·) according to the enhanced partial tree kernel. For the sake of understanding, consider c n 1 � s 1 a and c n 2 � s 2 b (a and b are the last children; s 1 and s 2 are subsequences of c n 1 and c n 2 , respectively); Δ p (c n 1 , c n 2 ) can be solved by developing a "recursive" function as follows: where s 1 [1: i] (resp., s 2 [1: j]) is the subsequence of s 1 (resp., s 2 ) ranging from 1 to i (resp., from 1 to j); |s 1 | (resp., |s 2 |) is the length of s 1 (resp., s 2 ). Note that, here, Δ(a, b) is calculated using equation (8), while Δ p−1 (·) is recursively calculated using equation (10), and the recursive process stops when it reaches the leaf node.

Datasets.
Following prior studies, the experiments are also conducted on semantic textual similarity (STS) datasets (2012-2015). ese datasets cover various domains, for example, news, web forums, images, glosses, and Twitter. Statistics of STS datasets are sorted by year as shown in Table 2 (datasets with the same name have different data in different years).

Word Embedding Setting.
is paper compares the most commonly used word embedding techniques, word2vec [21] and glove [27], for choosing high-quality word vectors. Here, WVD-tree uses the simple version of the enhanced tree kernel by setting Att weight � 1, μ � 0.1, and λ � 0.1 in equation (8). Considering that the numerical vector's key parameter is the vector dimension, which is set to 50, 100, 200, and 300, the compared results tested on the answers-forums dataset are shown in Figure 7. It can be found that dimension 300 gains the best performance. Based on this analysis, the number of dimensions in word embeddings is set as 300 throughout the experiment process. Meanwhile, it can be clearly seen that WVDtree word2vec (use Gaussian weight word2vec-embedded dependency tree) performs better than WVD-tree glove (use Gaussian weight glove-embedded dependency tree). us, WVD-tree word2vec (with its vector dimension being 300) is selected as the better method to conduct experiments on all STS datasets (2012-2015).

Impact of μ and λ.
e enhanced tree kernel function is related to two parameters, λ and μ. is section studies the impact of the two parameters on the accuracy of WVD-tree. Here, we use the simple version of enhanced tree kernel by setting Att weight in equation (8). Firstly, we set λ � 0.1 by default while studying the parameter μ. As shown in Figure 8(a), μ � 0.3 gets the best performance. en, we set μ � 0.3 while studying the parameter λ. As shown in Figure 8(b)), λ � 0.2 gets the best performance.

Impact of ω and b.
e enhanced tree kernel function is also related to two other parameters, ω and b. e two parameters are derived from equation (6). ey play an essential role in assigning Gaussian attention weight to each word. Firstly, we set b � −0.1 by default while studying the parameter ω. As shown in Figure 9(a), ω � 0.06 gets the best performance. en we set ω � 0.06 while studying the parameter b. As shown in Figure 9(b) Figure 6: An example of the WVD-tree.  MSRpar  750  Headlines  750  Deft-forum  450  Answers-forums  375  MSRvid  750  OnWN  561  Deft-news  300  Belief  375  SMTeuroparl  459  FNWN  189  Headlines  750  Images  750  OnWN  750  Images  750  Headlines  750  SMTnews  399  OnWN  750  Tweet-   sentence is viewed as a document, and all sentences in the dataset are used to compute the value of IDF.

Comparison with Other Methods.
Comparison results of the proposed method and other state-of-the-art methods for each task are provided in Table 3. It is easy to see that the proposed method obtains better or comparable performance. e methods that utilize word embedding/sentence embedding techniques include glove [28], word2vec [29], RNN [30], LSTM [31], and iRNN [32].
Glove denotes global vectors, which is is a word representation tool based on global word frequency statistics. e glove can map a word into a numerical vector that captures some semantic properties between words.
Word2vec denotes word to vector. Word2vec trains layers of neural networks to map one-hot word vectors to distributed word vectors.
RNN is the classical recurrent neural network that can solve the problem that the training sample is a continuous sequence with different lengths.
iRNN is an independently recurrent neural network that is improved based on RNN. iRNN's neurons in the same layer are independent of each other, and they are connected across layers. Multiple iRNNs can be stacked to construct a network that is deeper than the existing RNNs.
LSTM denotes long short-term memory, which is is a special kind of RNN. For remembering the long text's vital information during the training, the neural network cells will forget some information and remember some information.  Mathematical Problems in Engineering 9 Compared with these methods, our method accomplishes the best results on 13 out of 18 datasets. Even if the proposed method is not the best one on some datasets, it can also compete with the best results, such as 12′OnWn datasets and 15′belief datasets. It is just as that the result of the method on the 12′OnWn datasets is 68.8%, and the result of the best method is 70.2%. Table 3 shows the results of the accuracy of the state-of-the-art methods on the same datasets. e methods that use word embedding techniques include glove and word2vec. e performance of word2vec is better than that of glove. e reason is that word2vec is more suitable for processing sequential data than glove and the most typical sequential data is the sequence of text. Besides, word vectors trained by word2vec have stronger semantic correlation than glove. Because our method integrates word2vec into the dependency tree, meanwhile, Gaussian attention weight is added to highlight important features. e WVD-tree method, which contains semantic and syntactic information, is better than the word2vec method.

Analysis of Results.
e methods that use sentence embedding techniques include RNN, LSTM, and iRNN. Both LSTM and iRNN are improved based on the RNN method. e traditional RNN cannot cope well with the long sequence input due to vanishing gradients or exploding gradients. e vanishing gradients problem of RNN is solved by the LSTM method, but it cannot counter the exploding gradients problem. Another advantage of the LSTM compared to RNN is that it can handle long text. However, it can be seen from Table 3 that the performance of LSTM is not better than that of RNN. e reason is that the STS datasets are full of short sentences, so the advantages of LSTM method cannot be given full play. e performance of iRNN is better than those of RNN and LSTM, because, compared with RNN and LSTM, the gradients of iRNN are relatively stable, and iRNN can be stacked in multiple layers, making it possible to build deeper networks. Our method is generally superior to these neural network methods, mainly because they focus on semantic information training rather than syntactic information, which is different from the WVD-tree method.
Meanwhile, it is necessary to analyze why our method does not perform well on some datasets such as 12′SMTeuroparl and 12′SMTnews. Because of the particularity of the sentences in these datasets, such sentences contain many strings of numbers and special symbols that damage the method's performance, for example, the special string "(a5-0241/2000)" in 12′SMTeuroparl dataset.

Effectiveness of Syntactic Information.
For proving that syntactic information plays a role in the WVD-tree structure, two methods are compared in this section: (1) "Gaussian weight + word2vec," which uses the Gaussian attention weight and word2vec vector together, and (2) "Gaussian weight + word2vec + DT," which uses Gaussian attention weight, word2vec vector, and dependency tree together. Figure 10 displays the results obtained by using four datasets from the STS′15 task. It can find that "Gaussian weight + word2vec + DT" always performs better than the "Gaussian weight + word2vec." e experimental results of the method with "Gaussian weight + word2vec + DT" increased 7.8%, 7.2%, 7.3%, and 2.3%, respectively. e above results essentially prove the effectiveness of syntactic information. It should be noted that the STS′15 task is chosen for the experiment because the sentences of these datasets are clearly expressed and do not contain any special characters, and the improvement of experimental results by syntactic information could be found.

Effectiveness of Gaussian Attention Weight.
For proving that the Gaussian attention weight plays a role in WVDtree structure and works better than traditional attention mechanism, three methods are compared in this section: (1) "word2vec + DT," which uses the word2vec vector and dependency tree together, (2) "traditional weight + word2vec + DT," which uses traditional attention weight, word2vec vector, and dependency tree together, and (3) "Gaussian weight + word2vec + DT," which uses Gaussian attention weight, word2vec vector, and dependency tree together. Figure 11 shows the results obtained by using four datasets from the STS′15 task. It can be found that "Gaussian weight + word2vec + DT" always performs better than "word2vec + DT" and "traditional weight + word2vec + DT." e experimental results of "Gaussian weight + word2vec + DT" increased by 4.5%, 3.8%, 2.7%, and 2.3% compared to "word2vec + DT." e experimental results of "Gaussian weight + word2vec + DT" increased by 1.7%, 2.6%, 2.0%, and 0.8% compared to "traditional weight + word2vec + DT." is phenomenon proves that the Gaussian attention mechanism plays a positive role in performance.

Varying Tree Kernel.
e enhanced tree kernel introduced before (recall Section 3.3) is changed based on the partial tree kernel. However, the commonly used tree kernel includes subtree kernel (STK) [20] and subset tree kernel (SSTK) [27] in addition to partial tree kernel.

Subtree Kernel.
A subtree is characterized by containing all descendant nodes of the target node. e STK is a convolution kernel that computes common fragments shared between two trees. e fragment considered is a subtree, which means a node and its full children.

Subset Tree Kernel.
Since leaf nodes of subset tree can be nonterminal symbols, the subset tree is a more general structure than a subtree. e SSTK is a convolution kernel that computes common fragments shared between two trees. e fragment considered is a subset tree, that is, a node and some of its children (children may be incomplete in depth, but partial generation is not allowed).
Here we also change subtree kernel and subset tree kernel in the same way (add Gaussian attention weights and word vectors to traditional tree kernel) and then compare their performance. e comparison results tested on the STS′12 task are shown in Table 4. Comparison results of the STK and the SSTK have little change in accuracy rate. e reason is that STK and SSTK do not differ much in the depth of splitting tree fragments. e PTK has better performance than other tree kernels. e reason is that the PTK can get a greater number of tree fragments than other tree kernels so that PTK can compare more common details for computing the similarity between two structure trees.

Conclusions
is paper proposed a new method for textual similarity measurement. e proposed model aims to extract syntactic-semantic information of sentences by combining several techniques, including word embedding, dependency tree, and attention weight mechanism. In this model, the WVD-tree is a semantic-embedded dependency tree structure for sentence representation. Meanwhile, the enhanced tree kernel is an algorithm designed for similarity calculation. Experiment results on widely used STS tasks demonstrate that this model can achieve favourable performance. Moreover, we design the Gaussian attention weight for distinguishing the contribution of different words more effectively. Experiment results also show that the Gaussian attention mechanism outperforms the traditional attention mechanism. In general, the major advantages of our method are the following: (1) it can be used as a general architecture because the integrated techniques in our model can be viewed as building blocks, allowing users to replace them with other more useful techniques; (2) different from most of the neural network methods, our model does not require time-consuming training, once word embeddings are available. In the future, there are still many works to study. For example, (1) other state-of-the-art techniques can be integrated to improve our model further; (2) our model can be compared with some neural network methods (e.g., RNN, LSTM, and iRNN) on larger training datasets to see whether our model can still achieve competitive performance.

Data Availability
e data used to support the findings of this study are available from the corresponding author upon request.

Conflicts of Interest
e authors declare that there are no conflicts of interest regarding the publication of this paper.