Research Article Design and Implementation of Automatic Selection of the Most Efficient Itemset Algorithm Based on Spark

The combination of Spark distributed platform and High-Utility Itemset Mining can solve the problem of long running time issue of High-Utility Itemset Mining. In the experiment, we conclude that Spark-based parallel D2HUP and EFIM algorithms have a greater improvement in running time efficiency than serial algorithms. The existing research has shown that the EFIM and D2HUP algorithms are the two best algorithms for High-Utility Itemset Mining. This paper generates 118 datasets by generating and collecting the running time of the two algorithms in the real and simulated datasets, taking into account the characteristics of each dataset’s length, sparse degree, and dataset size as characteristics with running time as the prediction target and then establishing a model. The accuracy of the prediction was evaluated through experiments, and a set of rules based on decision trees was generated. According to the rules, the fastest algorithm between EFIM and D2HUP can be predicted very well.


Introduction
Data mining (DM) is a term derived from Knowledge-Discovery in Databases (KDD) [1], which is an important step in KDD. Born at the intersection of computer and statistics, it aims to show the valuable information and concepts hidden in data to users in a more intuitive way through statistical sampling, estimation test, pattern recognition, machine learning, distributed technology, visualization, and other methods and means [2]. e essence of data mining is a kind of deeper mathematical analysis. Its main feature is to model the structured data after cleaning and transforming the huge data in the database and analyze the results to extract the key data, which can help decision-makers make business decisions. e deep essence of data mining is a mathematical analysis, whose feature is that after the cleaning and transformation of the huge data in the database [3], the obtained structured data will be modeled, the obtained results will be analyzed, and the key data will be extracted to help decision-makers make business decisions [4].
Data analysis methods commonly used in data mining include classi cation, regression, clustering, and association rule analysis [5], which mine data from a di erent perspective. e concept of association rule was rst proposed by Agrawal et al. in 1993, which is used to nd meaningful connections hidden in large datasets. e discovered connections can be expressed in the form of association rule or frequent itemset. In the traditional frequent itemset mining [6], the weight of each itemset is the same, and more attention is paid to the degree of association between items and whether itemsets frequently occur, which will result in the loss of itemsets with low support but high utility value [7].
In e cient itemset mining, each item has the utility of quantity and pro t, and the combination of e cient itemsets that meet the user-set threshold is not necessarily frequent [8], so the antimonotone feature in frequent itemset mining is no longer applicable [9]. Mining efficient itemsets is a challenging task. With the increase in data volume, the mining time of an efficient itemset increases exponentially [10]. At present, there are few efficient itemset mining algorithms based on the Spark distributed general framework. e combination of Spark and efficient itemset mining is expected to better solve the problem of time-consuming in the process of efficient itemset mining [11]. e two-phase algorithm proposed in literature [12] is the most classic efficient itemset mining algorithm. e algorithm is divided into two stages. In the first stage, TWU (Transaction Weight Utility) is used to narrow the search space and generate candidate itemsets. In the second stage, the efficient itemset meeting the conditions is selected as the final result. However, the upper bound of the algorithm is not compact enough, and a large number of candidate itemsets will be generated in the first stage. At the same time, in the second stage of screening the final results, the database will be scanned repeatedly, resulting in low efficiency of the algorithm. e D2HUP algorithm [6] retains original transaction information by introducing a more compact upper bound and a linked list structure named CAUL and mining efficient itemsets by recursive search. In addition, EFIM algorithm is one of the best efficient itemset mining algorithms at the present stage, which compresses the search space by merging transaction databases and further improves the efficiency of pruning through a more compact upper bound. In addition, many new algorithms are constantly proposed. So far, the best performing algorithms are EFIM and D2HUP.  [13]. Spark's biggest feature is that it is based on a memory-based parallel computing framework, similar to Hadoop's MapReduce [14], but with advantages that Hadoop does not have. Spark can store the intermediate results of running in memory, reducing disk I/O and thus accelerating computing efficiency. So Spark performs well in data mining and machine learning algorithms that require iteration. In addition, Spark supports Scala, Java, Python, and other language interfaces. Developers can choose a language they are familiar with to implement. Spark also has a wealth of APIs for developers to call, which makes it extremely easy to use [15].

Literature Review of Spark
Spark is an ecosystem that provides solutions such as flow operations, iterative operations, and graph operations, as shown in Figure 1.
Spark SQL. It uses SQL expressions to do data query on Spark to analyze big data query [16] and can also call userdefined functions to combine query results with analysis results to reuse data, thus improving query efficiency.
Sparking Streaming. It is a stream processing system that can perform complex operations such as Map, Reduce, and Join on real-time data and save the results to external file systems. Graph X. It is an API provided by the Spark platform for graph operation. It provides the concept and operation of an elastic distributed attribute graph and uses its framework to realize various operations based on the graph.
MLBase/MLlib. It is a component that can provide related machine algorithms and utilities, including classification algorithms, regression algorithms, clustering algorithms, reduction and low-level optimization programs, as well as testing and data generation. Four boundaries are defined, that is, MLOptimizer, MLI, Machine Learning library (MLlib), and parallel computing framework [17].
Tachyon. is component is a distributed system that can provide high efficiency, high fault tolerance, and high reliability calculation [18].
Spark. It is the core computing framework of the ecosystem, which can be run independently or on a cluster [19]. Cluster operation requires the management system to distribute tasks to each node.
In short, the Spark system is based on RDD, and the Spark framework is a memory-based parallel computing big data platform, which can meet the computing needs of different systems with different products. e three operation modes are different in that they have different resource allocation methods and different task scheduling algorithms to perform computing tasks. But they have the same workflow. Figure 2 is the workflow of Spark.

Spark
Each Spark program has its own Executor process, but each Executor process has multiple threads corresponding to it. is parallel resource allocation and scheduling mode is conducive to resource sharing between different Spark programs and greatly improves execution efficiency. Figure 3 shows that the program uses the action to trigger the job. e job builds the DAG graph based on the dependencies of RDD, the built DAG graph is then resolved and built into different stages by the DAG Scheduler, and the dependencies between stages are calculated. en, the Task Scheduler assigns the Task Set to the work set and divides it into multiple threads for parallel computation, and the results are returned to the Task Scheduler and then to the DAG Scheduler [20]. When the results are completed, they are returned to the driver or saved in the external storage system, and all resources are released.

Main Ideas.
e main design ideas for the parallelization of the D2HUP algorithm based on data partition are as follows. First, read database D and minUtil, divide dataset D into N pieces on average, and distribute N pieces of data to different nodes for calculation. e minimum threshold of each node is minUtil/n. If an itemset is not an efficient itemset on all nodes, its total utility on all nodes must be less than minUtil. erefore, the D2HUP algorithm is used to mine candidate efficient itemsets on different nodes simultaneously, and each efficient itemset appears in at least one node. e second step is to rescan the database, calculate the real utility value of the candidate itemset obtained in each node, eliminate the itemset whose threshold value is less than minUtil, and get the final result.

Algorithm Design.
e following steps are used to implement the D2HUP algorithm based on data partition on Spark: Step 1: Read dataset D into RDD. Each row in the RDD stores a transaction in dataset D.
Step 2: Partition the RDD. e repartition() method is called to group the original RDD, and the different number of groups affects the parallelism of the program.
Step 3: Execute the D2HUP algorithm in parallel. e D2HUP algorithm is executed simultaneously for the grouped data obtained in the second step.
Step 4: Use broadcast() to broadcast the results.
Step 5: Iterate through the database, compare each transaction with the candidate itemset, and use the results reduceByKey(), adding the utilities of the same itemset.
Step 6: Execute filter() to remove all unmet conditions in RDD, and the final RDD is the final result.

e D2HUP Parallel Algorithm Based on Tree Structure
and Spark 3.2.1. e Parallelization Idea of the D2HUP Algorithm. e main design ideas of the D2HUP parallel algorithm are as follows. First, the database D is read, the entire database is compressed in CAUL, and each item satisfying the condition in CAUL is prefixed to conduct efficient parallel itemset mining. Because each prefix item contains a separate extension suffix, each item in the original CAUL can be grouped for parallel mining tasks. Each prefix item in CAUL is assigned to different compute nodes, and different compute nodes simultaneously mine the efficient itemset of different prefix items. After the mining task of each Worker node is completed, the mined high-efficiency itemset will be sent to the master node for collection. Each node is mined independently, so this pattern does not have much communication overhead. e essence of the D2HUP algorithm is a depth-first search tree, as shown in Figure 4. In the process of mining efficient itemsets of different prefix trees, each prefix item is treated as a child node, and the child node and its sibling nodes are independent of each other and do not affect each other. In the mining process, there is no need to exchange information, so the efficient itemset with different prefixes will only appear once on all branches. is is also an important reason for the parallel implementation of the D2HUP algorithm.

D2HUP Algorithm Parallelization Based on Tree
Structure. Spark is a programming model based on Map-Reduce, parallelization of the D2HUP algorithm is to transform CAUL into RDD (Resilient Distributed Dataset), and the associated transforms and actions are called to parallelize efficient itemset mining. Implementing the D2HUP algorithm on Spark has the following steps: Step 1: Read the dataset and build the initial linked list CAUL.
Step 2: Convert CAUL to RDD; because Spark operates on RDD, you need to convert CAUL to RDD, where each row stores a prefix item.
Step 3: According to the RDD in Step 2, divide the data into partitions. e number of partitions affects the degree of parallelism of the task.
Step 4: Execute the D2HUP algorithm in parallel. According to the grouping of Step 3, prefix items in each partition are grouped for efficient itemset mining. e specific steps are the same as the serial D2HUP algorithm. In the process of efficient itemset mining for each group, only the efficient itemset of that group needs to be generated because the prefixes allocated by Scientific Programming different groups are different, and there is no influence between prefixes.
Step 5: Aggregate the results. After the completion of the fourth step, all the efficient itemsets of the prefix branch are obtained for each node. rough the aggregation operation, the efficient itemsets of each node are gathered together to get the final result.

e EFIM Parallel Algorithm Based on Spark.
EFIM algorithm is also a depth-first and efficient itemset mining algorithm. After scanning database D and constructing Primary (α) and Secondary (α) items about root node α, the algorithm recursively calls Search() method for efficient itemset mining. is step has been tested to take most of the program's time, so consider parallelizing the Search() method. In the Search() method called for the first time, each item i ∈ Primary (α) is a prefix. e traversal of item i is essentially a depth-first search tree, so the extensions contained in each prefix are independent. erefore, all items i can be grouped, and multiple compute nodes can simultaneously mine the efficient itemset with item i as the root node. Note that the term i assigned to different nodes is different. After the efficient itemsets of all nodes are mined, the results are summarized to the master node. e result is the final efficient term set.
Similar to the parallelization of the D2HUP algorithm, the EFIM algorithm is also a depth-first search tree [21], as shown in Figure 5. Each child of the root node is different and has been sorted in TWU size. erefore, this child node and other sibling nodes are independent and independent of each other. Each candidate itemset can only appear once in the whole depth-first search tree, so the efficient itemset of the child node is the whole efficient itemset. is is also an important reason why the EFIM algorithm can be parallelized [12]. e parallelization of the EFIM algorithm is mainly concerned with the parallelization of the process of searching the efficient itemset recursively. After the whole database is scanned, the generated Primary (α) is transformed into RDD and grouped, and Action and Transform operators are called to perform parallel utility itemset mining on each subnode. e EFIM algorithm based on Spark mainly includes the following steps: Step 1: Read dataset D.
Step 2: Calculate the local utility of each item, and then build the Primary (α) and Secondary (α). Step 3: Primary (α) is converted to RDD on the first call to the Search() method, and there is a line in RDD for each item i ∈ α.
Step 4: Partition the transformed RDD. e number of partitions affects the degree of parallelism of the algorithm.
Step 5: Parallel EFIM algorithm is executed in each partition. According to the grouping of Step 3, the prefix item i of each partition is different, so the efficient items contained are also different. e specific steps are the same as the serial algorithm of EFIM. After the completion of this step, each group of efficient itemsets prefixed with item i will have unique grouping results of different prefixes, so they will not affect each other.
Step 6: After the result aggregation is completed in Step 4, the high-efficiency items of each group obtained are gathered to the master node and persisted to the hard disk or printed out.

e Result of the Experiment.
e parameters that have a great influence on Spark-based efficient mining algorithm are as follows: (1) minUtil, that is the threshold value given by the user.
Different threshold settings affect the number of efficient itemsets that meet the conditions finally mined. e threshold designed for the experiment in this paper refers to the threshold set for datasets in literature.
(2) e size of the dataset. In this paper, we use the common open dataset of the mining domain of the high efficient term set (Chess, Pumsb, BMS, Connect, Accidents, Kosarak, and Chainstore) and four simulated datasets. e real datasets can be downloaded from the UCI or SPMF. (3) e influence of the Spark parameter.
ere are several parameters that have a great impact on Spark. e total number of Executor processes in a cluster is executors, the number of single Executor processes is executor-cores, and the number of data partitions. e experiments carried out in this section aim at these three parameters and perform performance experiments by setting different thresholds.

e Influence of Partition Parameters on the Algorithm.
To study the effect of partition number on experimental results, set other parameters in Spark to a fixed value, as shown as follows: executors � 4, driver-memory � 100 GB, executor-cores � 10, and executor-memory � 100 GB. e effect of different partition numbers on the running efficiency of the program is shown in Figures 6-8. Figure 8 shows the influence of partition parameters on the running efficiency of the program, where the triangular broken line is the parallel EFIM algorithm, and the dotted broken line is the parallel D2HUP algorithm. As can be seen from the figure, the running efficiency of parallel EFIM and D2HUP programs is gradually accelerated as the number of  Scientific Programming partitions increases. It is worth noting that the threshold value (minUtil) in the experimental process is user-defined by the user, and the itemset larger than the threshold value is regarded as the efficient itemset. e threshold value is set with reference to the dataset in reference [14].

e Influence of Executors Parameters on the Algorithm.
To verify the effect of executors on the running time of the algorithm, set all parameters other than executor to a fixed value: driver-memory � 100 GB, executorcores � 2, executor-memory � 20 GB, and partition � 16. e results are shown as follows.
Tables 1 and 2 show the impact of executors' parameter on the running efficiency of the program on different datasets. It can be found from the above table that, with the increase of executors, the running time of both algorithms decreases to some extent.

3.4.3.
e Influence of Executor-Cores Parameters on the Algorithm. To verify the impact of executor-cores on algorithm runtime, parameters other than executor-cores are set as fixed values: driver-memory � 100 GB, executors � 4, executor-memory � 20 GB, and partition � 16. e results are shown as follows. Tables 3 and 4 show the influence of executor-cores on the operating efficiency of programs on different datasets. It can be found from the above table that, with the increase of executor-cores, the operating time of both algorithms decreases to some extent.

Time Ratio of Parallel Algorithm to Serial Algorithm.
Under the same threshold condition, serial time and partition parameter are set to 16 when both algorithms are parallel program time, as shown in Table 5.
Parallelization D2HUP/EFIM represents the ratio of the parallel time of two algorithms. From the above experiments, when the dataset is small, the acceleration effect is not obvious because the serial program consumes less time, while the parallel program uses Spark for startup time and other reasons. But when the dataset size increases, the serial program consumption time increases. Compared with the serial algorithm, the Spark D2HUP parallel and EFIM parallel algorithms have greatly improved the running time.

Design of Algorithm.
In order to realize automatic selection of the fastest efficient itemset mining algorithm, this paper adopts the data-driven method to carry out experiments. First, different dimensional features of the dataset are constructed, and these dimensional features are taken as attributes of the classification algorithm. Second, all datasets were tested on EFIM and D2HUP algorithms, respectively, and the running time was recorded. Finally, the running results of EFIM and D2HUP are taken as labels for prediction. e flowchart of the system algorithm is shown in Figure 9.

Scientific Programming
Given a dataset D, build the following feature dimensions of the dataset, which are shown in Table 6.
In this paper, the mainstream classification algorithm logistic regression, C45, random forest, and SVM were used to predict the constructed datasets, extract the classification rules, and draw conclusions through analysis.

Design of Experiment.
e exposed datasets of the efficient itemsets are Accidents, BMS, Chess, Connect, Foodmart, Mushroom, Chainstore, Pumsb, Kosarak, and so on. However, due to the small sample size of the real dataset, the following strategies were adopted to construct the simulated dataset: (1) Generate simulated datasets. Given the five parameters of maximum rows, maximum columns, item number, and maximum utility value of the dataset, 19 simulated datasets were generated by dividing them into fixed columns and random columns. (2) Split nine real datasets in proportion. e real dataset was split into 20%, 40%, 50%, 60%, and 80% and divided into random percentage and fixed first few hundredths to generate 90 split datasets (3) ere are nine real data sets.
According to the above two strategies, a total of 118 datasets were generated; the EFIM algorithm and D2HUP algorithm were used for experiments in the above datasets to record the running speed. If the EFIM algorithm is fast, the tag is 1. If the D2HUP algorithm is fast, the tag is 0.
K-folded is a method of cross validation.       In general, we evaluate the generalization error of the model through experimental tests so as to select the learner with better performance. e dataset was divided into a training set and a test set, through which the machine learning model was obtained, and the performance of the model was evaluated. Due to the limited number of actual samples, in order to reuse the data, this experiment uses the k-fold cross-validation method.
Cross-validation refers to dividing the dataset D into k mutually exclusive subsets of similar size. In the process of dividing, each subset shall try to maintain the consistency of data distribution. Each time, one subset of k − 1 is taken as the training set and the rest as the test set. In this experiment, k equals 10.

e Result of the Experiment.
e obtained datasets were trained on logistic regression, C45, random forest, and SVM, respectively, and the experimental records were recorded using 10-fold cross-validation. e experimental results are shown in Table 7. Figure 10 shows the decision rules extracted from the C45 classification algorithm generation model. Given a dataset D, the user can decide how to choose the optimal mining algorithm for a high-consumption itemset according to the decision rules generated in Figure 10. First, if the number of transactions with a length of 6-10 in database D is no more than 490, then the EFIM algorithm is selected for the experiment (tag 1). Otherwise, for further judgment, if    the maximum transaction length (transMaxLen) in the database is greater than 267, then the EFIM algorithm is selected for the experiment (tag 1). Determine whether the average length of items in the database (itemsAvg) is greater than 3881.7. If it is greater than 3881.7, the EFIM algorithm is selected; otherwise, the D2HUP algorithm is selected. It is worth noting that, according to Figure 10, the important parameters are the number of transactions with a length of 6-10 in database D (transLen6, 10), the average length of items in database D (itemsAvg), and the length of the longest transaction in the database (transMaxLen).

Cross-Dataset Experiment Result.
In order to further verify the classification model based on the Spark set of high efficiency with the prediction accuracy of the parallel algorithm, 118 will generate the dataset of eight datasets on the Spark of the results as a test dataset (Chess, Pumsb, BMS, Connect, Accidents, Kosarak, dh26, dh29, etc.) and the rest of the dataset as the training dataset for training method (left); the final classification results are obtained as follows. rough experiments, it can be concluded that the classification algorithm model based on serial D2HUP and EFIM algorithm also has a good prediction of the running results of parallel programs.
In large-scale datasets, the efficiency of efficient itemset mining is an urgent problem to be solved. is paper parallelizes the serial algorithm based on the Spark platform, which greatly improves the running efficiency of the algorithm. In this paper, parallel D2HUP and EFIM algorithms are designed and implemented. e parallelization of an efficient itemset based on Spark is realized to reduce the running time of the mining algorithm.

Conclusions
In large-scale datasets, the efficient mining of itemsets is an urgent problem to be solved. In this paper, the serial algorithm is parallelized based on the Spark platform, which greatly improves the running efficiency of the algorithm. At the same time, since the D2HUP algorithm and EFIM algorithm have their own advantages and disadvantages in different datasets, and their running times are greatly different, it is a meaningful work to select an algorithm for efficient itemset mining in a given dataset. e main work contents of this paper are as follows: (1) By studying serial D2HUP and EFIM algorithms, design and implement parallel D2HUP and EFIM algorithms. Among them, the D2HUP algorithm realizes two different versions based on data partitioning and structure partitioning, and through large-scale experimental comparative analysis, it is concluded that the parallel efficient itemset mining algorithm has a greater improvement in running time efficiency than the serial algorithm. (2) In this paper, 118 datasets were obtained by generating and sorting out the running time of the two algorithms in the real and simulated datasets. e length, sparsity, and size of each data were taken as the features, and the running time was taken as the prediction target to establish the model. e decision rules are extracted from the model so that users can better choose the fastest and most efficient itemset mining algorithm to conduct experiments based on the given rules.
In conclusion, the automatic selection efficient itemset algorithm based on Spark has a fast running time and the most accurate prediction.

Data Availability
ere are commonly used Accidents in public data sets of efficient use, such as BMS, Chess, Connect, Foodmat, Mushroom, Chainstore, Pumsb, and Kosarak. However, due to the small number of samples of real datasets, it is necessary to build simulated datasets. Please refer to the experimental design for the construction of simulated data. e constructed simulated dataset was verified and compared on Spark based the EFIM algorithm and D2HUP algorithm, and the experimental dismissal was obtained.

Conflicts of Interest
e authors all declare that they have no conflicts of interest. 10 Scientific Programming