The multistage graph problem is a special kind of single-source single-sink shortest path problem. It is difficult even impossible to solve the large-scale multistage graphs using a single machine with sequential algorithms. There are many distributed graph computing systems that can solve this problem, but they are often designed for general large-scale graphs, which do not consider the special characteristics of multistage graphs. This paper proposes DMGA (Distributed Multistage Graph Algorithm) to solve the shortest path problem according to the structural characteristics of multistage graphs. The algorithm first allocates the graph to a set of computing nodes to store the vertices of the same stage to the same computing node. Next, DMGA calculates the shortest paths between any pair of starting and ending vertices within a partition by the classical dynamic programming algorithm. Finally, the global shortest path is calculated by subresults exchanging between computing nodes in an iterative method. Our experiments show that the proposed algorithm can effectively reduce the time to solve the shortest path of multistage graphs.
With the continuous development of big data and information technology, graph has been widely applied in many applications, and various graph structures and algorithms have been proposed. Among them, the multistage graph is a special kind of weighted directed graphs, which are widely used in engineering technology, concurrency control, transportation, task schedule in high-performance computing, and other fields. Many coordination or dynamic scheduling problems can be transformed into multistage graph problems [
Recently, the scale of graph data has grown tremendously, so it is difficult even impossible to store and process such large-scale graphs by a single computer or sequential processing method [ It presents a partitioning method for multistage graphs on distributed computing systems, which can make best use of characteristics of multistage graphs to achieve the best load balance and reduce the communication cost It designs a distributed algorithm of the shortest path problem of multistage graphs based on dynamic programming idea It performs extensive experiments to verify the performance of the proposed algorithm, compared to the classical parallel Dijkstra algorithm and the SSSP (single-source shortest path) algorithm on Pregel
Table
Notation overview.
Symbol | Definition |
---|---|
Graph with vertices set | |
Number of stages in a multistage graph | |
Set of vertices of stage | |
Number of vertices of stage | |
The | |
Weight of edge < | |
Set of edges from | |
Cost of the shortest path from vertex | |
The | |
Number of partitions | |
Cap | Capacity of each computing node |
ML | Maximum load of each partition |
Sum | Number of edges of a partition |
The first stage of CN | |
The last stage of CN | |
Index of the previous vertex of | |
List of vertices of the shortest path from | |
Set of partition IDs | |
SPL | Set of the shortest paths storing on CN |
Finding the shortest path is a classical problem of graph theory, and the well-known sequential algorithms are Dijkstra, Floyd, and Bellman–Ford algorithms [
The single-source shortest path (SSSP) is one of the most important shortest path problems. Peng et al. [
The above algorithms do not consider the structural characteristics of multistage graphs, so they will produce a large amount of communication overhead, resulting in tedious execution time.
The basis of the distributed graph processing system is to partition the entire graph into a set of computing nodes. The graph partition algorithms are classified into vertex-cut and edge-cut. Edge-cut partitioning assigns each vertex to a unique partition, and the edge spanning partitions are called cut edge. As shown in Figure
Two graph partitioning methods. (a) Edge-cut partition. (b) Vertex-cut partition.
The distributed graph processing systems often use the vertex-centric programming model [
The existing graph partitioning heuristic solutions are basically divided into offline and online partitioning strategies. The offline partitioning strategy refers to dividing the graph into several subgraphs before being loaded by the distributed system. F·Rahimian et al. [
The online partitioning strategy refers to partitioning the graph during the data loading process, where the input data is usually a vertex stream or an edge stream. Tsourakakis et al. [
The above graph partitioning algorithm are all designed for general graphs, which do not consider the special characteristics of multistage graphs, so it is necessary to design the graph partitioning algorithm for multistage graphs to accelerate the distributed processing.
A multistage graph
Figure
A multistage graph.
The shortest path problem of a multistage graph is to find the minimum cost path from the source vertex to the sink vertex. Let
Given a large-scale multistage graph
DMGA is run on the homogeneous cluster, which means all computing nodes have the same performance in terms of CPU, memory, and bandwidth. This algorithm partitions the entire graph to the given cluster first, and then, each computing node computes the shortest path of the partition on it. Finally, the computing nodes communicate with each other to obtain the shortest path of the whole graph.
Algorithm
Ensure: The shortest path from v1,1 to vm,1 Partition G Get the local shortest path of each partition Get the global shortest path of the whole graph
In order to determine the graph partition strategy, we should analyse their impacts on the communication overhead after partition. According to the feature of multistage graphs, it is a better scheme to divide the vertices of the same stage into the same partition because it is easy to implement load balance and parallel shortest path solution. Suppose
Two graph partitioning strategies. (a) Edge-cut strategy. (b) Vertex-cut strategy.
Since the multistage graphs studied in this paper are dense, we use the number of edges to represent the load of a partition. Let Cap be the capacity of each computing node, which is also the maximum number of edges that can be stored by a partition, then the number of partitions of a given
The above equation gives the lower limit of the number of computing nodes. It may lead to the load of the last computing node far lower than those of the other computing nodes. For example, if
The idea of multistage graph partition is to assign the vertices of the same stage to the same partition. Figure
Flow diagram of graph partition.
Calculate while if for all Assign it to end for else end if end while
After partitioning the graph, each computing node calculates the shortest path of the subgraph stored on it. The shortest path of each partition is referred to as the local shortest path, and the shortest path of the whole graph is referred to as the global shortest path.
If
We prove it by using reduction to absurdity. Suppose
The above theorem shows that any part of the shortest path is also the shortest path, so the global shortest path is composed of the local shortest paths of all partitions, and
Based on the above equation, it is necessary to calculate the shortest paths between any pair of vertices of the first and last stages for each partition. Specifically, each computing node uses the idea of dynamic programming shown as (
Figure
Flow diagram of local shortest path calculation.
Ensure: for end for for for for for all if end if end for end for end for end for for for while end while end for end for
After finding the shortest path of each partition, the global shortest path is calculated by message exchanging among computing nodes. Figure
Local shortest paths merging.
Figure
Flow diagram of global shortest path calculation.
Require: The local shortest paths of each partition Ensure: The global shortest path and its cost for end for while for all end for for all for all for all if if else if Update end if end if end for end for end for end while
Let us take Figure
Partition of the example graph. (a) CN1, 15 edges. (b) CN2, 17 edges. (c) CN3, 15 edges.
Next, each computing node calculates the local shortest path of its partition. Let us take Initially, The vertices of The vertices of The vertices of Finally, the vertices of
Similar to the above process, each
The last step is to merge the subresults of all partitions.
Because there are no public multistage graph datasets, we synthesize 5 datasets using Java on IntelliJ IDEA, where the number of vertices of each stage and the weights of edges are random values satisfying (
Multistage graph datasets.
Graph | Number of vertices | Number of edges | Number of stages |
---|---|---|---|
Graph_1 | 2000 | 5052 | 22 |
Graph_2 | 7002 | 21008 | 12 |
Graph_3 | 40002 | 199702 | 500 |
Graph_4 | 100000 | 499700 | 1000 |
Graph_5 | 1000020 | 5008710 | 10000 |
The shortest paths algorithms are run on Hadoop [
In order to compare the performance of DMGA with existing algorithms, all graphs are partitioned to 8 partitions, which is a little different from Algorithm
At present, there is no partition algorithm for multistage graphs, so we compare the partition quality of the DMGA algorithm with the Hash partitioning algorithm. Hash is the default partitioning algorithm in many distributed graph processing systems, and it is the basis of most of existing distributed algorithms for solving the shortest path.
For the vertex-cut partitioning method, the number of mirror vertices reflects the communication overhead. The fewer the mirror vertices, the less the communication overhead, and the corresponding calculation time will be reduced. Figure
Number of mirror vertices.
This shows that the DMGA algorithm has a better partitioning result for multistage graphs than the Hash algorithm. It can be deduced that the number of mirror vertices will be reduced in further if (
For the homogeneous cluster which consists of computing nodes with the same configuration, the load of each computing node should be balanced as much as possible in order to reduce the calculation time. Table
Edge distribution of graphs on computing nodes.
CN | Graph_1 | Graph_2 | Graph_3 | Graph_4 | Graph_5 | |||||
---|---|---|---|---|---|---|---|---|---|---|
Hash | DMGA | Hash | DMGA | Hash | DMGA | Hash | DMGA | Hash | DMGA | |
1 | 658 | 623 | 2553 | 2206 | 25138 | 25109 | 62395 | 62595 | 625049 | 620573 |
2 | 726 | 756 | 2835 | 3226 | 25139 | 26012 | 62677 | 60501 | 625274 | 605519 |
3 | 579 | 512 | 2661 | 2723 | 25145 | 24535 | 62279 | 64101 | 635371 | 646101 |
4 | 613 | 758 | 2544 | 3021 | 25078 | 26105 | 62039 | 61205 | 628324 | 612056 |
5 | 615 | 532 | 2579 | 2669 | 24901 | 25125 | 62901 | 63632 | 627591 | 636302 |
6 | 618 | 718 | 2572 | 2845 | 24590 | 24019 | 62410 | 61614 | 619432 | 621614 |
7 | 617 | 524 | 2636 | 2389 | 24836 | 22016 | 62836 | 63894 | 623853 | 639094 |
8 | 626 | 629 | 2628 | 1929 | 24875 | 26781 | 62163 | 62158 | 623816 | 627451 |
The average deviation and standard deviation are further analysed to check the loads among partitions, and they are shown in Figure
Average and standard deviations of edge distribution.
We compare the execution time of four algorithms: single-machine, DMGA, SSSP [
Table Graph_1: the sequential algorithm is faster than DMGA because the communication among computing nodes of DMGA takes up a lot of time. Graph_2: the running time of these two algorithms is almost the same. Graph_3 and Graph 4: the running time of the sequential algorithm is, respectively, 1.72 and 3.13 times of that of DMGA. Graph_5: the sequential algorithm meets “memory overflow” error, so this graph cannot be solved by a single machine. On the contrary, DMGA can obtain the shortest path.
Running time of four algorithms (unit: s).
Graph | Single machine | DMGA | SSSP | Parallel Dijkstra |
---|---|---|---|---|
Graph_1 | 4.06 | 5.56 | 8.56 | 101.57 |
Graph_2 | 17.91 | 15.52 | 11.14 | 452.36 |
Graph_3 | 59.13 | 34.46 | 1095.23 | 2642.58 |
Graph_4 | 381.56 | 122.06 | 1856.59 | 15893.45 |
Graph_5 | – | 7564.70 | 29324.73 | 85481.36 |
Comparison of running time of four algorithms.
In further, parallel Dijkstra has a relatively longer running time than the DMGA and SSSP, due to its high time and space complexity. SSSP uses Pregel’s default graph partition method which does not take the structural feature of multistage graphs into account, so it has a large amount of communication. SSSP needs longer time for the larger scale graph, which implies that the communication overhead increases significantly with the increase of the scale of the graph specifically. Graph_2: SSSP has the least running time, and DMGA only needs 4 seconds longer than SSSP, but parallel Dijkstra needs more than 30 times of that of DMGA. Other graphs: DMGA needs the least running time. The larger the scale of the graph, the more obvious the advantage of DMGA. For example, the running time of DMGA is, respectively, 25.8% and 8.8% of that of SSSP and parallel Dijkstra for Graph_5.
The above results show that DMGA makes full use of the special structural characteristic of multistage graphs, and it has extremely low communication overhead.
Nowadays, graph models are widely applied in many fields, and the scale of the graph increases significantly. The existing distributed graph computing systems cannot make full use of the special characteristics of the multistage graphs. To this end, this paper proposes DMGA which is used to solve the shortest path problem of large-scale multistage graphs on a distributed computing system. DMGA consists of three phases: graph partition, local shortest path calculation, and global shortest path calculation. The experiment results demonstrate its high-performance. However, the load of DMGA is not balanced, and it only considers of the vertex-cut partitioning method. In future, we will focus on reducing the number of mirror vertices and solution time as much as possible under the premise of load balance and propose the special algorithm based on edge-cut partitioning idea.
The experimental data used to support the findings of this study are available from the corresponding author upon request and also provided in the supplementary information files.
The authors declare that there are no conflicts of interest regarding the publication of this paper.
This work was supported in part by the National Key R&D Plan of China under Grant 2018YFC1406203, Shandong Postgraduate Tutor Guidance Ability Improvement Project under Grant SDYY17040, and Science and Technology Support Plan of Youth Innovation Team of Shandong Higher School under Grant 2019KJN024.