Communication analysis of distributed programs

Capturing and examining the causal and concurrent relationships of a distributed system is essential to a wide range of distributed systems applications. Many approaches to gathering this information rely on trace files of executions. The information obtained through tracing is limited to those executions observed. We present a methodology that analyzes the source code of the distributed system. Our analysis considers each process’s source code and produces a single comprehensive graph of the system’s possible behaviors. The graph, termed the partial order graph (POG), uniquely represents each possible partial order of the system. Causal and concurrent relationships can be extracted relative either to a particular partial order, which is synonymous to a single execution, or to a collection of partial orders. The graph provides a means of reasoning about the system in terms of relationships that will definitely occur, may possible occur, and will never occur. Distributed assert statements provide a means to monitor distributed system executions. By constructing the POG prior to system execution, the causality information provided by the POG enables run-time evaluation of the assert statement without relying on traces or addition messages.


Introduction
Core to a wide range of distributed system challenges is examining the causal and concurrent relationships among events of an execution.Some examples of these challenges are deadlock detection [2], debugging [6,11,12], rollback and recovery [5,9], termination detection [8,20], mutual exclusion violation [1] and global predicate evaluation [3,7,13,18].Techniques designed to meet these challenges is an ongoing area of research.Most current techniques make use of trace files generated during execution.Trace files of an execution provide the information needed to deduce the causal and concurrent relationships for that particular execution but are limited to the execution from which they were gathered.Other executions of the systems can define different relationships not present in the trace.
We present a methodology that provides a comprehensive examination of a distributed system.Each possible partial order is discovered and combined into a single, representative graph.This partial order graph, or POG, provides a complete view of the system without being tethered to any particular execution.From the POG, causal and concurrent relationships can be extracted.The extracted relationships can convey a view of the system ranging from a single execution to all possible executions.We have developed a prototype tool based on static analysis of a system's source code to produce the POG for asynchronous communication systems [14,15].
Taylor [19] has developed an algorithm for statically analyzing the synchronous communication of a distributed program.Synchronous communication occurs when the transmitting process blocks until the message is received by the destination process.Our research into the static analysis of source code was partially motivated by Taylor's results, but differs in both objectives and outcomes.
Taylor's work determines all possible total orders of execution where a single partial order may be embedded in many total orders.In enumerating the total orders, the concurrency defined by a partial order is not preserved.Our work uniquely identifies each partial order of the system.From each identified partial order, we preserve the inherent concurrency.
In Section 2, our system model is presented.Sections 3, 4 and 5 present each step in generating the POG from source code.Although we intitially present the method in a simplified environment, Section 6 explains how looping constructs are incorporated.Event relationships are extracted from the POG in Section 7. In Section 8 an example of the utility of the technique is provided.Conclusions are found in Section 9.

Model
A distributed system consists of a fixed number of processes Π = {P 0 , . . ., P N −1 }.The execution of a distributed program is viewed as a set of events E = E 0 ∪ . . .∪ E N −1 where E i represents the events of P i , and an irreflexive partial order is defined on these events [10]: →⊆ E × E. The → relation is commonly referred to as happens before or causally precedes.For e, f ∈ E, e → f if and only if e has potential causal impact upon f .
Interprocess communication defines the happens before relationship among events on different processes.Asynchronous communication [17] occurs when a process places a message "on the network," and continues execution.This type of communication, for example, is offered by the transport level datagram service UDP.The destination process blocks until it receives the message, then continues execution.We assume FIFO point-to-point communication.
In an asynchronous communication regime, the happens before relationship is the smallest relation satisfying the following three conditions: (1) if e and f are events in the same process, and e precedes f , then e → f ; (2) if e is the transmission of a message and f is the receipt of the same message, then e → f ; and (3) if e → f and f → g, then e → g.
If e → f , e causally precedes f and f causally succeeds e.If e → f and f → e, then e and f are causally unrelated or concurrent, denoted e f , and neither can causally affect the other.
One of possibly many partial orders is defined when a distributed system executes.The potential for many partial orders exists from branches in control of execution, from communication delays, and from unpredictable process execution speeds.Consider the source code of a three process distributed system shown in Fig. 1.The function async send(i, msg) transmits msg to P i , and async recv(j, msg) receives msg from P j .
One of two possible partial orders is defined when this program executes.The time-space diagrams representing the two possible partial orders, P O 1 and P O 2 , are shown in Fig. 2. Set P is the set of possible partial orders of a distributed system's execution.For the example system, P = {P O 1 , P O 2 }.For any given execution, one partial order, ρ ∈ P, is produced.
The first step in generating the POG is to examine the source code and create a flow graph for each process.Each flow graph enumerates the possible execution paths for that process.Communication analysis of the flow graphs is used to construct an intermediate graph that represents the causal and concurrent relationships possible in all executions of the system.An optimized graph, the POG, is derived from the intermediate graph.
A compiler for ANSI start C has been implemented to create a POG from source code.First, the steps to generate a POG are presented for source code without looping constructs using the small example pseudo code shown in Fig. 1 as an example.Looping constructs are then incorporated into the POG generation methodologies and two more complex program examples are given.as ync r ecv( 0, x) z = 4 x = 2 as ync r ecv( 2, y) a s y n cs end( 1, y) as ync s end ( 1, x) else end end as ync r ecv( 2, y) as ync r ecv( 0, x) endif as ync r ecv( 0, x) end Fig. 1.A simple 3 process system.

Control flow graphs
The possible flows of control through each process, P i , in the system are represented by a control flow graph F G i .The graph F G i = {V i , A i , α i } where V i is the set of nodes, A i is the set of arcs, and α i ∈ V i is the root node of the graph.Nodes represent computation, communication, or control constructs in the source code.The root node, α i , is the start of execution of process P i .Arcs represent the flow of execution of a process created from the source code.If an arc exists from node v to node v , then v can be executed immediately following the execution of v. Multiple paths may exist through F G i from α i , but all paths will terminate in a single leaf node, ω i if the program terminates.
Returning to the example source code of Fig. 1, we see that processes P 0 and P 2 have a single execution path.Process P 1 , containing a selection construct, has two possible execution paths.
A slight reduction in the complexity of each F G i is realized through the collapse of consecutive computation statements into a single representative node, c.We represent communication constructs as t j , the transmission of a message to P j , and r j , the receipt of a message from P j .Nodes representing statements in the body of a selection construct are bounded by an IF node and an ENDIF node. Figure 3 shows the control flow graphs derived from the example source code.
The nodes of F G i represent syntactic constructs in the source code of P i .An execution of P i may be viewed as a traversal of F G i , from α i to ω i .An event created from the execution of a statement in P i corresponds to the locus of control passing through the node of F G i representing that source code statement.In the remaining discussion of the flow graphs, the symbol representing a node of F G i is also used to represent the event corresponding to the execution of the source code associated with that node.Nodes and events will be distinguished by context.
To summarize the discussion of F G i , the graph F G i represents all possible execution paths of P i .A path from α i to ω i represents a single, possible execution of P i .The order of events on the α i to ω i path indicates the execution order of those events if the selected path is executed.If both events v and v are executed in P i such that v → v , then a path in F G i will exist from v to v .We define an immediate communication successor set, ICS(v), for each communication node, v ∈ V i .Node v is an immediate communication successor of node v if (1) there exists a path from v to v , (2) v ∈ {t j , r j , ω i }, and (3) there does not exist a communication node v = v on the path from v to v .Immediate communication successor sets are used in the following construction of the intermediate graph.

Intermediate graph
From the flow graphs of the distributed system's constituent processes, an intermediate graph S is constructed.This graph represents all possible executions by representing all possible partial orders of the system.Concurrency as well as causality among the events is preserved.A partial order graph, POG, will be constructed from S by combining nodes that are both causally equivalent and derived from equivalent partial orders.
We begin by generating concurrent communication states, CCSs, from the flow graphs.Each CCS is an ordered Ntuple, (v 0 , v 1 , . . ., v N −1 ), where v i ∈ V i is either α i , t j , r j , or ω i .If v i ∈ {t j , r j }, it denotes the next communication command to be executed in P i .Each CCS is represented by a node in S.
The root node of S, CCS 0 , contains the root nodes of each flow graph.
Successor CCSs are generated using the set of immediate communication successors, ICS(v), for each entry, v, of the CCS.The S graph represents the hierarchy of successor CCSs.
(α 0 , α 1 ,α 2 ) Causal communication constraints force message receipts to be delayed until their corresponding transmissions have been processed.Those receives that can not be immediately processed are not "ready."A "ready" receive in a CCS is one whose corresponding transmission command occurred in an ancestor CCS.Root nodes of each flow graph, α i , and message transmission elements, t j , are intrinsically "ready".
All the communication commands of a CCS that are ready to execute are concurrent.Other concurrent relationships may also be present in S as will be examined in a later section.
A CCS with no ready elements has no successor states.The immediate successor states, ISS(CCS), are determined from the immediate communication successor sets of the CCS's ready commands.The states in ISS(CCS) will be children of CCS in S. The following steps determine ISS(CCS): 3. Repeating step 2 until all unique CCS's are generated from the ready commands' successor sets.The number of successor states of CCS is no ready commands and one or more receive commands (that are not ready) has no successors and is an invalid terminal state of the distributed system.A CCS comprised of all ω i elements is a valid terminal state.
Ready communication events in each CCS will be annotated with an integer counter which will be used in later steps of POG construction.The annotation on each message transmission to P i indicates the number of transmissions to P i encountered in predecessors of the current CCS.For example, the entry 3t 1 indicates that this message transmitted to P 1 has been preceded by two other messages sent to P 1 .The number prepended to the transmit events does not imply the receipt order of messages.Message delivery order is determined by the execution order of receive events in the current partial order.
A ready receive is prepended with the number taken from the matching transmit.For example, the r j ∈ P 1 matching the transmit, 3t 1 , would be prepended with the number 3. While transmissions are numbered sequentially, numbers associated with receipts may be unordered.Values attached to receipts will be used to distinguish textually identical receipts that occur in different orders, thus defined by different partial orders.Not only does the numeric , 2t ) , α ) attachment distinguish between syntactically identical communication commands, it also provides a method of matching transmissions with corresponding receipts.Figure 4 shows the S graph derived from the flow graphs of Fig. 3. Appended numbers are shown, and "ready" events are underlined.Consider the left partial order in Fig. 4. The first node following the root consists of a receive in P 1 that is not ready and two ready transmissions that are sequentially numbered.In the successor node, the receive from P 0 is now ready since the the transmit from P 0 to P 1 occurred in an ancestor node.In the right partial order, the first ready receive of P 1 is a receipt from P 2 .Referring back to the source code of Fig. 1, the order of receives in P 1 differs within the if-else construct.
Receives that are not ready in a node of S are indicators of what can happen next in a process.The ready events are the only events that occur, and they forge the causal relationships among events on different processes.If only the ready events of each node in S are considered, we observe that nodes can contain identical events.In the following section, the partial order graph of the system is constructed based on these observations.

Partial order graph
The construction of the partial order graph, POG, from the S graph is based on the observations outlined in the previous section.Only ready commands will be represented in the POG, where in the S graph, all commands were represented to facilitate the matching of transmits and receives.In some cases, two or more branches of S represent the same partial order.From the tree S, the POG is constructed by combining branches that represent the same partial order into a single representative branch containing only ready commands.
Consider the simple S graph shown in Fig. 5 constructed from a four process system.The receives of P 0 are not ready.The transmits of each CCS are replaced in the child CCSs with a terminal node of F G i .Both leaf node branches indicate that P 0 does not complete execution.The two branches shown represent the same partial order.Receives that are not ready are not part of the partial order.
A POG is a directed graph (N, A, η) where N is the set of nodes, A is the set of arcs, and η ∈ N is the root node of POG.The nodes of the POG are generated from nodes of S such that the POG nodes represent the transmit and ready receive commands of the S nodes.
Since only the ready events of an S node need representation in a POG node, a node of the POG may not contain an element from each P i .The executing process of an event in a node will not be positionally identified as in S. Instead, subscripts will identify the process executing the command.A transmission entry has the format ct j i where c is the counter, i is the process executing the transmit and j is the destination process.A ready receive entry has the format cr j i where c is the counter, i is the process executing the receive and j is the sender.In constructing the POG, it will be necessary to determine if a set of CCS's has equivalent transmit and ready receive communication entries.Equivalent communications are found in CCS and CCS' if each ready communication command in one CCS exactly matches (in both prepended value and communicating partner identification) the positionally corresponding command in the other CCS.Specifically, CCS and CCS' have equivalent communications if the following conditions are met.
-At least one transmit or ready receive command is in either CCS or CCS'.
where χ is either t or r and cχ j is ready, then v i = cχ j ∈ CCS' and cχ j is ready.If CCS and CCS' have equivalent communication commands, the equivalent communication commands are all the transmit and ready receive commands that occur in CCS and CCS'.
We begin by traversing S breath-first and identifying the equivalent nodes.From these nodes, a representative node is created in the POG that is labeled with the equivalent communication commands of the identified nodes.An algorithm is presented for the POG construction.The following data structures and functions are essential to that algorithm.
VisitNodes :: a queue with entries of the format <POGnode, SnodeSet>.POGnode is a reference to a node of the POG and SnodeSet is a set of S nodes.SuccSet :: a set of S nodes built from the successors of equivalent nodes.Create PNode(CommoLabel) :: creates a node of the POG and labels the newly created node with the string CommoLabel.ISS(Snode) :: constructs a set of nodes from S that are immediate successors of node Snode.Equiv Set(SnodeSet) :: extracts from SnodeSet those nodes that have equivalent communication commands and returns these nodes as a set.CreateLabel(SnodeSet) :: creates a label from the transmits and ready receives of the equivalent nodes in SnodeSet.
A string label is returned.Note that SnodeSet may only have one entry and the label returned is for the ready commands of this one node.
The algorithm for creating the POG from S is given in Fig. 6.The construction prunes the tree S so that one branch of the POG from root to leaf node represents an unique partial order ρ ∈ P. The resulting POG represents all partial orders of the system.
The communication behavior of the system directly influences the size of the POG.The width of the graph is determined by the number of possible partial orders.For each control branch that results in two communication choices, a new partial order is created thereby producing a branch in the POG.Admittedly, the POG's width can grow large, but the graph accurately represents the complexity of the communication.
The worst case performance of the static analysis is exponential in the number of possible concurrent states.For the worst case, an unlikely system must exist.Assume every node of a flow graph can occur in the same concurrent state with every node from the other processes' flow graphs.If we let T be the number of nodes of all the processes' flow graphs, then an upper bound on the number of nodes of one flow graph is O(T ).The worst case bound on the number of concurrent states is O(T N ), where N is the number of processes in the distributed application.Although static analysis can have exponential performance, the time spent analyzing does not affect the execution of the distributed system.The analysis is done prior to execution, and provides insight into the application's behavior that can be leveraged at run-time.
Figure 7 shows the POG representing the example distributed program of Fig. 1, and generated from S of Fig. 4. Notice that the two partial orders of Fig. 2 are each represented as a path from root to an end node in the POG.In particular, the left path of the POG represents P O 1 , and the right path of the POG represents P O 2 .

Looping constructs
Three looping constructs are considered: do-while, while and for.Each type of loop has one unique entry point and one unique exit point.When loops are nested, each loop has it's own entry and exit point.Each loop in a process's source code is represented as a cycle in the process's corresponding flow graph.The cycle is accomplished with a back edge from the exit point of the loop to the entry point of the loop.
To demonstrate the incorporation of loops into the steps for generating the POG, two distributed programs have been selected.The first is set partition [4], SETPART, which was chosen for its concise applicability.SETPART consists of two distributed processes, P 0 and P 1 , that partition disjoint integer sets S and T .P 0 maintains S and P 1 maintains T .An element of S is exchanged with an element of T until the elements of S are less than the elements of T .The source code of SETPART is shown in Fig. 8 and the corresponding flow graphs are shown in Fig. 9.
The second program is a distributed sort, DS, which was chosen for its increased size and complexity.A total of q integers are sorted in ascending order by six processes.The process are connected in a logical ring so that P i 's neighbors are P i−1 and P i+1 .Initially each process is assigned q/6 elements.Each process follows the general source code as shown with P i in Fig. 10.The corresponding flow graphs are shown in Fig. 11.With the possibility of loops in the source code of each process, loops are also possible in S. Additions are required for detecting the repeated execution of communication commands and representing these repetitions as cycles in S. Cycles occur in S if 1. a send command is in the body of a loop, 2. a send command and its matching ready receive are each in a loop body, or 3. a combination of ( 1) and ( 2).
Following are the steps taken to incorporate loops into S.When a node n is added to S, a check is made to determine if the state represented by n has already been represented by another node in n's execution path.This is done by comparing n with its ancestors.First n is compared with its immediate predecessor, i.e., parent node.If the parent does not represent the same state, then the grandparent is compared against n.This continues until either a node that represents the same state of n is found or the root of S is reached.
Two comparisons are required two determine if node n and its ancestor node n represent the same state.Nodes n and n represent the same state if -for each entry v i ∈ n, there exists v i ∈ n that is identical with the exception of the counter value, and -for each v i and v i pair, v i and v i correspond to the same node of F G i .
If nodes n and n represent the same state, then node n is possibly the entry point of a loop, and the parent of n is possibly the exit point of this loop.The next decision is whether to add a back edge from the parent of n to n to form the loop.
If n is the parent of n, then a loop has been detected.A back edge is added from n to itself and node n is removed from S. If n is not the parent of n, then n must be an ancestor of the parent of n.That is, other nodes are found between n and n in S. To identify a loop, the body must be repeated as nodes in S. A loop is detected only if the nodes from n to the parent of n, the loop body, are repeated immediately following node n.A loop detection results in a back edge inserted from the parent of n to n and the removal of n and its descendants from S.
The S graph for SETPART is shown in Fig. 12 and the S graph for DS is shown in Fig. 13.Note that for SETPART only one cyclic behavior exists where as with DS four loops exist.The details of the algorithms and the implementation for generating S can be found in [14].
Loops incorporated into S propagate to the POG.Only the function Equiv Sets() of the algorithm to create the POG needs modification to handle the back edges of S. Additional tests are needed to determine the equivalency of S nodes.Suppose CSS and CSS' are found to have equivalent communication commands, and the nodes that represent CSS and CSS' are n and n .Function Equiv Sets() must check whether both n and n have an incoming back edge or both n and n have an outgoing back edge.
If no back edges are found, then n and n are equivalent.If only one of the nodes is referenced by a back edge, then n and n are not equivalent.When nodes n and n are each pointed to by a back edge, both nodes are entry points of loops in S. Next is to determine whether the loop associated with node n is equivalent to the loop associated with node n .
Suppose node b is the node that has a back edge to node n.Nodes n and b form a subtree rooted at n, where the nodes that are descendants of n but not descendants of b comprise the nodes of the subtree.Also suppose node b is the node that has a back edge to node n , then nodes n and b also form a subtree.Nodes are compared as the subtrees are traversed in lock step, starting at the root nodes, in depth first order to determine if the loops are  equivalent.
The current node c of one subtree is compared against current node c of the other subtree.If the CCS of node c is equivalent to the CCS of node c , and the number of children of c is equal to the number of children c then the traversal continues.If either condition is false, the loops are not equivalent and the traversal stops.If the subtrees are traversed without falsifying either condition, then the loops are equivalent and nodes n and n are represented by a single node in the POG.
If both n and n both have an outgoing back edge, then an addition test is required to determine the equivalence of the nodes.Let d be the node pointed to by the back edge of n and let d be the node pointed to by the back edge of n .If d and d are equivalent, then so are n and n .Both n and n will be represented by a single node in the POG.A single back edge will be added from the POG node representing n and n to the POG node representing the equivalent S nodes pointed to by the back edge of n and n .Figures 14 and 15 show the POGs generated for SETPART and DS.

Event relationships
The POG conveys concurrent and causal relationships for each possible partial order.Some of the concurrent relationships are explicitly represented in each node of the POG where other concurrent and causal relationships can easily be derived.Consider the node in level 2 of Fig. 7.The two transmissions, 1t 1 0 and 2t 1 2 , are explicitly shown to be concurrent.Specifically, the communication events represented by a node of the POG are concurrent in that partial order.Other concurrency relationships are implicit and must be derived from absence of causality.If a transmission event, ct j i , is an entry of node n of the POG, then the matching receive, cr i j , is found in a descendant node of n.For example, consider again 2t 1  2 of level 2. The matching receive, 2r 2 1 , is found in the left partial order in a descendant node at level 4. The matching receive is also found in the right partial order in a descendant node at level 3.
The positional relationships between nodes, as well as the event content of nodes, is used to derive relationships among events.Consider, for example, event 1r 0 1 in level 3 and event 3r 0 1 in level 5 of the left partial order.The common subscript indicates that both events are executed on P 1 , thereby ensuring that a causal relationship exists between them.Since 1r 0 1 occurs in an ancestor node of 3r 0 1 , we conclude that 1r 0 1 → 3r 0 1 .Deriving the remaining interprocess relationships requires additional reasoning.
A communication transitive path is relevant to a particular partial order and follows the definition of happens before.A communication transitive path of t + 1 events, from e 0 j to e t i =j , is a series of communication events e 0 j , . . ., e t i such that e v k → e v+1 l ,where there does not exist an event e that is an event of the path such that e  If e 0 j , e 1 k , e 2 k , e 3 l , e 4 l , e 5 i is a valid communication transitive path of length 6, e 0 j is a transmission to P k , and e 1 k is the corresponding receipt, e 2 k is a transmission to P l , and e 3 l is the corresponding receipt, and e 4 l is a transmission to P i and e 5 i is the corresponding receipt.For any two communication events, e and e , represented in the POG , exactly one of the following relationships will hold: e and e are in different partial orders; e and e are concurrently related, e e ; or e and e are causally related, either e → e or e → e.
If node n, representing event e, and node n , representing event e , are not on a single path from the root to the END, then the nodes are contained in different partial orders.The two events can not occur in the same execution since they are in different partial orders.Causal and concurrent relationships do not exist between events in different executions.Therefore, e and e are not related: neither through causality nor concurrency.
If, on the other hand, a single path from the root to the END contains both n and n , then the nodes are contained in the same partial order.Events e and e , now from the same execution of the system, are related.If a communicative transitive path exists from e to e , then e → e .If no communicative transitive path exists between the events, then e e ; the events are concurrent.
The information derived from the POG is not limited to a single execution of the system.Since all partial orders are represented, regarding the possibilities of execution can be answered.For example, -Is it possible that e and e can execute concurrently?-Is there a single event that will always precede event e? -Which events can have an immediate causal effect on e? -Can event e have a causal effect on event e ?These questions indicate the significance of a unified representation of a distributed system.Challenges of distributed systems that require reasoning about event relationships can be answered with respect to one, any, or all possible executions.

Monitoring execution
Reasoning about causal and concurrent relationships is fundamental to monitoring and debugging distributed systems.Monitoring the execution of a distributed program requires reasoning about constituent processes' execution as a single collective entity.We have extrapolated the semantics of the assert statement for sequential programs into the distributed context and developed a run-time methodology for monitoring and debugging using distributed assert statements.The POG is the basis for the efficient evaluation methodology of the distributed assert statement.
The distributed assert has the format assert(P where P is a global predicate that is anchored at a control point of one processes and evaluated when the process executes the assert.If P is true then the program continues its execution.If P is false, however, the program is aborted, and a diagnostic message is produced.Our global predicate is not restricted by the format of the logical comparisons, and variables from different processes can be compared directly.Both stable and unstable properties can be monitored.A distributed assert statement monitors a distributed system's execution, but only a subset of the system execution states are relevant for evaluation.In particular, the distributed assert presented in this paper monitors the execution having the most recent causal impact on the assert statement.
The property that is monitored is determined by the predicate.Examples of deadlock detection, mutual exclusion violation and specific behaviors of a program can be found in [14].Distributed assert statements have been developed for the two example programs SETPART and DS as shown in Figs 16 and 17, respectively.
The assert statement in P 1 of SETPART, is evaluated on each exchange of data between the processes.The clauses of the assert predicate compare variables from both processes.A false evaluation indicates erroneous execution of the program.SETPART's error is identified by the assert's falsifying clause.If y is not equal to max(S); P 0 did not send the correct value.If max(S) mn; processing should have stopped on the last exchange, and a likely error is P 0 's exchange loop condition.If mn > x; either a value other than the minimum of T was chosen, or P 0 has erroneously altered the variable x since the last exchange.If the new size of S has changed, P 0 has not correctly added or removed a value from S since the last exchange.If the intersection of S and T is not equal to y; either S or T has not been correctly updated since the last exchange, and the results of the other clauses help in identifying the incorrect set.For example, suppose the programmer mistypes async send(1, mx) of P 0 as async send (1, x).The clause y =max(S) evaluates to false and identifies an incorrect value sent by P 0 .
For DS, process P 2 's source code is shown below with the two assert statements A 2a and A 2b .Two assert statement could be in any one of the six processes and provide the same meaningful information.Process P 2 was arbitrarily selected.The clause P i .listP i .recvlist tests whether every element in P i .list is less than or equal to all elements of P i .recvlist, and the clause P i .listP i .recvlist tests whether every element in P i .list is greater than or equal to all elements P i .recvlist.The clause P i .recvlist = P i+1 .list,for i = 2 . . .4, of assert A 2a determines whether process P i received the correct list from its right neighbor P i+1 .The clause P i .recvlist = P i−1 .list,for  i = 1 . . .2, of assert A 2b determines whether process P i received the correct list from its left neighbor P i−1 .The clauses P i .listP i .recvlist and P i .listP i .recvlist ensure that merge sort( ) correctly sorted and halved the merged list.
For a distributed assert to be evaluated during run-time, the local state information of the constituent processes must be shipped to the process executing the assert.This is necessary since the assert predicate can compare variables from the different processes but the value of these non-local variables are not available in the process executing the assert.
A brute-force implementation of evaluating distributed assert statements would have each process piggyback its state information on each of its outgoing messages.This approach would alter every message in the distributed execution and porportionally increase the execution time of the system [14].By using the POG, we can reduce the number of messages piggybacking state information.The amount of reduction varies according to the distribruted program, but, in general, the reduction is significant.For each partial order, exactly one message in each process is identified for piggybacking the correct state information.
If event e in P i is the evaluation of the assert statement, LCP (e, j) where j = i, denotes event e's latest causally preceding in P j .We define LCP (e, j) = f if and only if f is an event in P j , such that f happens before e, and there does not exist an event f in P j such that f happens before f and f happens before e.Given a partial order and an event e, there exist exactly one LCP event in each process.
The LCP events are the means by which we reduce the number of messages piggybacking state information, which can be identified from the POG.By using the POG, a distributed assert is evaluated but the methodology neither generates and analyzes traces, nor adds messages to the original distributed application.Only the LCP messages are increased in size to propagate state information to the assert control point for assertion evaluation [16].
For event e, a causal cut through e is the set of events consisting of e and the LCP event of e of each process for a partial order α.The causal cut through event e, denoted CC(e), is defined as Intuitively, CC(e) is the "latest" set of events of a partial order which can have a causal impact upon e.For each partial order, there is one causal cut for a given event.The LCP events that comprise the causal cut for one partial order may differ from the LCP events that comprise the causal cut for a different partial order.
From each causal cut CC, a global state exists for evaluating a distributed assert statement.If e is the execution of a causal distributed assert statement in P i , then the causal global state, anchored on e, is The pre(f ) denotes the local state of P i in which the execution of f is begun.When the system executes, one of the possible partial orders is identified.The global state corresponding to this partial order can then be used for assertion evaluation.
In [16], it is proved that LCP events are communication events.In particular, for event e of P i , each LCP(e,j) is a send event.Corresponding to each LCP send event is a receive event, denoted LCP'.A causal cut consists of LCP send events.The LCP and LCP' events of a partial order comprise the communication events that are sufficient for delivering the CGState data to the process evaluating the assert.In Figs 14 and 15 the underlined entries indicate the LCP and LCP' events.
In [14,15], a two-pass compiler prototype system for enabling the evaluation of a causal distributed assert statement is presented.This system ensures that when an assert is executed, the relevant components of the causal global state are immediately available at the process executing the assert.From the POG, the causal cuts can be identified prior to program execution.Our prototype statically analyzes the distributed source code to -generate the POG, -identify the LCP and LCP' events of the distributed assert statements from the POG, and -append the necessary causal global state information to the already existing LCP send commands.

Conclusions
This paper has presented a methodology for analyzing the communication of a distributed system.Source code is analyzed to produce a partial order graph, POG, that conveys all possible execution scenarios.In particular, the POG represents all possible causal relationships that can be forged by executing the system's processes.Concurrent relationships are upheld in the POG by not imposing an order on the events where causality does not exists.
Our methodology first produces a graph representing the flow of control through each program of the distributed system.These flow graphs represent the possible orders of execution of statements in the source code.Each path through the graph from root to leaf provides the execution order of events within a single execution of the program.
The flow graphs of the individual programs are combined to create a unified representation of the system, resulting in an intermediate graph S. The graph S represents all partial orders of execution while preserving concurrency.However, the graph may contain duplicate nodes that can be combined to produce a more compressed, yet still accurate, representation of the system.
The final resulting graph, the POG, is constructed from the intermediate graph by combining duplicate representations of the same partial order.The POG is derived from S so that each path from the root node to a leaf node represents one partial order and each partial order is presented in one path of the POG.The generation of the POG from C source code has been accomplished with a prototype system.
Causal and concurrent relationships can be derived from the POG and this information can aid in solving difficult distributed system challenges.Distributed assert statements provide a means to monitor the execution of a distributed system.By using the causal information contained in the POG, messages can be identified prior to execution for delivering the state information to the assert's control point for run-time evaluation.The POG enables the identification of the latest causally preceding messages relative to the assert's control point.A prototype two-pass compiler exists for identifying the latest causally preceding messages and automating the piggybacking of relevant statement information on these messages.The POG enables this ability.
v k → e → e v+1 l , -for e v k and e v+1 l , where k = l, e v k and e v+1 l are a transmit/receive pair (e v k being the transmission and e v+1 l being the receipt), and -for e v k , e v+1 l , where k = l, the next event of the path (if it exists), e v+2 l , must occur on P l .
Fig. 17.P 2 of Distributed Sort with asserts.