PERFORMANCE LIMITATIONS OF PARALLEL SIMULATIONS

This study shows how the performance of a parallel simulation may be affected by the structure of the system being simulated. We consider a wide class of "linearly synchronous" simulations consisting of asynchronous and synchronous parallel simulations (or other distributed-processing systems), with conservative or optimistic protocols, in which the differences in the virtual times of the logical processes being simulated in real time t are of the order o(t) as t tends to infinity. Using a random time transformation idea, we show how a simulation’s processing rate in real time is related to the throughput rates in virtual time of the system being simulated. This relation is the basis for establishing upper bounds on simulation processing rates. The bounds for the rates are tight and are close to the actual rates as numerical experiments indicate. We use the bounds to determine the maximum number of processors that a simulation can effectively use. The bounds also give insight into efficient assignment of processors to the logical processes in a simulation.


Introduction
Analysts rely on discrete-event simulations to evaluate the performance of large-scale, complex systems, such as telecommunications networks and computer systems.Exist- ing simulation packages based on serial processing of events, however, are often inade- quate for large, realistic systems.The alternative is to use simulations based on parallel processing.Several protocols for parallel simulations have been developed for general systems as well as for special purpose applications.For a survey of these protocol, see Fujimoto [5].Each protocol has its strengths and weaknesses depending on the application at hand and the mechanisms and techniques used for synchroniz- ing the parallel processors.There have been several studies of the speedup of parallel simulations for particular protocols and applications; see for instance [1, 6, 8, 10].
The approach in these studies is to model both simulator and simulated system as a single Markovian stochastic system at a detailed level.Another approach is to cap- ture the major characteristics of a parallel simulation protocol by using coarser per- formance measures based on macro-level assumptions that are not too sensitive to detailed properties of the simulation protocol and the simulated system.
The present paper is such a macro-level study of parallel simulations.The aim is to give insights into the following issues"  What is the maximum number of processors that can be usefully employed in a parallel simulation?
Does the structure of the system being simulated limit the maximum poten- tial processing rate of the simulation?How do non-homogeneous processors differ from homogeneous ones in affect- ing a simulation 's execution rate?How is the maximum potential processing rate of a simulation affected by processor scheduling: the way in which processors are assigned to execute events of the processes?
How is the processing rate (in real time) of a simulation related to the throughput rates (in virtual time) of the system being simulated?
In this paper, we study the potential performance of a parallel simulation of a general discrete-event system consisting of several interacting logical processes.Events associated with the logical processes are executed by a set of processors over an infinite time horizon.The simulation may be synchronous or asynchronous, and conservative or optimistic.Although we present our results in the setting of discrete- event simulations, they also apply to other types of discrete-event systems using distri- buted computations.
The evolution of a logical process in the simulation is presented by its virtual time (simulated time) Ti(t).This is a random function of the events it processes in the real time (simulation time) t.In a synchronous parallel simulation, the virtual times of all processes are equal (Ti(t) Tj(t) for all i,j and t).We consider systems with a significantly weaker virtual-time conservation principle that all processes have the same long-run average virtual speeds (as defined in the next section).This does not mean that all logical processes or processors in a simulation must be homogeneous, but only that their virtual time flows have the same rate in the long run.We show that this principle is satisfied for a wide class of simulations that are linearly synchronous.In such a simulation, the virtual times of the processes may vary considerably as long as their difference from each other (or from the simulation's global virtual time) is of the (linear) order o(t) as t tends to infinity.In an aggressive Time Warp simulation, for instance, it is often the case that the difference between a processor 's virtual time and the global virtual time is bounded by a constant (this may even be a constraint inherent in the protocol).Such simulations are therefore linearly synchronous.We also show that the linearly synchronous property is equivalent to the virtual-time conservation principle.
For the class of simulations that are linearly synchronous, we show that their simu- lation processing rates have a natural, simple representation in terms of the through- put rates of the simulated systems.The proof of this is based on the fundamental property that the number of events the simulation executes at real time is a random time transformation of the number of events in the simulated system (see expression (2)).This time transformation idea relating the simulation to the system being simu- lated is implicit in studies of parallel simulations, but it has not been exploited expli- citly as we do here.The analysis also uses sample-path properties of stochastic pro- cesses and strong laws of large numbers.
After characterizing linearly synchronous processes, we study their maximum potential processing rates under the following processor scheduling strategies.
Autonomous processor assignments.Each processor is assigned to a subset of processes, and the subsets are disjoint.
Group processor assignments.Disjoint groups of processors are assigned to disjoint subsets of processes (a special case is global scheduling all processors are assigned to all processes).
Using the relation between the processing rate of the simulation and the system throughputs, we derive upper bounds on the simulation's processing rate under these processor scheduling strategies.The bounds are tight and our numerical examples in- dicate that they tend to be close to the actual processing rates.We describe how the bounds can be used to obtain efficient processor scheduling strategies.
The main interest in the bounds is that they show how a simulation may be limited by the structure of the system being simulated.A conventional view is that if K homogeneous processors are employed in a parallel simulation, then in an ideal situation, there would be a simulation protocol such that the speedup of the simula- tion is approximately K. Our bounds show, however, that for most applications, the speedup is much less than K no matter what simulation protocol is used.The reason is that the efficiency of a parallel simulation may be limited by the structure of the simulated system.We give expressions for the maximum number of processors that can be usefully employed in a simulation; more processors beyond this maximum will not add to the speed of the simulation.
In a related study of parallel simulations of queueing networks, Wagner and Lazowska [11] show that, under a conservative protocol, the structure of the network limits the parallel simulation speed.They also gave an upper bound on the speedup of a specific queueing network simulation.Another study on speedup bounds for self- initiating systems is Nicol [9].Also, Felderman and Kleinrock [4] give several per- formance bounds for asynchronous simulations of specific applications under the Time Warp protocol.The ideas and analysis techniques used in these studies are geared to the particular models they consider and do not apply to the general setting herein covering both conservative and optimistic protocols and a wide class of protocols that obey the virtual-time conservation principle.The rest of the paper is organized as follows.Section 2 describes the class of linear- ly synchronous parallel simulations that satisfy the virtual-time conservation princi- ple.Section 3 contains upper bounds on the simulation speed under autonomous and group processor assignments.Section 4 gives a numerical example, and Section 5 gives insight into obtaining efficient processor assignments.

Conservation Principle and Linearly Synchronous Simulations
In this section, we discuss a conservation principle that is satisfied by a large class of what we call linearly synchronous simulations.For such a simulation, we derive a simple relation between the simulation's processing rate and the throughput rates of the system being simulated.
We shall consider a parallel simulation of a discrete-event system, denoted by S, consisting of n logical processes that are simulated by a set of processors over an infin- ite time horizon.We also write S {1,...,n}.Associated with a logical process is a stream of events for the system (e.g., a stream of service times, waiting times and de- parture times for a node in a queueing network).The events of one process might de- pend on events from other processes.Each event contains several items of information including its virtual timestamp the virtual time that it is scheduled to occur in the simulated system.In this section, we do not specify how the processors in the simulation are assigned to the processes, such assignments are the subject of the next section.Messages are exchanged between processes (or processors) for synchronizing or coordinating their executions of events in the simulation.Following the standard convention, we also call these messages events.An event in the simulation is said to be a true event if it actually occurs in the simulated system S.
The other events are called synchronizing events; they are generated only for synchronizing the executions of the processes.The simulation may be asynchronous or synchronous and its protocol may be conservative (like the Chandy-Misra protocol) or optimistic (like an aggressive Time Warp protocol).Aside from its processing rate information, our analysis does not require other details of the protocol.
The speed of the simulation is described in terms of general processing rates of events as follows.Simulation time (the time the simulation has been running) is re- ferred to as real time and is denoted by t.The simulated time (elapsed time) of pro- cess in the simulation is called its virtual lime, and is denoted by Ti(l).This time is usually the minimum of the timestamps of the events of the process that may have been executed but are still subject to being canceled or rolled back (uncommitted events).This Ti(t is a random function of real time that may decrease periodical- ly but it eventually tends to infinity.Under a conservative protocol, the Ti(t will typically be nondecreasing in real time, but under an optimistic protocol, this virtual time may decrease periodically due to rollbacks.The simulation speed of process is measured by its virlual speed v -lim t-1Ti(t).
This limit and the others herein are with probability one.We assume that the limit v exists and is a positive constant.This is a rather natural law of large numbers for an infinite horizon simulation.The v can also be interpreted as the average amount of virtual time simulated for process in one unit of real time.Let Ni(r denote the number of (true) events in the system S associated with process in the virtual time interval (0, r].We define the virtual throughput of the process E S as This "i is the rate in virtual time of true events passing through process i.We assume this limit exists and is a positive constant for each process i. Keep in mind that the throughput rate h is a parameter of the simulated system S while v is a parameter of the simulation. The relevant parameters of the simulation are as follows.We shall view the parallel simulation of the system S also as a discrete-event system and denote it by S. The two systems S and S have the same network structures and routing probabilities for the true events, but the systems run on different time scales.Another difference is that S may contain synchronizing events while S does not.Let Ni(t denote the number of true events simulated at process in the real time interval (0, t].(The bar over a parameter denotes that it is associated with the simulation S).This quantity is related to the variables of the system S by (2) In other words, the simulation's cumulative work stochastic process N is a random time transformation of the system's throughput process N i.This is a fundamental re- lation for parallel simulations that allows one to express performance parameters of the simulation in terms of parameters of the system being simulated.Although this time transformation is implicit in some studies, we have not seen it exploited as expli- citly as we do here.The rate of the stochastic process N is called the simulation pro- cessing rate or execution rate of process i.This rate exists and is given by ilim t-li(t) vi/i. (3) This expression follows since, from the existence of v and "i, we have lim T-li(t) lim t-1Ti(t)Ti(t 1Ni(Ti(t)) =t--,lim t-1Ti(t tli_,rnTi(t)-1Ni(Ti(t)) The last limit uses the facts that Ti(t)c as t-cxz since v > 0 exists.
The following is a rather natural conservation principle for the simulation describ- ed above.
Definition 1" The simulation satisfies a virtual-time conservation principle if v vj for each i, j E S (the virtual speeds of all of its processes are equal).
This principle says that the long-run average virtual speeds are equal, although in the short run they may vary considerably.To see what type of simulation satisfies this principle, consider the simulation's global virtual time GVT(t) min Ti(t).l<i<n This minimum of the processes' virtual times at the simulation time t is a measure of the simulation's progress.Typically, the GVT(t) is nondecreasing in t and an event that is executed at a virtual time less than the GVT is committed or put in the fossil collection forming the simulation output; such events are never rolled back.The following is another notion related to the conservation principle.Definition 2: The simulation described above is linearly synchronous if t-l[Ti(t -GVT(t)]-O, as t--,c, for each E S. (4) This says that the virtual times never get too far away from each other their derivation is of the order o(t) on the time interval [0, t].This is satisfied, for instance, when the deviation of the virtual times is bounded by some constant; this may even be a natural or desired constraint built into the simulation protocol.Note that at the termination of a simulation, all virtual times are equal (otherwise, the simulation would not be complete).This suggests that for an infinite time horizon representa- tion of a simulation, the virtual times should be relatively close to each other for a large t (as in a linearly synchronous system).
The following is a characterization of the conservation principle.It says, in parti- cular, that a system satisfies this principle if and only if it is linearly synchronous.
Theorem 3: The following statements are equivalent.
(a) The simulation satisfies the virtual-time conservation principle.
a.u i, j (C) "A/,A--,B/,B, for any subsets A,B of S.
(d) The simulation is linearly synchronous.Proof: The equivalence of (a) and (b) follows directly from v -.i/Ai, which was noted in (3).Clearly (b) is a special case of (c), and, (c) follows from (b) by using the fact that , e A)i-e AVii and v -vj.Finally, from the definition v limt__,oot-Ti(T), we have t-1GVT(t) mint-1Tj(t) minvj.
Using these limits in (4), it follows that (d) is equivalent to v -minjvj, for each process i, which in turn is equivalent to (a).
Statement (c) in Theorem 3 relates the processing rates of a linearly synchronous simulation to the throughputs of the system.The equivalence of (b) and (c) says that the equality in (b) for any pair of processes also applies to any pair of subsets of S. We will use these relations in the next section to derive performance bounds for simulations.
If the simulated system S is a closed system, it may be easier to express the simula- tion's processing rate in terms of visit ratios.Assume that there are a fixed number of true events that circulate in the closed system S.The visit ratio Pi of a process E S is the average number of visits a true event makes to the process between successive visits to process 1 (an arbitrary fixed process).For a large class of systems with Markovian routing of events, the visit ratios satisfy the equations PJ-E PiPij' J G S, (5) where Pl-1 and Pij is the routing probability that a (true) event in the system S moves from process to process j.For these systems, the throughputs are related to visit ratios by li/Pi-j/Pj" The following immediate consequence of Theorem 3 is a relation between the processing rates of the simulation and the visit ratios.
Corollary 4: If the simulation is linearly synchronous, then AA/PA-/B/PB, for any subsets A and B of S.

Upper Bounds of Simulation Speed
In this section we discuss upper bounds on the simulation speed under several process- or-sharing scheduling strategies that are suggested for parallel simulations.We shall consider a parallel simulation of a linearly synchronous system as describ- ed above.The system consists of n logical processes that are simulated by K process- ors.We will study the following strategies for assigning processors to the processes: Autonomous Processor Assignments: The set of processes S is partitioned into K subsets S1,...,S K and processor k is assigned to simulate the events associated with the subset of processes S k.Note that K cannot exceed the number of processes n.
The special case in which each S k is a single process is called single-processor assign- ments.
Group Processor Assignments: The set of processes S is partitioned into M sub- sets S1,...,S M and the set of processors G-{1,...,K} is partitioned into M subsets or groups G1,...,G M. The group G. m is assigned to simulate the events for the set Sm.The idle processors of G m reside in a pool and whenever a set of events need exe- cution and the pool is not empty, then one processor is taken from the pool, and it executes one event in the event set with the smallest timestamp.The special case in which all K processors are assigned to all the processes (M-1) is called global pro- cessor assignments.
To measure the efficiency or quality of the simulation, we will use the total pro- cessing rate A S ] n 1Aj of the true events in the simulation.Recall that A S y= 1Aj is the total throughput of the system S being simulated.An important parameter affecting the simulation processing rate is the execution rate #k of process- or k, which is the number of events that processor k can process continuously, includ- ing any overhead time.The #k is therefore the maximal processing rate of processor k or a bound on the rate of events processed by that processor.We also write #B-E #k, B C G. kB The following results give upper bounds for the simulation processing rate under the processor scheduling strategies described above.
Theorem 5: Under autonomous processor assignments As -< min{#a, In particular, S < min{#G, AscG min Aj-1}. (9) Theorem 5 clearly reduces to the following when all K processors used in the simu- lation are homogeneous (here a a #).where G. m denotes the size of the set Gm.
lmarks: (a) The inequalities in the preceding results are tight there are elementary examples of systems in which the processing rate s equals the specified upper bounds.Furthermore A S is apparently fairly close to these bounds for standard examples, as the next section illustrates.
(b) In the bounds on A S in ( 6), ( 7) and ( 9), the first term just states the obvious property that this rate is limited by the maximal processing rate #a of K processors.
The second terms in these bounds, however, are more interesting.They reveal how the processing rate may be limited by the system parameters.
(c) Be mindful that the bounds do not consider idleness of processors that may be experienced, for instance, when a large subgroup of processors serves a small number of processors.In these cases, one should decrease #k to account for idleness.A rea- sonable compensation for idleness would be to multiply aGm by ,Sm/].tGmwhich re- presents the "traffic intensity" of events or the fraction of time that processors in G m are busy (necessarily ASm <_ #Gin, otherwise the simulation would be unstable).This compensation may be good for space-division protocols, but it may not be needed for more efficient time and space division protocols that typically have small processor idleness.
(d) For closed networks as discussed in Corollary 4, the upper bounds on A s in the preceding results have obvious analogues in terms of visit ratios; just replace Aj by pj.
The major interest in the upper bounds of the simulation processing rate is that they give insight into the maximum number of processors that can be usefully em- ployed in the simulation.To describe this, we say that K* is the maximum effective number of processors for the simulation if its processing rate A s may be constrained by the system as the number of processors K exceeds K*.For convenience, one might want to assume that the processors are ordered such that #1-> #2 >-It follows that the K* is the smallest value of K for which the "processor constraint" #G, for all larger K, exceeds the "system constraint" as represented by the second terms in the bounds above.The following is a formal statement of this.
Corollary 7: Under autonomous processor assignments In particular, for global processor assignments of homogeneous processors, the K* is -1 the smallest integer greater than min <_ j <_ nAj We now prove the main result.
Proof of Theorem 5: First, suppose the simulation uses autonomous processor assignments in which processor k is assigned to the set of processes S k.The simula- tion's processing rate on any set of processes cannot exceed the maximum execution rate for that set; otherwise, the simulation would be unstable.Consequently, As k -< #k and AS -< #a" (12) Next, note that the linearly synchronous property of the simulation implies by Theo- rem 3 that A S ASAsk/Ask for any k.This equality and the first inequality in (12)   yield A S Asn<,nKISk/ASk <_ Asn<,nK#k/,\Sk.(13) Combining this and the last inequality in (12), we obtain the assertion (6).
Next, suppose the simulation uses group processor assignments with processor set G m assigned to the set of processes Sm.By Theorem 3, we have, similarly to the equality part of (13), As-AS min A /AS ( 14) min ( 15) Now the portion of events for the processes in S m that are executed by processor k is k/am.Then the average maximum execution rate for any event from S m is kGm m (recall (8)).Consequently, similarly to (12), we have Aj <_ c a for each j E S m.
m Applying this inequality to (15) and using A S _< #a (as in ( 12)) yields m m min A-1}.
AS <-Asrnrn_nM{min{#am/ASrn Carn j e S m (16) Combining this with A S _< #a yields the assertion (7).Finally, note that the inequali- ty (9) for global processor assignments is the special case of ( 7) with M 1.

E! 4. Numerical Example
Our experience with numerical examples of Time Warp parallel simulations of queue- ing networks indicates that the upper bounds on simulation rates in the last section are fairly close to the actual ones.This is illustrated by the following example.
We considered the parallel simulation of a closed queueing network with eight sin- gle-server stations as shown in Figure 1.Each station has a general service time dis- tribution with a common service rate and the service discipline is first-in-first-out.The customers moving along the stations are homogeneous and they move indepen- dently according to the probabilities shown on the arcs.We ran several simulations of the queueing network for which the number of customers was set at 8, 32, 64, 256, 512 and 1024.The efficiency of the simulation is measured by its speedup defined as the ratio AS/# (the simulation rate per unit of processing rate).According to Corollary 6, S/# < min{8 S minAj-1}, j_<8 Graphs of the actual simulation speedup and the preceding upper bound, as functions of the number of customers, are shown in Figure 2 below.Note that the actual speed- up is very close to the bound.Although the speedup is not close to the maximum 8, its value near 3.4 is reasonable.In this case, the speedup bound is the constant Asminl < J < 8mina < J < 8Aj-1_ 3.45.This bound does not vary with the number of _custmer-s_ i the ngtwBrk.Indeed, if A were another set of rates, then }/ We used an aggressive Time Warp parallel protocol with preemptive rollbacks to simulate the queueing network.The simulation consisted of eight processes representing the eight respective service stations.The main events of a process contain the arrival, service and departure times of customers at the node it represents.These pro- cesses were simulated on the KSR-1 parallel computer by eight homogeneous process- ors under single processor assignments.Upon finishing the processing of an event, a processor typically sends one or more events to other processors before it can execute another event.The processing time therefore includes communication overhead, which is a major cost for parallel processing.In this section, we use the simulation rate bounds to gain insight into efficient assign- ments of processors to processes.
For a linearly synchronous simulation under group processor assignments, the pro- cessing rate bound in Theorem 5 is where A S _ min(#a,sUM}, U M min M{#Gm/ASm, aG min A-1}. (18) l_m_ m jES m J For those simulations, as in the last section, in which the inequality ( 17) is close to being an equality, one could achieve the highest processing rate A S by maximizing U M. In this regard, the following optimal processor assignment problem may be of interest.
Suppose the rates 1," ", An and #1," ", #K are fixed and one can vary the processor assignments.Then one could ensure a maximal processing rate s by choosing the integer M and sets S1,...,S M and G1,...,G M that maximize U M. This optional pro- cessor assignment can be obtained by solving the following optimization problem suc- cessively for M-1..., min{ K n }.
Assume that M is a fixed integer.We can write where X;m 1 or 0 according as j is or is not in Sm, and Ykm 1 or 0 according as k is or is not in G m. Similarly, Then from its definition (18), U M is maximized by the following mathematical pro- In this nonlinear programming problem, the variable x 0 plays the role of U M and the two inequality constraints just guarantee that x 0 does not exceed the two terms on the right side of (18).The essence of the problem is to find the largest value of x 0 for which there are feasible Xjm and Ykm" One could solve this problem by fixing x 0 at various values and running a standard linear integer programming package, for each fixed x0, to find whether there are feasible X jm and Ykm (one can linearize the constraint (19) by introducing new variables in the usual way).Note that for the case in which the processors are homogeneous, one can replace the Ykm with the variables kl,...,k M that denote the respective numbers of processors assigned to S1,...,SM, where k 1 /... + k M K. Then the two main constraints reduce to XO E "jXjrnkin#' XO'jXjrn -#" 3--1 Finally, to optimize the M, one would solve the preceding problem successively for M-1,...,min{ K, n }.
Although the processor assignment optimization problem we just discussed is rather narrow, it may be useful for gleaning a little more speed from a simulation.Other ways to substantially increase the speed might involve varying the processing rates, assigning processors dynamically depending on the state of the simulation and using more efficient time and space divisions techniques.These approaches would lead to interesting optimization problems, but they would require more intricate details of the simulation structure than we have been discussing.
Corollary 6: Suppose all m processors are homogeneous with common execution rate #.Then under autonomous processor assignments, s < # min { g As min A # min {K, AS <_ m <_ ASm' j e S m