Game Theoretical Method for Anomaly-Based Intrusion Detection

,


Introduction
Nowadays, network devices and communication services are vulnerable to various kinds of intrusion attacks, such as DoS/ DDoS, false data injection, and botnet attacks. e intrusion attacks tend to be more intelligent and the unexpected attack modes arise frequently. Consequently, great challenges are brought into network security control and management. As one of the most important techniques to tackle with various attacks, anomaly-based intrusion detection system (A-IDS) has been widely adopted in almost all kinds of network environments [1,2]. An anomaly-based intrusion detector attempts to estimate the normal behavior with a profile and generates an anomaly alarm once the profile collected from real-time observation exceeds a predefined threshold [2].
In an intrusion detection system, the attacker and defender can naturally be regarded as two players who try to maximize their payoffs, respectively, by executing certain optimal strategies. us, the game theoretical method is an effective tool which enables a defender to earn the maximum payoff (or the minimum loss) while fighting with the attacks. A number of results on game theory-based intrusion detection methods have been reported for different network environments and security requirements. Excellent surveys about this topic can be found in [3][4][5][6]. In [7], two-player noncooperative strategic game models are established for some general intrusion detection problems and Nash equilibriums are analyzed explicitly. In [8][9][10][11], game theoretical intrusion detection methods are investigated to solve the security resource allocation problems of large-scale heterogeneous networks. Note that, in [8][9][10], it is assumed that the defender scan always correctly identify the malicious behaviors of the attackers without any errors, while such an assumption may not be satisfied in some cases. For example, for intelligent APT attacks, the attackers often disguise themselves as no attack happens, which may make the detector to not always preciously identify the malicious actions. To handle these uncertainties, Bayesian games are considered in intrusion detection by updating the defender's belief to her/his opponent based on the past behaviors [12][13][14][15]. e main idea of Bayesian game-based intrusion detection is to use probability to represent the uncertainties and further use Bayesian iteration to update the dynamics. For self-organizing ad hoc networks, some nodes may be malicious and how to detect the malicious actions is an important work. Some strategic games are presented to stimulate the cooperation among distinct regular nodes, based on which the hidden malicious nodes can be detected [16][17][18][19][20][21]. In [22], a two-player Stackelberg stochastic game is analyzed for achieving the best response against the intrusion. In [23,24], game theory-based analysis methods for distributed intrusion detection are proposed, where consensus-based distributed detection method is presented and then game analysis is provided for the optimal defense and attack strategies. In [25], the privacy defense problem is also considered in the collaborative security scheme design problem by using the game theoretical analysis method. In [26], a differential game model is established to analyze the dynamic process of the attack and defense.
In a game between an attacker and a defender, the rational attacker will not launch an attack otherwise she/he can get a positive payoff. Moreover, the attack intensity needs be chosen to maximize her/his positive payoff. On the contrary, the defender will perform a defense action to resist the attack according to a similar rule. In an A-IDS, a predefined detection threshold needs be cautiously determined. In general, a higher threshold with a larger normal coverage area will result in a smaller false alarm rate but a larger missing alarm rate. Note that the missing alarm rate is also closely related to the attack intensity. More specifically, larger attack intensity will cause a lower missing alarm rate. ough attack intensity and detection threshold are two important factors affecting the false and missing alarm rates, which correspond to the payoffs of attackers and defenders in an intrusion detection game, they are seldom considered in the aforementioned results. In most of the aforementioned works, the false and missing alarm rates are assumed to be known constants and only binary actions "do" or "not do" are considered in their game models. In [11], the detection threshold and attack intensity are considered, while the focus is mainly related to distributed resource allocation of the heterogeneous networks.
Motivated by the limitations mentioned above in the literature, a more realistic two-stage form of the intrusion detection game model is presented in this paper. e attack intensity and detection threshold are considered as two strategic variables. In the first stage, the attackers and defenders make decisions on whether the attack and defense actions should be executed, respectively. Once the attack/ monitoring actions are decided to be executed, optimal attack intensity and detection threshold are determined to maximize their utilities in the second stage. e existence and uniqueness of the Nash equilibrium are discussed for the first stage of our presented game model under different scenarios, when the strategic variables of the second stage are restricted to certain regions. en, the optimal attack intensity and detection threshold are derived for each scenario, correspondingly. e contributions of this paper can be summarized as follows: (1) A two-stage game model is presented for anomalybased intrusion detection confrontation. In contrast to the existing work, where only binary actions "do" or "not do" are considered in the game model, the attack intensity and the detection threshold are considered as two key strategic variables, and the false and missing alarm rates are the functions of the attack intensity and the detection threshold, instead of being assumed to be constant. e two stages of the game model are tightly coupled with each other and thus the game model is more complex. (2) e existence, uniqueness, and calculation of Nash equilibriums are discussed. Based on the results, optimal selections of attack intensity and detection threshold for achieving the maximum payoffs of the attackers and defenders are provided. e results provide a new method to determine the detection threshold in the defense, from the perspectives of the optimization and confrontation. So, the presented game model and Nash equilibrium solution give a more realistic theoretical analysis framework for the anomaly-based security detection. e rest of this paper is organized as follows. In Section 2, some definitions are introduced and a two-stage game model of the A-IDS is presented. In Section 3, the Nash equilibrium of the proposed game model is analyzed. Simulation results are given to show the effectiveness of our game theoretical analysis methods in Section 4, followed by the conclusions of the paper summarized in Section 5.

A Two-Stage Intrusion Detection Game Model
Suppose that there is a network unit vulnerable to intrusion attacks. Typical examples for such a unit include a software system, network equipment, and a communication channel.
Here, we adopt similar attack and A-IDS detection models as that in [11]. e strategic form of two-player noncooperative game is given in Table 1. U A and U D denote the payoffs of the attacker and the defender, respectively. In the following, we give the physical meanings of the corresponding variables in Table 1. e variable x denotes the attack intensity, for example, the number of attack packets in a DoS/DDoS attack, or the number of bogus packets in a DNS cache poisoning attack or jamming strength in a communication attack, or the magnitude of false data injection. It is assumed that x ∈ [x, x], where 1 ≥ x > x > 0. e function s(x) ∈ R is used to represent the extent of damage to the security of the unit, when it is suffered from an attack with intensity x. It is natural to consider s(x) as a strictly increasing function such that (zs(x)/zx) > 0 and s(x) ∈ [s, s] with 1 ≥ s > s > 0. e term c 1 W + u(x)W, where c 1 ∈ (0, 1) is a constant, W is the security asset of the unit, and u(x) is a strictly increasing function, denotes the cost of launching the attack. e variable y denotes the detection threshold. It is assumed that y ∈ (y, y) with y > y > 0 and a larger y corresponds to a larger coverage area for normal behavior. e function p denotes the false alarm rate, i.e., it represents the probability that an alarm is generated though no attack is activated. Obviously, p is determined completely by the threshold y and p(y) is a strictly decreasing function in this paper. e function q denotes the missing alarm rate, i.e., it represents the probability that no alarm is generated though an attack is executed. e function q is determined by both attack intensity x and threshold y. It can be easily derived that q is strictly decreasing and increasing with respect to x and y, respectively. e parameters c 2 ∈ (0, 1) and c 3 ∈ (0, 1) are two constants.
Clearly, the game model described in Table 1 contains the following two stages. In the first stage, the optimal strategy set "Attack/Not attack" and "Monitor/Not monitor" needs be determined by the attacker and defender. en, both players proceed to the next stage to select optimal attack intensity x and detection threshold y. For better understanding, the two-stage pure-strategic intrusion detection game model with one attacker and one defender is described in Table 2 in a more rigorous way.

Remark 1.
e attack and detection models are similar to that in [11], while the results of [11] mainly consider the attack and defense resource allocation problem for heterogeneous distributed networks. In this paper, we consider the confrontation problem for one network unit, as expressed by the game model in Table 2. us, it is essentially different from the work in [11]. Besides, we establish a twostage game model by considering the attack intensity and detection threshold as the key strategic variables, which is also different from the existing works.

Nash Equilibrium Analysis of the Game
As mentioned in Section 2, the attacker/defender needs to decide whether to launch an attack/to monitor the unit or keep silence in the first stage of the presented game model. For simplicity, an extra assumption is imposed that if the payoffs of a player choosing to perform the action and to keep silence are the same, she/he will keep silence. In other words, the attacker/defender tends to do nothing if she/he cannot earn larger payoffs by launching an attack/monitoring. Note that the value of W has no impact on the analysis of Nash equilibrium (hereinafter referred to as NE) of the game from Table 1. us, without loss of generality, we set W � 1.
Denote the feasible set of x and y by π with π ∈ [x, x] × [y, y]. For convenience in later analysis, π is divided into the following subsets: (1) It can be readily shown that π 1 ∪ π 2 ∪ π 3 ∪ π 4 � π and (π 1 ∪ π 2 ∪ π 3 ) ∩ π 4 � ∅. e results of NE for the game as described in Tables 1 and 2 will be obtained from the following scenarios. In Scenario L.1, only one subset of π 1 , π 2 , π 3 , and π 4 is nonempty. In Scenarios L.2∼L.5, π 4 is empty while at least two subsets of π 1 , π 2 , and π 3 are nonempty. In Scenario L.6, π 4 and at least one subset of π 1 , π 2 , and π 3 are nonempty. Clearly, there is no overlap between any two scenarios and the six scenarios include all the possibilities. In the following, the sufficient and necessary conditions on x and y for the existence and uniqueness of NE are first derived for Scenarios L.1∼L.6, respectively. en, the optimal values of x and y, denoted by x * and y * , are further provided.
For convenient expression in what follows, two variables x ′ and x ″ are first defined, i.e., e optimization problems presented by (2) and (3) can be solved by classical optimization methods such as the gradient method and Lagrangian multiplier method [27].
e following conclusions can be drawn.
Theorem 1. In Scenario L.1, the NE of the game, as described in Table 1, is derived as follows: (1) If only the subset π 1 ≠ ∅, "not attack, not monitor" is the unique NE (2) If only the subset π 2 ≠ ∅, "attack, not monitor" is the unique NE and x * � x ″ (3) If only the subset π 3 ≠ ∅, "attack, monitor" is the unique NE and x * � x ′ , y * � y (4) If only the subset π 4 ≠ ∅, no NE exists Proof. Firstly, the strategy combination "attack, not monitor" will not be the NE.
is is because, − p(y)c 3 − c 2 < 0, the defender tends to "not monitor" the unit to earn zero payoff: is indicates that the attacker has no incentive to launch an attack either. erefore, "not attack, not monitor" is the unique NE.
(2) If only π 2 ≠ ∅, as the payoff of the attacker s(x) − c 1 − u(x) is positive for any attack intensity x, the attacker will select "attack." Besides, the defender will never get more payoffs when she/he selects "moni- for an arbitrary threshold y. us, the defender will select "monitor." e optimal attack intensity x * should be derived by maximizing the payoff of the attack; therefore, x * � x ″ based on (3).
(3) If only π 3 ≠ ∅, the attacker will always select "attack." is is because for any attack intensity x and detection threshold y the payoff of the attacker satisfies for an arbitrary y, the defender will select "monitor." en, for the defender, the optimal threshold is computed by Based on the property that (zq(x, y)/zy) > 0 in Table 2, we have y * � y. en, the optimal attack intensity is given by "not attack, not monitor" cannot be the NE, either.
Combining with the result derived in the beginning that "not attack, monitor" cannot be the NE, it is concluded that no NE exists.
Remark 2. From eorem 1, the payoffs of the two players are, respectively, expressed as (2) (2) in Scenario L.1, the defender compensates for part of the loss by executing monitoring action in this scenario as q < 1. us, the payoff earned by the attacker decreases. As discussed previously, Scenarios L.2∼L.5 cover the possibilities that π 4 is empty while at least two subsets of π 1 , π 2 , and π 3 are nonempty. Details are given as below.
e following results about the NE for this scenario can be shown.

Theorem 2.
In Scenario L.2, the strategy combination "attack, not monitor" is the unique NE and x * � x ″ . Proof.
e subset π 2 ≠ ∅ indicates that there exists an x such that the payoff of the attacker s(x) − u(x) − c 1 is positive. us, the attacker will select the strategy "attack." Besides, the payoff of the defender satisfies − q(x, y)s(x) − c 2 ≤ − s(x) for any threshold y, so the defender will select "not monitor." Besides, the optimal attack intensity is given by  Table 1) Game target e players choose their strategies to maximize their payoffs U A , U D
Main results for this scenario are formally stated in the following theorem.

Theorem 3. In Scenario L.3, the strategy combination "attack, monitor" is the unique NE if and only if
e optimal attack intensity and detection threshold are x * � x ′ and y * � y.
Proof. Necessity: if "attack, monitor" is the unique NE, then from (2) and (4), there are x * � x ′ and y * � y. e payoff of the attacker with x * and y * must be positive; thus, , the attacker can earn a positive maximum payoff if the defender selects the strategy "monitor" and y * � y. us, the attacker will select to "attack" and y * � y. As q < 1 and It follows that x ′ ∉ π 1 and (x ′ , y) ∈ π 3 for y ∈ [y, y]. From the definition of π 3 , it can be concluded . is indicates that no matter how the threshold is selected, the defender will earn larger payoff when she/he selects the strategy "monitor" rather than "not monitor." Clearly, the defender will select "monitor" and the optimal threshold is set as y * � y from (4). erefore, the strategy combination "attack, monitor" is the unique NE and x * � x ′ and y * � y. □ Scenario L.4. π 2 ≠ ∅, π 3 ≠ ∅, and π 1 � π 4 � ∅.
e following conclusions can be drawn for this scenario. (1) Necessity: under the strategy combination "attack, not monitor", the attacker will select x ″ as the optimal attack intensity.
, the defender will select "monitor" to earn larger payoffs, which is a contradiction to the premise that "attack, not monitor" is the NE. us, the necessity is shown. Sufficiency: from the definitions of π 2 and π 3 , the attacker can always earn positive maximum payoff when s/he selects "attack." As zq/zy > 0, there is is means when the attacker selects x * � x ″ , the defender never earn larger payoffs than she/he does nothing no matter how the threshold is set. us, "attack, not monitor" is the NE and x * � x ″ . e sufficiency is shown.
(2) Necessity: under the strategy combination "attack, monitor," the defender and attacker will select y and x ′ as the optimal detection threshold and attack intensity from (4) and (2).
is means the defender never earns larger payoffs than she/he does nothing, which is a contradiction to the premise that "attack, monitor" is the NE. us, the necessity is shown.
Sufficiency: the attacker always selects "attack" from the definitions of π 2 and π 3 . If the attacker selects , the defender will select "monitor" to obtain larger payoffs than "not monitor" and the optimal detection threshold is y from (4). Meanwhile, when the defender selects "monitor" and y * � y, from (2), the attack will select "attack" and x * � x ′ to earn the maximum positive payoff. us, the sufficiency is shown.

Corollary 1. In Scenario L.4, (1) If and only if
Different from Scenario L.4, there exists x ∈ [x, x] belonging to π 1 such that s(x) − c 1 − u(x) ≤ 0. Since the attacker can always find an x such that she/he earns a positive payoff, the strategy combination "not attack, not monitor" Security and Communication Networks 5 cannot be the NE in this scenario. e main results about the NE in this scenario can be formally stated in the following theorem.
Theorem 5. In Scenario L.5, , "attack, monitor" is the unique NE and x * � x ′ and y * � y

Proof
(1) e proof is similar to that of (1) in eorem 4 and is omitted here.
belonging to π 1 such that as q(x, y) < 1. us, compared to (2) in eorem 4, an extra condition q(x ′ , y)s(x ′ ) − c 1 − u(x ′ ) > 0 needs be added to ensure that "attack, monitor" still be the NE. e remaining proof is similar to that of (2) in eorem 4 and is omitted here.
(3) and (4) By following similar analysis in the proof of Corollary 1, the uniqueness of the NE in this case can also be concluded.
Remark 4. In this paper, we assume that the attackers are completely rational, while this assumption may not be satisfied in some scenarios. However, based on our method, we present an optimal defense strategy for the worst case.
at is, we can guarantee that the maximum damage in the worst case can be minimized by our method.

Simulation Studies
In this section, simulation results are provided to validate the theoretical results as presented above. In A-IDS, a profile is generally selected to cause distinctions between normal and abnormal states. Such a profile is normally described by a random variable in many cases. Here, we assume it follows a Gaussian distribution with zero mean under normal states. Similar assumptions can be seen in many intrusion detection application areas such as network traffic detection and Kalman filtering-based anomaly detection. Let the intensity of the attack be denoted as x. Other parameters in simulation are chosen as x � y � 0.1, x � y � 2, s � 0.5x, u � 0.1x, and c 3 � 0.2. e false alarm rate and missing alarm rate can be expressed by respectively. Parameters c 1 and c 2 are used to represent the costs of the attacker and the defender, respectively.

Case 1.
We first select c 1 ∈ [0, 0.2] and c 2 � 0.2. en, it can be calculated by (1) that (a) If c 1 ∈ [0, 0.04], there are π 2 ≠ ∅, π 3 ≠ ∅, and π 1 � π 4 � ∅, which corresponds to Scenario L.4 (b) If c 1 ∈ [0.04, 0.08], there are π 1 ≠ ∅, π 2 ≠ ∅, π 3 ≠ ∅, and π 4 � ∅, which corresponds to Scenario L.5 (c) If c 1 ∈ [0.08, 0.2], all the four subsets are nonempty, which corresponds to Scenario L.6 en, it can be checked whether the inequality conditions in eorems 4 and 5 and (4) in eorem 6 are satisfied for the above three scenarios, as given in Table 3. 'IC 4.1', 'IC 4.2', 'IC 5.1', and 'IC 5.2' refer to the inequality conditions in (1) and (2) in eorem 4 and (1) and (2) eorem 5, respectively. It is worth noting that the inequality conditions in (4) in eorem 6 are the same as those in eorem 5. From the theoretical analysis given in Section 2, the following conclusions on the NEs can be drawn: (a) Based on (2)  Similarly, Table 4 is given to show whether the inequality conditions in eorems 3 and 5 are satisfied, where 'IC 3' refers to the inequality condition in eorem 3. en, the following conclusions on the NEs can be drawn:  Figure 2. Clearly, the defender loses some security asset as U D < 0. Moreover, the lost security asset will increase as the defense cost c 2 increases.
At last, we make some comparisons with the existing methods in [7][8][9][10][11][12][13][14][15], where attack intensity and detection threshold are scarcely considered and majority of them assume that the false and missing alarm rates, and the game model of detection problem can be modelled as Table 5.
It can be seen that, without considering the attack intensity and detection threshold, the payoffs of the game model will be reduced to be constant and the Nash equilibrium analysis can be easily done. From the definition of the Nash equilibrium, it can be calculated that if q + c 2 > 1, (Attack, Monitor) will be the unique NE.
ough the existing analysis methods in [7][8][9][10][11][12][13][14][15] can determine the optimal action strategies, while our results can further determine the optimal explicit attack intensity and detection threshold, different results can be obtained. First, the existing work considers only the strategy do or not do; thus, the one-stage game model, as expressed in Table 3, is established to help analyze the optimal actions, while we further consider the attack intensity and detection threshold in the game model, as these two parameters are two key strategies used for the defender and the attacker. Moreover, we establish a more detailed two-stage game model to consider both the action do or not do and the attack intensity Security and Communication Networks 7 and detection threshold. Based on the experimental results, we can see that the attack intensity and detection threshold play an important role in the determination of the Nash equilibrium. Intuitively, for the game in Table 3, the NE are completely determined by the parameter x and y; however, this conclusion seems not to make sense as the false alarm rate and other parameters have no any effect on the Nash equilibrium. Alternatively, for our game model, we can see that all parameters will jointly determine the Nash equilibriumthus, our analysis results are more realistic. In practical, the false and missing alarm rates are not constant, as the attacks are always dynamically changing. In A-IDS methods, the false and missing alarm rates are commonly determined by the attack intensity and detection threshold. Our method just considers this real scenario and establishes a more explicit game model, based on which the optimal strategies are completely determined.

Conclusion
For anomaly-based intrusion detection system, we present a game theoretical analysis method to provide the optimal strategies. We first establish a more realistic game model by considering the attack intensity and detection threshold as two strategies for the players. e necessary and sufficient conditions, for which strategies are the Nash equilibriums, are presented. Simulation studies are provided to validate our theoretical results. e results provide a new method to determine the detection threshold in the security defense. In the future, some more research work could be considered, for example, the game theoretical analysis method for

Monitor
Not monitor Attack Security and Communication Networks specific scenarios such as Internet of ings and DoS/DDoS attacks. Besides, dynamic game analysis is also an interesting topic for dynamic security confrontation process, for example, Stackelberg game analysis can be adopted to solve the sequential problem of the attack and defense actions.
Data Availability e manuscripts of game theory algorithm in this article are from the databases of Cambridge University and Columbia University. Copies of these data can be obtained from https://dl.acm.org/doi/book/10.5555/1951874 and https://doi.org/10.1016/j.ins.2018.04.051.

Conflicts of Interest
e authors declared that they have no conflicts of interest.