Mining Experiential Patterns from Game-Logs of Board Game

and


Introduction
Gaming is one of the main intellections in our daily life.To some extent, gaming shows the wisdom of humankind more directly.Artificial intelligence for games (Game AI) is an important subfield of AI research.The current generations of computer games offer an interesting testbed for Game AI [1].Meanwhile, applying AI techniques to game design will make the games more interesting.The research of Game AI plays an important role in promoting the development of the computer game industry.
Board game is one of the earliest objects that AI focused on [2,3].By contrast with other types of games, board games reflect human's intelligence more purely and thus attract the interest of AI researchers [4,5].At first, researchers concentrated on the algorithm design about the chess road searching and the chess manual matching and made some important progresses.John von Neumann presented the Minimax algorithm in 1920s [6].Shannon [2] proposed the game tree searching algorithm with his chess program in 1950.Knuth and Moore deeply analyzed the alpha-beta pruning algorithm in 1958 [7].Then, studies on AI in board games were widely launched [8][9][10][11][12][13].Many contests of board games between machines and real persons showed the advances in AI research [14][15][16].Recently, researchers find that game-logs (game records) cover a lot of important and interesting knowledge.Chen et al. [17] proposed an approach to abstract expert knowledge from annotated Chinese chess game records.GO game records are used for the life and death predicting, the pattern acquisition, and the pattern matching [18][19][20].Esaki and Hashiyama [21] carried out some experiments to extract human players' strategies from the game records of Shogi.Weber and Mateas [22] used a data mining approach to process game-logs for the strategy prediction.Takeuchi et al. [23] presented a way of evaluating game tree search methods by game records.Moreover, Wender [24] expounded how to use data mining and machine learning techniques to analyze the game-logs in his project.
The common design of board games is to factitiously set the difficulty level on an invincible computer player, but this way is just a simulation.However, human players often expect a computer player to act and think as a real person rather than a superman.Game-logs in a board game are considered as cumulative experience, which will become richer and more effective with the increase of game times.With the accumulation of the experience, computer players can gradually enhance their intelligence level, just like human beings.Data mining techniques can be used to extract experience from game-logs.Nevertheless, there are few works on this topic.
In this paper, an approach is presented to mine experiential patterns from the growing game-logs of a sample board International Journal of Computer Games Technology game.The computer player can learn from these patterns to improve its intelligence naturally.Firstly, a Chinese checkers game was designed as a test platform for collecting game-logs and launching empirical studies.Then, series of algorithms based on a sequence-tree were proposed to mine experiential patterns from game-logs.These patterns included Experience Rules, Key States, and Checker Usage.Finally, the mined patterns were applied back to the test platform, and then a series of experimental tests was elaborated to verify the effectiveness and performance of our approach.
Experimental results demonstrate that our approach is effective and efficient and easy to be transplanted to fit for other types of games.

Methods
The application of the experiential pattern mining requires two basic conditions listed below.
(i) The computer players of a game have the minimum ability to play the game correctly without any experiential pattern.
Doing so ensures that the game can run properly in the initial stage.
(ii) There are sufficient and valid game-logs to make the pattern mining task runnable.
Only when the number of game-logs reaches a certain size can the regularity hidden in them appear.In addition, our purpose is to make computer players learn from human players' strategies.So, these game-logs should be from human players' games.
To satisfy the above conditions, we designed a simplified Chinese checkers game as a test platform to provide the basic move algorithms and collect the game-logs.Subsequent sections cover the following topics in detail: (a) the concepts concerned with the game and the mining algorithms; (b) the algorithms for mining experiential patterns; (c) the methods for applying experiential patterns to the game.

Key Concepts.
The traditional Chinese checkers game has flexible gameplay (see [25] for more details).In this work, the game was simplified to be more convenient for experimenting (the design details are provided in Appendix).
The key concepts used in the following sections are briefly explained below.
(i) Checker.The checker means the position used to put pieces on the chessboard.Each checker has its own coordinate.
(ii) Checkers-State.The checkers-state describes all the checkers on the chessboard and can be traversed to get or update the information of every checker.
(iii) Piece.Each player holds ten pieces.The coordinate of a piece is dependent on the current position of the piece.
(iv) Pieces-State.Pieces-states are used to store the coordinates of pieces.A pieces-state is a set that consists of several subsets.The number of the subsets depends on how many players are there in the game.Each subset contains ten elements, which correspond to the ten pieces of a player.For 2-player mode, there are two subsets, called the active side (SAS) and the passive side (SPS), respectively.
(v) Game-Log.During gaming, once a piece moves a step, the test platform will save the corresponding pieces-state to a game-log.The pieces-states in a game-log are sorted by their creation time.Hence, a game-log is a sequence of piecesstates.Separate XML files are used to store the game-logs.
( Key States are a group of particular pieces-states.They appear in victor games frequently and thus indicate some good game situation.By algorithm designing, one can help the computer player in approaching these Key States.Furthermore, approaching the Key States may also promote the applying of some Experience Rules, because some Key States may just be the last states or the next states.
Checker Usage describes the usage of a checker and is denoted by .The  of a checker can be calculated by where   represents the  of checker ,    holds the times that  has been used as the falling checker, and  stores the number of game-logs in database.

Mining Experiential Patterns
. By using some traditional move algorithms, the computer player may figure out the best move within several rounds.However, being subject to the computing capability of the computer and the requirement of game runtime, the computer player is hard to make more skillful moves or longer layouts.Humans improve their chess skill by accumulating the playing experience.A computer player can simulate this process to study experience from humans' excellent games and thus make the moves more strategic.The three experiential patterns introduced in Section 2.1 are mined from the game-logs generated in the games of human-human mode.The data mining process is run in server-side, and its workflow is shown in Figure 1.As shown in Figure 1, the data mining process consists of three phases, including data extraction, data mining, and data representation.

Data Extraction.
Each game makes a single game-log file, and this will generate lots of XML files stored in serverside.These XML files are not suitable to be the direct data source of data mining.Therefore, the sequential data with consistent format needs to be extracted from game-logs and stored in database first.
Three data entities, PiecesState, PiecesStateSequence, and CheckerUsage, are used to format the game-logs.Their attributes are, respectively, shown in Tables 1, 2, and 3.The pieces-states in PiecesState are distinct, and each of them has a unique ID.The sequences in PiecesStateSequence are  29)-(37) add a win/loss mark to the nondraw game and use minus to mark the pieces-states created by the loss player.Lines (38)-(41) combine all the PiecesState IDs from the same game into a single sequential string.

Data Mining.
Experiential pattern mining means finding useful patterns from a large number of sequences composed of pieces-states.The types of these patterns include but are not limited to those mentioned in Section 2.1.Mining the Experience Rules, which are the most important experiential patterns in this work, is an objective of sequential pattern mining.Sequential pattern mining is a topic of data mining concerned with finding statistically relevant patterns between data examples where the values are delivered in a sequence [26].In this field, many efforts of mining sequential patterns have been devoted to developing efficient algorithms, such as GSP [27], SPADE [28], CloSpan [29], PrefixSpan [30], and MEMISP [31].In recent years, researchers have paid more attention to the applied research of sequential pattern mining as discussed elsewhere [32][33][34][35][36].
In this section, we describe the methods for mining the three kinds of experiential patterns explained in Section 2.1.a set of items, where each item is a sequence of piecesstates.Given a minimum support threshold minSupport, the aim is to find out all the consecutive binary subsequences (CBSSs) whose frequency is not less than minSupport.Each CBSS consists of two pieces-states that are consecutive in the original sequence, and this can be regarded as a constraint in data mining.In this paper, an algorithm based on patterngrowth is proposed for mining CBSSs.This algorithm uses a sequence-tree (see Figure 2) as the data structure to load all the sequences in database.The sequence-tree is different from the FP-tree [37] of frequent pattern mining, and we do not have to scan database previously to obtain the supports of every pieces-state.Algorithm 2 describes the procedure of creating a sequence-tree.

Experience
In Figure 2, value stores the ID of the pieces-state, weight represents the support of the related CBSS, and next stores the linkage pointing to the next node.
On a certain player's perspective, if two same pieces-states appear, the moves between the two pieces-states will become noneffective interference data.Eliminating these data will reduce the unnecessary data analyses and improve the reliability of mining results (see lines (05)-( 13) in Algorithm 2).Not all the frequent CBSSs are good, since some of them may be from losing games.The support count of a CBSS denoted by suppCount can be calculated by where posSuppCount is the support count of the CBSS generated in winning games and negSuppCount in losing games.
By means of the win/loss mark, the suppCount of each CBSS has been calculated during the process of creating the sequence-tree.Hence, the weight of each branch is   equal to the suppCount of the relevant CBSS.Moreover, the same pairs of nodes are not allowed to appear on the same sequence-tree; otherwise the suppCount values will be incomputable (as illustrated in Figure 3).To solve this problem, the algorithm for shifting branches is designed (see Algorithm 3).
To obtain the frequent CBSSs, the following formula is given: where supp denotes the support of a CBSS and totalSeqs denotes the total number of the sequences in database.Given a minSupport, the frequent CBSSs are those whose supports are not less than minSupport.All the frequent CBSSs can be found out by traversing the sequence-tree only once, and the script to do so is shown in Algorithm 4. In Algorithm 4, the structure of the object SequentialPattern is shown in Figure 4.The found CBSSs will serve as the data source for selecting Experience Rules.Key State.Given a threshold percentage minFreq, Key States are those pieces-states of the frequencies not less than minFreq.The frequency (denoted by freq) of a pieces-state can be calculated by where  stores the total number of sequences in database and count denotes the occurrence number of the pieces-state in the net winning games.The value of count is figured out

Premise Result Support
The ID of pieces-state during the process of data extraction (see lines (17) and (23) in Algorithm 1).
In respect of threshold selection, adjusting minFreq dynamically in accordance with the number of Experience Rules can make the Key States more efficient.
Checker Usage.This pattern can be calculated by formula (1), in which the    is figured out during the process of data extraction (see line (12) in Algorithm 1).

Data Representation.
In this phase, the main task is to make the mined experiential patterns recognizable to the computer player.The experiential pattern here mainly refers to Experience Rule.The following format is used to represent an Experience Rule:  →  : . ( According to the data structure of the Experience Rule, the process of data representation can be described as follows.Replace the IDs, which denote the last states or next states of Experience Rules, with the corresponding pieces-states, and then write these pieces-states to an XML file in server-side.
However, the mined patterns may not be all reliable.For example, given a minSupport, the obtained Experience Rules are as follows: That is, one last state has multiple next states.When piecesstate   appears, it is clearly reliable to select the piecesstate   , which is of the highest support.Hence, the reliability selection should be done before data presentation.Algorithm 5 shows this process.

Applying Experiential Patterns.
The way of applying experiential patterns depends on the specific gameplay.In this section, we give a computer player's strategy as shown in Figure 5 and then explain how to apply the mined experiential patterns to the actual game.
In the course of the game, the computer player uses Experience Rules first.Algorithm 6 demonstrates the procedure of matching Experience Rules.
When there are no available Experience Rules, the computer player will automatically approximate a certain Key State.Doing so can promote the formation of some good game situation and may activate some Experience Rules.Obviously, the approached Key State is the nearest piecesstate after the current one.So, all of the Key States are sorted The concept of dissimilarity (denoted by Diss) is used to represent the closeness between two pieces-states.The Diss between   and   is equal to the least moves from   to   .The less the value of Diss, the closer the two pieces-states.For calculating Diss, the following definitions are given.Definition 2. Let  be a piece and let  be a set of piecesstates.If the coordinate of  is included in , then  belongs to , which can be expressed as  ∈ .Definition 3. Let   and   be two checkers, (  ,   ) and (  ,   ) their coordinates, respectively, and   +   ≥   +   .Then, the minimal number of steps for a move from   to   is called move distance from   to   and denoted by   , which can be calculated by where the function  is defined as Algorithm 7 demonstrates the calculation of the distance between two pieces.
Let   and   be two pieces-states.According to Definitions 2-4, the dissimilarity matrices between   and   are defined as follows: where DissMatrixAS is defined on the active side and DissMa-trixPS on the passive side.Then, the Diss between two pieces can be calculated by Read the experiential patterns. Server-side

Remote access
Scan chessboard to obtain pieces-states.

Begin
Is there any experientialrule?
Use the next state of experiential-rule to refresh chessboard.
Is there any Key State behind the current piecesstate?
Calculate the dissimilarity Diss between the current pieces-state and the Key State.
Is Diss more than a given threshold S?
Find out the pieces-state that cuts down Diss greatly, and use this pieces-state to refresh chessboard.
Save the current pieces-state to game-log.where   and   satisfy the following conditions, respectively:  It follows that the calculation of the Diss between pieces is a typical assignment problem in linear programming.This problem can be solved by Hungarian algorithm as discussed by Edmonds [38].

Is any condition of game over met
When there are no Key States behind the current piecesstate or the Diss between the nearest Key State and the current pieces-state is more than a given threshold, the computer player will execute a tentative move algorithm named Tenta-tiveMove, which was designed based on the traditional alphabeta pruning algorithm [7] (see Appendix for more details).Moreover, the pattern Checker Usage is one of the parameters of TentativeMove.

Results and Discussions
In order to evaluate the effect and performance of our proposed approach, we conducted an experiment on our designed test platform mentioned in Section 2. The experiment contains a series of tests.To maintain the objectivity of the experiment, we chose the same group of testers, who are ten experienced human players.

Test in Human-Computer Mode without any Experiential
Patterns.Each tester played against the computer player for 30 games, serving as offensive player (OffP) and defensive player (DefP) alternately.The computer player was only allowed to use TentativeMove using a default value of Checker Usage as the parameter.A total of 300 game-logs in this test were recorded.The test results are shown in Table 4.
Furthermore, we observed the performance of our algorithms by using the parameter ATMC (the average time for each move of the computer player).A script embedded into the program was used to calculate the response time of the computer player during testing and give the value of ATMC in the end.In this test, the value of ATMC was 5.85 s (s = seconds).

Descriptions of Effects. (a)
The computer player has the ability to play correctly; (b) the winning rate of the computer player in competition with every tester is not more than a half; (c) the computer player spends a little long time to move a piece.
Discussions.(a) Condition (i) mentioned at the beginning of Section 2 is satisfied by the TentativeMove algorithm; (b) the moves driven by TentativeMove may not be optimal due to the error of its evaluation function and the restraint of searching depth, which affects the winning rate of the computer player; (c) TentativeMove has a deeper recursion depth.As the game goes on, the checkers to be searched in each recursive layer will increase rapidly, which degrades the performance of TentativeMove.

Test in Human-Human Mode for Collecting Game-Logs.
Each tester played with all the others for 30 games, serving as OffP and DefP alternately.A total of 1350 game-logs in this test were recorded.This satisfied condition (ii) mentioned at the beginning of Section 2. Subsequently, additional tests in human-computer mode were performed to select the appropriate minSupport and minFreq.These tests were similar to those in Section 3.1, and the results are shown in Table 5.Finally, the experiential pattern mining program was executed with the selected minSupport and minFreq.As a result, we gained 528 Experience Rules and 575 Key States, and the Checker Usage of every checker was updated in real time.
As shown in Table 5, when minSupport was 2% and minFreq was 1%, the crest value of Auep was reached, which meant the mined experiential patterns were better utilized.

Test in Human-Computer Mode with Experiential Patterns.
The test plan here was the same as that in Section 3.1, except for the applying of the experiential patterns obtained in the test mentioned in Section 3.2.The results are shown in Table 6, and the comparison with the test results in Section 3.1 Winning rates of the computer player not using experiential pattern Winning rates of the computer player using experiential patterns is illustrated in Figure 6.In this test, the value of ATMC was 0.93 s. 6 shows that, by using experiential patterns, the computer player has increased its winning rates in most of the games at different degrees; (b) the average number of the rounds per game has increased; (c) the value of ATMC has declined dramatically.

Discussions. (a)
The effectiveness of experiential patterns will become more obvious with the increase of game-logs, and this will give the human players an impression that the computer player is learning from the experience; (b) in a few games such as 02-OffP and 07-DefP shown in Figure 6, although the computer player's winning rates are not increased, the average number of the rounds per game is greatly raised (see the lines with bold font shown in Table 6).The testers obviously feel the improvement of the computer player's chess skill; (c) if the Experience Rules exist, the computer player will match and use them first.Compared with TentativeMove, the pattern matching algorithm is of linear complexity and consumes less time (see Algorithm 6).

Test in Computer-Computer Mode.
In this test, we set two computer players.One could use the experiential patterns  and the other could not.They took turns to serve as OffP and played with each other for 30 games.The results are shown in Table 7.
Description of the Effect.The computer player using experience patterns has a great advantage in winning rate and ATMC.
Discussion.Experiential patterns can help the computer player in making more strategic moves; still, we are aware of four losing games of CP1 (as shown in Table 7).This indicates that some of the experiential patterns may not be really good.However, the invalid patterns will die out with the growth of game-logs.

Comparative Analysis on the Algorithms of Pattern Mining.
A sequence database consists of sequences of ordered elements, and these elements contain some unordered items.
The common objectives of the previous works [27][28][29][30][31] are to find the interesting relations between these items.In this work, however, the pattern mining algorithms aim at finding only the frequent CBSSs in the game-logs.This can be considered as a special case of the sequential pattern mining on a sequence database.For ease of comparison, we implemented the traditional algorithm PrefixSpan and added some restraints to make it fit for mining in our game-logs.These restraints were (a) setting length of each subsequence to 2 and (b) setting size of each element to 1.The comparative results are shown in Figure 7.
Here the minSupport was set to 2%, and the given data set contained 1,650 sequences (i.e., game-logs), 155,100 elements (i.e., pieces-states).Figure 7 indicates that our algorithms compared with PrefixSpan are more efficient.
3.6.Discussion on the Using of Game-Logs.Previous works [17][18][19][20][21][22][23][24] mainly discuss the methods of extracting expert knowledge from game records.The extracted knowledge is commonly used to construct a game agent of high intelligence.Differently, our work aims at helping a computer player in learning from growing game-logs, acquiring human's experience, and gradually becoming more skillful.It is a practical approach to make the game more fascinating.Furthermore, by satisfying the two conditions mentioned at the beginning of Section 2, any board game can utilize this approach after a minor adjustment.And this shows the extensibility of our approach.

Conclusions
In this paper, a novel approach is proposed to mine experiential patterns from the game-logs of board game.Those experiential patterns can be utilized to improve the intelligence of a computer player.This approach makes computer players learn from human's experience progressively during gaming and become more experienced with the increase of game-logs.This will make the game more interesting.We conducted an experiment on our designed test platform of Chinese checkers game, and the results demonstrate that our approach is effective, efficient, and extensible.Nevertheless, our approach still needs improvements from several aspects, such as the following: (i) designing algorithms to automatically adjust the parameters minSupport and minFreq; (ii) optimizing the related algorithms to improve their access performance for a huge sequence-tree; (iii) researching the fast matching problem in massive experiential patterns; (iv) developing the approach for online analysis of gamelogs; (v) looking for more experiential patterns making games interesting.

Appendix
(1) Rules and Data Structures.The rules and data structures used for programming are described as follows.
(i) Only use the 2-player mode.
(ii) Move rules: there are two move rules, shift and jump.
The shift means moving a single piece one step in any direction to an adjacent empty checker.The jump here means single-piece-jump, which is the simplest and most usual rule.An example of single-piece-jump is shown in Figure 8.
(iii) Checker: a computer can get the complete appearance of a game by scanning all the checkers on the chessboard.A checker has three properties including color, position, and status.The color stores a hexadecimal number to represent the color of the piece on the checker.The position stores the coordinate of the checker.The status uses the values of 1, −1, and 0 to describe who is on the checker.
(iv) Chessboard: for the convenience of programming, the chessboard is designed to fit for 2-player mode.That means the other four useless star corners on traditional chessboard are cut out (as illustrated in Figure 9).
We used a coordinate system to locate the positions on the chessboard.This coordinate system is unfixed and has an included angle of 60 ∘ .Players have their own frames of axes with the same form (as shown in Figure 10).This design has two advantages.First,    (i) Human-human mode: this mode is mainly used to collect the experience from humans.All the players in this mode are real persons.(ii) Computer-computer mode: this mode is only used for carrying out experiments.All the players in this mode are controlled by the computer programs.(iii) Human-computer mode: in this mode, a human player is allowed to play against the computer player, and the game-logs are also collected.This mode is in charge of effect analysis.
Taking the human-human mode as an example, the gameplay is shown in Figure 15.And the other two play modes can be designed in a similar way.
In the gameplay, there are three problems to be solved.
Problem 1.It is to find out all the falling checkers related to the specified starting checker.
Problem 2. It is to determine the legality of a move.
Problem 3. It is to make the computer player have the ability to move spontaneously.
both sides' players to move, and the evaluation values of every move are recorded.Then, the algorithm calculates the difference of the sum of evaluation values between its own side and the opponent's and selects the move with the maximum difference.Through testing, we set steps to 11 and obtained effect preferably.

Figure 2 :
Figure 2: Data structure of a node on the sequence-tree.

Figure 3 :
Figure 3: Branch shifting.The two pairs of nodes B-C marked with dotted circles on the left tree should be merged together, and their branch weights should be added up too.The right tree is the result of this branch shifting.

Figure 5 :
Figure 5: Strategy of the computer player.

Figure 6 :
Figure 6: Comparison of the winning rates of the computer player.

Figure 7 :
Figure 7: Performance of the two algorithms on a given data set.ErsMining denotes the set of algorithms, including Sequence-TreeCreation, BranchShifting, CBSSFinding, and ReliabilitySelection, for mining Experience Rules.

Figure 9 :
Figure 9: Chessboard.(a) The traditional chessboard.(b) The simplified chessboard without the useless star corners.

Figure 10 :
Figure 10: Unfixed coordinate system.The frames of axes in (a) and (b) are used by green and red player, respectively.

Figure 11 :Figure 15 :
Figure 11: Every checker has two coordinates due to different frames of axes.For example, checkers C1 in (a) and C2 in (b) have the same coordinate.The two coordinates of checker C2 are(7,5) and(1,3), respectively.
appears, then the player should achieve the piecesstate   by moving a piece.The rule can be represented as   →   , where   is called last state and   next state.
vi) Experiential Pattern.Experiential patterns are mined from game-logs and easily used by computer players.In this work, we considered three kinds of experiential patterns, including Experience Rule, Key State, and Checker Usage.An Experience Rule is similar to an if-then rule.If piecesstate

Table 1 :
Attributes of the PiecesState.

Table 2 :
Attributes of the PiecesStateSequence.

Table 3 :
Attributes of the CheckerUsage.
their occurrence time.Let   and   be two different pieces-states.Their order is defined below.Assume that   ̸ =   .If   ⋅ SAS ≥   ⋅ SAS and   ⋅ SPS ≥   ⋅ SPS, then   is behind   , which can be expressed as   ≥   . by

Table 4 :
Data records of the test in human-computer mode without any experiential pattern.
Note: Gts: game times; Role: role of the computer player; Wts: winning times of the computer player; Anrs: average number of the rounds per game; and Wr: winning rate of the computer player.

Table 5 :
Data records of the additional tests for selecting thresholds.A total of 1650 game-logs were used as the data source.

Table 6 :
Data records of the test in human-computer mode with experiential patterns.

Table 7 :
Data records of the test in computer-computer mode.The computer player using experience patterns is denoted by CP1 and the other by CP2.