Vehicle Trajectory Prediction by Knowledge-Driven LSTM Network in Urban Environments

. An accurate prediction of future trajectories of surrounding vehicles can ensure safe and reasonable interaction between intelligent vehicles and other types of vehicles. Vehicle trajectories are not only constrained by a priori knowledge about road structure, traﬃc signs, and traﬃc rules but also aﬀected by posterior knowledge about diﬀerent driving styles of drivers. The existing prediction models cannot fully combine the prior and posterior knowledge in the driving scene and perform well only in a speciﬁc traﬃc scenario. This paper presents a long short-term memory (LSTM) neural network driven by knowledge. First, a driving knowledge base is constructed to describe the prior knowledge about a driving scenario. Then, the prediction reference baseline (PRB) based on driving knowledge base is determined by using the rule-based online reasoning system. Finally, the future trajectory of the target vehicle is predicted by an LSTM neural network based on the prediction reference baseline, while the predicted trajectory considers both posterior and prior knowledge without increasing the computation complexity. The experimental results show that the proposed trajectory prediction model can adapt to diﬀerent driving scenarios and predict trajectories with high accuracy due to the unique combination of the prior and posterior knowledge in the driving scene.


Introduction
Since the 1980s, autonomous vehicles have been regarded as effective solutions to the problems of road safety, traffic congestion, and energy crisis. However, autonomous vehicles still face many driving difficulties in the real urban traffic environment. A major problem is how to interact safely and reasonably with other types of vehicles in a driving scene. Experienced human drivers can predict the future trajectory of other vehicles in a driving scene, thereby making safe, reasonable, and efficient decisions. Accurately predicting the future trajectory of a vehicle not only can reduce or eliminate the collision risk when autonomous vehicles perform complex driving maneuvers, such as merge, lane change, and overtaking, but also can improve the driving efficiency and comfort of autonomous vehicles [1]. In a real urban traffic scenario, the vehicle's driving trajectory is not only constrained by prior knowledge, such as that about the road structure, traffic signs, and traffic rules, but also by uncertain posterior knowledge, including subjective driving intentions of the driver. e influence of driving knowledge on vehicle trajectory is shown in Figure 1, where it can be seen that when the road structure constraints are not considered, the predicted future trajectory, denoted as the red curve, is incorrect. As shown in Figure 1(b), there is a large slowmoving truck in front of the target vehicle. In such a case, based on human driving experience, the target vehicle is likely to adopt a lane change strategy. erefore, how to fully combine the prior and posterior knowledge in a driving scene in the prediction process is crucial for accuracy improvement of the long-term trajectory prediction and safe interaction with other vehicles.
According to the specific prediction process, the existing prediction models can be roughly divided into three categories: physics-based models, maneuver-based models, and learning-based models [2]. e physics-based models use vehicle kinematics and dynamics model to predict the future position of a target vehicle, and they include the constant turn rate and acceleration model [3], switching Kalman filters [4], and Monte Carlo simulation [5]. However, these models ignore the prior and posterior knowledge about a driving scenario, such as road structure, traffic rules, and driver's subjective intentions, which limits these models to short-term prediction (less than 1 s) [6].
Maneuver-based models divide the prediction process into two parts. First, driving intention is estimated according to the physical state of a vehicle, information about the road network, and driver behavior, and then the predicted trajectory is fitted based on the driving intention. For maneuver classification in more complex scenarios, discriminative learning algorithms, including the multilayer perceptions (MLPs) [7], logistic regression [8], relevance vector machines (RVMs) [9], and support vector machines (SVMs) [10], have been very popular. Complex vehicle motion is decomposed into predefined driving action sequences, which makes driving intention easier to identify and classify, and the prediction result is more stable and accurate than that of the physics-based models, and the prediction horizon is longer. However, in complex traffic scenarios, the traditional algorithms, such as finite vector machines and conditional random fields, have the problem of low scene adaptability, while Bayesian network and Markov model can solve the problem of driving maneuver classification in uncertain environments. In addition, the state space of the above models is extremely large, and these models are prone to "curse of dimensionality," and unable to real-time prediction. e approach of [11] predicts the fictive collision probabilities stemming from the execution of each intention by updating a prior intention distribution based on postulation that drivers do not perform maneuvers with high collision risks, but this assumption prevents the detection of specific dangerous maneuvers that are conducive to driving efficiency. Recently, artificial neural networks have been used to classify vehicle driving actions, but the existing highquality calibrated datasets are limited and cannot cover all possible driving scenarios (data sparsity) [12], which makes the network training difficult and challenging, and scene adaptability is low.
Learning-based models skip the step of maneuver recognition and perform trajectory prediction directly based on the historical observation of a target vehicle, so the posterior knowledge in driving scenarios can be effectively learned, and incorrect driving motion recognition can be avoided. Recently, artificial neural networks have been used to predict future trajectories of vehicles, bicycles, and pedestrians [13][14][15]. As a type of recurrent neural network (RNN), the long short-term memory (LSTM) neural network has been proven to be very effective in solving the time series problems, and thus has been widely used in pedestrian trajectory prediction, intersection vehicle destination prediction, and highway vehicle trajectory prediction. However, in previous works, specific-scenario models, such as lane change models for nonintersection sections and left-/rightturn models for intersection areas, have been proposed [10,16,17], and the training data needed manual annotation, which increased the training difficulty of the model. In [18], an encoder-decoder LSTM model is proposed for predicting vehicle trajectory by using an occupancy grid map, and the maximum prediction horizon of this model is two seconds, which is not sufficient for applications.
When an intelligent vehicle is driving in a real urban environment, the driving scene changes dynamically over time, which means that the prediction model should automatically adapt to a driving scene. In order to solve the problem of vehicle adaptability to the driving scene, many studies incorporated the maneuver-based and learningbased models. In [19], two LSTMs were used to identify high-level driver intentions and analyze low-level complex vehicle motion dynamics. is method is better geographyadaptive than the traditional LSTM networks. An LSTM model for interaction aware motion prediction of surrounding vehicles on freeways was presented in [20]. is model assigns confidence values to maneuvers being performed by vehicles and outputs a multimodal distribution over future motion based on these values. e mentioned methods predict the multimodal trajectory based on maneuver classes, which improves the road adaptability, but the prior knowledge in driving scenarios is not used. In [21], a long short-term memory (LSTM) network was employed to anticipate the driving policy of a vehicle (such as forward, yield, turn left, and turn right) using its sequential history observations. e policy was then used to guide a low-level Figure 1: Influence of driving knowledge on trajectory prediction: (a) influence of road structure on trajectory prediction; (b) influence of driving experience on trajectory prediction.
optimization-based context reasoning process. is method combines the prior knowledge in the driving scene and constructs the cost map to perform the second optimization of the previously obtained driving intention to generate the final predicted trajectory, but the driving intention estimation of the upper-level does not utilize the prior knowledge of the driving scene, and the weight of the function cannot be adjusted adaptively to a driving scenario. Deo and Trivedi [22] adopt a convolutional social pooling LSTM-based model. is approach predicts a distribution of future vehicle trajectory dependent on maneuver, but this approach ignores the impact of the interaction of the road users. Dai et al. [23] proposed a spatiotemporal LSTM-based model, which considers the spatial interactions of the surrounding vehicles, but the constraints of other prior knowledge such as road structure, traffic rules, and driving experience are not considered. e dual learning model (DLM) which takes information from two different inputs to predict vehicle trajectory was presented in [24]. is model embeds the occupancy map and risk map into the trajectory model to consider a comprehensive definition of risk in the traffic scene, but the computational complexity usually grows exponentially if the dimensionality of the feature space increases. us, it becomes difficult to meet the online requirement.
In this article, an integrated trajectory prediction model, which combines knowledge reasoning and LSTM neural networks, is proposed. e contribution of this study can be summarized as follows: (1) In order to consider the constraints of the prior knowledge. e prediction reference baseline obtained by knowledge reasoning is introduced into the LSTM network, where the proposed model can effectively combine the prior knowledge without increasing the computation complexity.
(2) In order to learn the spatial interactions of the surrounding vehicles and solve combinatorial explosion problem caused by a large number of condition attributes. A method of deterministic scene evaluation is employed to classify and analyze the main conditions that affect the future trajectory of a vehicle from the perspectives of safety, legitimacy, and reasonableness, which simplifies modeling of the spatial interactions. (3) In order to improve the adaptability of the proposed model. e Frenet coordinates based on the PRB are used to train the LSTM network, and it is not necessary to annotate the training data set manually according to the specific driving scenario. e results of the field test prove the adaptive performance of the proposed model. (4) e performance of the proposed model is evaluated with state-of-the-art methods on a naturalistic highway driving dataset (NGSIM), and the results show that our proposed model outperforms the state-of-the-art methods. e rest of the paper is organized as follows. e prediction reference baseline determination method and the proposed LSTM network are presented in Section 2. e proposed prediction model is evaluated by both simulations and real-traffic urban roadway experiments, and the obtained results are presented and discussed in Section 3. Finally, the main conclusions, limitations, and future work are presented in Section 4.

Problem Formulation and Method
e proposed trajectory prediction model is divided into two layers. e first layer determines the PRB of a target vehicle, and the second layer predicts the future trajectory based on the PRB. PRB is a trajectory that indicates the driving intention of the target vehicle based on prior driving knowledge, which connects the online reasoning system and the LSTM network. e process of driving intention prediction for a target vehicle V i at time t is presented in Figure 2, where it can be seen that it is necessary to understand and evaluate driving scene S t v i , and generate the scene evaluation parameters E t v i of the target vehicle, including the safety assessment SAFE t v i , legitimacy assessment LEGAL t v i , and reasonable assessment REASONABLE t v i , which is expressed as According to the prior knowledge of driving scenarios, such as traffic rules and driving experience, the driving intention b t v i of the target vehicle is inferred based on the prolog online reasoning system. A maneuver b t v i is classified by the lateral movement of the vehicle, which is expressed by a finite set B: Finally, driving intention b t v i is fitted to the PRB PRB t v i by the cubic Bezier curves. e second layer predicts the future vehicle trajectory. First, the coordinate transformation is performed on the historical trajectory of the target vehicle based on the PRB PRB t v i , as shown in Figure 3. In Figure 3, s t v i denotes the distance the target vehicle has traveled along the PRB, and l t v i is the transverse distance between the target vehicle and the PRB t

Journal of Advanced Transportation 3
where a set (s t v i , l t v i ) denotes the Frenet coordinates, κ t v i denotes the curvature, θ t v i represents vehicle heading, v t v i denotes the vehicle speed, a t v i denotes the vehicle acceleration, and M is the input step of the network. e network output P t(n) v i is expressed as where K denotes the output step of the network. Finally, the predicted trajectory denoted as Tra t(n) v i is obtained by transforming the reference coordinates to the absolute coordinates represented by the latitude and longitude, which is expressed as

Overview of the Proposed Approach.
is paper proposes a trajectory prediction model based on knowledge reasoning and LSTM neural network. e architecture of the proposed model is shown in Figure 4, where it can be seen that the proposed model consists of two phases: PRB determination phase and trajectory prediction phase. During the PRB determination phase, by analyzing the relationship between "human-vehicle-road" in the driving scene and extracting the knowledge of road network, traffic participants, and road traffic facilities, the conceptual ontology model of the driving scene is established. e main conditional attributes that affect the behavior decision-making process are classified and analyzed from the perspectives of safety, legitimacy, and reasonableness using the proposed deterministic situation assessment method, and situation parameters in the horizontal and vertical directions are obtained. e behavior prediction rule base is constructed using the situation parameters, traffic rules, and driving experience. Based on the prolog online reasoning system, the behavioral prediction rules are matched with the factual knowledge obtained by the conceptual ontology model, and the driving intentions are inferred. Finally, a third-order Bezier curve is used to fit the driving intention to a PRB. e trajectory prediction phase uses the LSTM network to learn the continuous features of the historical trajectory of a target vehicle on the basis of the PRB and generates the final predicted trajectory.

Prediction Reference Baseline Determination.
e architecture of the proposed PRB determination method is presented in Figure 5, where it can be seen that this method consists of online and offline phases. e offline phase establishes the conceptual ontology model (Tbox) of a driving scene and extracts the behavioral prediction rules based on traffic rules and driving experience. According to the conceptual ontology model, the road network and real-time environment perception information are used to instantiate the entities and related relationships in the driving scene (Abox). e entities and entity relationships in the driving scene are classified by the deterministic scene assessment method and analyzed from the perspectives of safety, legitimacy, and reasonableness. e scene evaluation parameters in both horizontal and vertical directions are  generated, and the behavioral prediction rules are matched with the scene evaluation parameters by using the prolog online reasoning system. e driving intentions are inferred, and finally, the third-order Bezier curve is used to fit the driving intention to the PRB.

Semantic Modeling of Driving Scene.
In a driving scenario, there are various road element entities, such as traffic participants, road networks, and road traffic facilities in urban driving scenarios. e environment perception system can provide only the spatial location of each entity, but it cannot describe the correlation between entities, and make full use of prior information, such as traffic rules and driving experience, which is crucial for improving the prediction model adaptability to the driving scene. Ontology, as a form of knowledge expression, is used to model the concepts of specific domains and relationships between concepts, which can be used to model driving scenarios effectively [25][26][27]. e conceptual ontology model is divided into two module types: entities and attributes. is study takes the target vehicle as a perspective and summarizes five entity types on the basis of [28]: (1) Target vehicle e target vehicle entity describes the vehicle to be predicted.   Journal of Advanced Transportation behavior and includes three behaviors: Change-ToLeft, ChangeToRight, and KeepLane. e AdvancedBehavior represents advanced driving behavior and includes two behaviors: Overtake and Merge.
(3) Obstacle e obstacle entity represents a collection of obstacle entities encountered by a vehicle during driving. is work divides obstacles according to the behavior characteristics of obstacle entities in driving scenarios into two categories: StaticObstacle and DynamicObstacle.
(4) Road network e road network entity represents the topological connection of roads by intersecting points and lines. RoadType includes different road types. RoadPart describes the components of the road network and is divided into AreaEntities and PointEntities. Area-Entities refers to road entities that can be abstracted into lines and areas, such as lane, side walk, junction, and segment, while PointEntities refers to road entities that can be abstracted into points, such as road signs, traffic signs, and traffic lights. (5) Driving scenario e driving scene entity refers to a collection of road entity elements encountered when a vehicle travels in different road areas. In this work, driving scenarios are divided into three categories: InSpecialAreascenario (special area driving scenario), OnRoadscenario (road driving scenario), and NearSpecialAreascenario (near special region driving scenario). InSpecialAreascenario category can be further divided into IntersectionScenario (intersection scene), TunnelScenario (tunnel scene), Bridge-Scenario (elevated scene), and UturnScenario (Uturn scene). e object attribute is used to describe the relationship between concept classes.
is attribute restricts the described relationship regarding the domain and range. e data attribute restricts the described relationship through the definition and value domains. e definition domain is a class type. e described ontology modeling process of driving scenario is equivalent to filling the background knowledge of the TBox that constitutes the ontology knowledge base, but the situational knowledge in the ABox is still lacking. According to the road elements of a real driving scenario, the driving scenario needs to be re-expressed using the conceptual model of the TBox, which is an instantiation of the ontology model. A real-traffic scenario is displayed in Figure 6(a); a concrete driving scene that includes instances of defined classes is presented in Figure 6(b), and its semantic description is presented in Figure 6(c). e instances of RoadNetwork are added to the ABox as prior knowledge, and instances of the obstacle are asserted in real time.

Situation Assessment.
After obtaining a semantic description of a driving scene, it is necessary to determine and evaluate condition attributes that affect the driving intention in the driving scene, so as to estimate the driving intention of a target vehicle. In order to solve the problem of combinatorial explosion due to numerous condition attributes [28], a deterministic scenario assessment method is adopted to classify and analyze the key attributes that affect driving intentions from the perspectives of safety, legitimacy, and reasonableness.
Deterministic scenario assessment methods use the threat assessment indicators: TTC (time to collision), THW (time headway), TTB (time to brake), DST (deceleration to safety time), and MSM (minimal safety margin) in rule-based systems, and the probability of collision is estimated as a binary value. For instance, Glaser et al. [29] used the TTC and TIV (time intervehicles) indicators to evaluate the possibility of collision. Noh et al. [30] proposed a distributed reasoning method by dividing the current and adjacent lanes into the front and rear areas, and the TTB and MSM indicators were used to evaluate the possibility of collision in the front area, while the TTC and MSM indicators were used to evaluate the collision of rear area collision possibility.
e proposed deterministic scenario assessment method consists of two parts. First, the driving scenario is determined by querying the knowledge base with the current vehicle position. en, a reasoning structure of an obstacle is constructed in eight regions of interest to make safety assessment, and a binary result (safe or dangerous) is calculated for each region using critical indicators TTC and TIV. Finally, legitimacy and reasonableness assessments are made to predict the maneuver of the target vehicle.
(1) Safety Assessment. Safety primarily refers to whether the surrounding obstacles pose a threat to a vehicle, especially in the area ahead, but it also refers to whether the left or right lane can provide a safe lane change. is paper constructs the eight-direction obstacle inference model. For each area, the TTC and TIV indicators are used for safety assessment. e TTC indicator is defined as a time when two vehicles continue to collide on the same trajectory at the current speed, and it is defined by where D i denotes the relative distance between the following vehicle and followed vehicle, V denotes the speed of the following vehicle, and V i is the speed of followed vehicle. e threshold value TTC th is used to judge whether a vehicle is dangerous in high-speed scenarios. e risk assessment formula is as follows: and when the calculated collision time t between the following vehicle and followed vehicle is greater than TTC th , 6 Journal of Advanced Transportation the current scene is considered to be safe; otherwise, it is considered to be dangerous. e TIV indicator is used to detect low-speed difference scenarios. When the speeds of two vehicles are similar in value, the TIV indicator is used to judge the degree of danger, and it is calculated by reshold TIV th is used to distinguish between safe and dangerous scenes in low workshop distance scenes. e TIV risk assessment formula is as follows: When the calculated vehicle interval time t between the following vehicle and followed vehicle is greater than TIV th , the current scene is considered to be safe; otherwise, it is considered to be dangerous, and in that case, the following car needs to perform a certain action to avoid a possible collision.
In each region, only when R TTC (t) and R TIV (t) are calculated safety synchronously, the region is considered to be safe. e risk assessment of an region is determined as Journal of Advanced Transportation where Region represents one of the eight regions, as shown in Figure 7; when r Region (t) has a value of zero, the area is considered to be safe, and when r Region (t) has a value of one, the area is considered to be dangerous. e degree of danger in the area ahead is calculated by and when the calculated values of the TTC and TIV indicators in the current area are greater than the predefined threshold, the following result message is obtained: safe-ToGo (targetVehicle, keep); when the calculated values of TTC and TIV indicators in the current area are both less than the predefined deceleration threshold, but greater than the corresponding parking threshold, the result message is safeToGo (targetVehicle, dec); in this case, a lane change can be performed to improve driving efficiency; otherwise, the result message is safeToGo (targetVehicle, stop). e safety assessment of adjacent lanes is conducted using the same assessment formula as that of the area ahead. If the left lane is taken as an example, then the degree of danger is expressed as and when there are vehicles in the left area, the left lane can be considered to be dangerous; otherwise, the safety of the left front and left rear areas are, respectively, evaluated by erefore, the safety assessment of the left lane is as follows: If the left lane is safe, safe To Left(target Vehicle, true) will be generated; otherwise, the scene evaluation parameters will be instantiated as safe To Left(target Vehicle, false).
(2) Legitimacy Assessment. e legality assessment includes three assumptions. First, when a vehicle is driving on the road, it cannot exceed the maximum speed limit of the road; second, when the vehicle is driving to the preintersection, it is necessary to pay attention to the change in traffic lights and obey the traffic rules; third, when a vehicle is about to change the lane, the adjacent lane should allow lane changes.
(3) Reasonableness Assessment. Reasonableness assessment generally refers to whether lane changing and other driving behaviors affect the current goal of a target vehicle. Based on the current lane of the target vehicle, the specific road section or lane to be driven can be known. For instance, on the one hand, if the next area to be driven by the target vehicle is an intersection, and the distance between the target vehicle and the stop line is less than δ, then lane change is not recommended. On the other hand, if the distance between the target vehicle and the stop line is greater than δ, lane change can be performed. Reasonableness assessment introduces a situation parameter set (reasonable To Left, reasonable To Right), which is defined as data properties in the ontology model.

Rule-Based Reasoning.
e driving intention is determined based on traffic rules and driving experience, where the traffic rules are mainly used to limit the driving behavior while the driving experience is utilized to summarize the understanding and cognition of human drivers in different scenes and obtain some rules that are not specific traffic rules but are conducive to the reasonable driving. According to the different driving scenarios defined in the driving knowledge base, the rules stored in the driving knowledge base are divided into several categories. Different scenarios have different key road entities. For instance, unlike On Road Scenario, in Near Intersection Scenario, traffic lights are considered. Also, the classification of traffic rules can reduce the rule search space and reasoning time.
In order to save computing resources and reduce reasoning time, SWI-Prolog language is used to write rule knowledge, which is represented as a set of driving scenedriving behavior mapping pairs, where driving behavior is described as a rule head, and the driving scene is described as a rule body. On the basis of [28], this paper adds the scene assessment as an intermediate link of mapping driving scene to the driving behavior and reorganizes 57 rules. Some of the prediction rules are presented in Table 1.
e online reasoning process can be described as follows. First, the real-time facts related to the scene are used as input, and each rule statement is matched. If all the facts of the corresponding rule are matched, the matched prediction result will be obtained, and the next rule statement will be matched until each rule is matched. When all matched results are obtained, the final result denotes the predicted driving intention.

Prediction Reference Baseline Fitting.
After obtaining the driving intention of the target vehicle, the driving intention is converted into the prediction reference baseline using the cubic Bezier curves. As shown in Figure 8, first, a target lane is selected based on the driving intention and road network, where p 0 represents the current position of the target vehicle, and p 3 is selected from the centerline of the target lane with distance L d from p 0 , and L d is obtained based on the driving experience. e prediction reference baseline is divided into three parts by p 0 and p 3 : the predicted extension, the historical extension, and the intention segment. In addition, L p and L h are determined by the input and output steps of the LSTM neural network.

Journal of Advanced Transportation
As shown in Figure 8(c), the cubic Bezier curve constructed by four control points is used to generate the intention segment, which is expressed as where B i,3 is the Bernstein polynomial and it is given by e coordinate system (X ″ , Y ″ ) is built with the origin P 0 at the vehicle center. e x-axis direction is the vehicle's initial heading, the terminal state will be the end point P 3 , and P 1 and P 2 are obtained by moving forward for distance d along the vehicle's initial heading direction from the start point and backward for distance d along the terminal heading from the end point P 3 , respectively. e position of the control points in the above coordinate system is expressed as where L x and L y are lateral and longitudinal offsets of the terminal state P 3 to P 0 , respectively; ω is the angle between the terminal heading and the direction of the x-axis. e terminal heading is defined as the tangential direction of the closest point on the PRB to P 3 .
Equations (15) and (16) can be rewritten by applying (17), so the Bezier curve can be represented as Besides, the curvature of the generated path can be derived by applying (19) and (20), which leads to e maximum of the curvature should satisfy the condition given by equation (21) to meet the vehicle's nonholonomic constraint: In equation (21), L denotes the vehicle wheelbase and φ max denotes the maximum steering angle of the vehicle. e maximum of the curvature κ(t m ) is a function of d. e suitable value of d that satisfies the vehicle's nonholonomic constraint can be found by brutal searching from d ′ /6 to d ′ /2, where d ′ denotes the distance between P 0 and P 3 . e processing time can be reduced by building a lookup table that matches a given set with the corresponding maximum curvature of the Bezier curve.

LSTM Network Driven by Knowledge.
Since different drivers have different driving styles, in order to accurately predict the future trajectory of a vehicle, in this work, an LSTM neural network is employed to learn the continuous features of the historical trajectory. e LSTM is an RNN type that can effectively overcome the problem of gradient disappearance [31]. e LSTM is composed of a unit memory that stores the previous input sequence information and a gating mechanism that controls the information flow between input, output, and unit memory. ere are three gates in the core design of the LSTM network, namely, the input gate, the forget gate, and the output gate. e specific network structure is shown in Figure 9. e forget gate is used to control how much information is retained in c t . e input gate determines how much information of x t remains in c t , and finally, the output gate determines how much information in the output o t is output to h t by the control unit c t . e work of the LSTM is described by the following recursive equations: where x t denotes the input vector,σ(x) denotes the activation function, W denotes the linear transformation matrix, b denotes the offset vector, i t , f t , and c t are gate vectors, c t represents the amount of cell memory, and lastly, h t denotes the output.
In this work, the network presented in Figure 10 is used as a reference structure. is network has two layers consisting of 256 LSTM cells, followed by one time-distributed layer consisting of 128 neurons, and the final dense output layer containing as many cells as the number of outputs. e network input is a tensor of track histories of a vehicle. e network output consists of the future coordinates and velocity of the vehicle. Since the prior knowledge about the driving scene is expressed by prediction reference baseline, the network can learn the posterior knowledge about the driving scene only from the relative relationship between the historical trajectory and the prediction reference baseline. Compared with the existing prediction models based on the LSTM network, the proposed prediction model reduces the network training difficulty and decreases demand for the computing performance of the vehicle platform.

e Training Dataset.
e next generation simulation (NGSIM) dataset in I-80 and US101 sections is used for model training and testing [32], and this dataset is derived from the US Federal Highway Administration, which is currently the largest public natural driving public data source, and thus has been widely used in the literature [33,34]. e layouts and top-down views of the US101 and I-80 sections are shown in Figure 11. Each data frame includes many vehicle's parameters, including the position, velocity, yaw rate, size, and others. e sampling frequency of the dataset is 10 Hz; therefore, in this work, Δt is set to 0.1 s.

Data Preparation.
e vehicle positioning data in the NGSIM dataset are obtained by video analysis, so the recorded trajectory contains a lot of noise [26]. erefore, the vehicle kinematics model and the road geometric are used to filter the original data, which is expresses as e vehicle position is transformed to the Frenet coordinates based on the centerline. As shown in Figure 12(b), the centerline of each lane is extracted and fitted using the shapefile, and the centerline that the target vehicle was initially driven is selected as a reference baseline. For each original coordinate point (x v , y v ), the corresponding mapping point (x p , y p ) on the reference baseline is determined, and the Frenet coordinates (s v , l v ) are obtained by where l v denotes the Euclidean distance between (x v , y v ) and (x p , y p ), and S(x p , y p ) denotes the length from the mapping point (x p , y p ) to the starting point of the reference trajectory.   In addition, four other features, curvature κ, velocity V, acceleration a, and heading θ, are also selected so as to compose the observation vector o with the Frenet coordinates (s, l).

Training Details.
ere were 8311 filtered trajectories; 80% of the trajectories were selected as the training set, 10% as the test set, and the remaining 10% was used as the verification set to observe if the model is overfitted. e network was trained using minibatches with a size of 64. Due to the limitation on a sensor measurement range and noise in practical application scenarios, it was difficult to track dynamic vehicles stably for a long time, so the network was trained using windows that consisted of 30 inputs, representing a total of 3 s past observations. e Adam optimizer was used; the learning rate was 0.0005, and ReLU activation with α � 0.1. e loss function adopted the MSE (mean square error) between the predicted sequence and the ground truth sequence; the code used to generate the model was written in Keras, and the training was performed on NVIDIA GTX 2080 s GPU using the TensorFlow backend. e model training contained 16 epochs, and the average training time for each epoch of the full training set is around 2300 seconds.

e Impact of the Prediction Reference Baseline.
To investigate the impact of considering prediction reference baseline on the accuracy of the proposed method, we test the RMSE performance of the proposed modelwith three modifications using the NGSIM dataset.
In one experiment, the system is trained and tested with the absolute coordinate, in the second, the centerline that the target vehicle was initially driven is selected as a PRB, and finally, in the third experiment, the PRB of the target vehicle is determined by the method in Section 2, while the other attributes of the three models are unchanged. Figure 13 shows the accuracy of the trajectory prediction for different time horizons, and the RMSE value of the model is decreased by adding the PRB for both lateral and longitudinal trajectories.

Comparative Study.
To evaluate the proposed approach, we purse a direct comparison with state-of-the-art vehicle trajectory prediction using the same dataset (i.e., NGSIM). e results show that the proposed method outperforms the state-of-the-art model and decreases the overall RMSE value of the system by 10 percent on average. Table 2 summarizes the RMSE values comparing the proposed methods with the baseline trajectory prediction models in the literature [20,[22][23][24]35].
e comparison results show that the proposed knowledge-driven LSTM network has better performance in RMSE for NGSIM. Note that as compared to the baseline [24], the prediction accuracy gets a slight improvement, but the proposed method enhances the real-time performance and much reduces the computational complexity due to the reduction of the feature space dimension.

Simulation Experimental Platform.
e simulation experiments were based on the JAC's automatic driving hardware-in-the-loop test platform. e experimental simulation platform is presented in Figure 14, where it can be seen that the experimental platform included the real vehicle braking system, steering system, sensor system, and network communication system, which had dSPACE (Matlab/Simulink) as a core. e controller rapid prototyping platform was built, virtual reality interfaces and environment-aware sensor modules were provided using the PreScan software, and the CarSim software was used to run the vehicle dynamic model and provide a platform that could be quickly verified for automatic driving algorithm testing.

Simulation Results.
e simulation scenario shown in Figure 15 was established according to the real urban traffic scenario. Two typical traffic scenarios were selected to verify the adjustment effect of the PRB on the predicted trajectory. Figure 16(a) shows the driving scene on the road, and Figure 16(b) shows the driving scene at the intersection. In the first scene, three PRB intent segments were fitted to lane keeping (LK), lane change left (LCL), and lane change right (LCR), as shown by the blue curve in Figure 16(a). e historical observation vector of the target vehicle denoted the network input, and it was unchanged; the output network vector was converted according to three PRB to obtain three predicted trajectories, as shown by the green curve in Figure 16(c). In Figure 16(b), the intersection driving scenario is presented, where two PRB intent segments of go straight (GS) and turn right (TR) are fitted, respectively; the converted network output results are shown by the green curve in Figure 16(d). Since the network learns the relative relationship between the historical and PRB, even at the same network input, the predicted trajectory will be affected by the prediction reference baseline. e experimental results prove that the priori knowledge about the driving scene can be used to adjust the predicted trajectory effectively based on the prediction reference baseline.

Experimental Platform Construction.
In order to verify if the simulation results obtained in coincide well with the real-world scenario results, an instrumented vehicle was used to collect data, as shown in Figure 17(a). e vehicle loading sensors included an IBEO four-layer laser scan instrument, a Velodyne HDL-64E lidar, two high-resolution cameras, and a differential GPS/INS (SPAN-CPT) system. e sensor configuration of the vehicle and its sensing range are shown in Figure 17(b). e differential GPS module provided the information on the position, speed, and heading of the ego vehicle. Based on our previous work [36], moving obstacles were detected and tracked by a four-layer laser scanner, which was located at the front of the vehicle. According to the space-time relationship between the moving obstacles, such as pedestrians and vehicles, and experimental vehicles, the position, speed, size, and type of the sports vehicles can be measured. We conducted a real vehicle experiment in Hefei, Anhui Province, China. e test road is shown in Figure 18(a). e prediction model was exemplarily implemented on NVIDIA Xavier platform using the C++ programming language.
Before conducting the actual vehicle experiment, the high-resolution maps were collected to establish based on our experimental vehicle. ere were more than 1,140 road entities on the map, including the stop signs, lane markings, and lane lines, covering approximately 8 km of the roadways (as Figure 18 shows).

Field Test and Discussion.
e experimental driving route was located on a typical urban roadway, with a total length of about 4.7 km. It includes multiple intersections, Y-shaped intersections, T-shaped intersections, and other common urban road scenarios. Due to the long experimental route and a large number of scenes encountered, it was inconvenient to conduct the prediction process for each scene. erefore, two typical scenes were selected for detailed trajectory prediction process analysis.
In Scenario 1, vehicle054 was on the road. e input conditions of the scene evaluation module are shown in Figure 19(b). In front of the target vehicle, there was a large truck denoted as vehicle055 that was moving with a speed of 5 km/h. At that time, the speed of vehicle054 was 24 km/ h. e calculated value of the TIV was less than the deceleration threshold. erefore, it was judged that the target vehicle would have an intension to change lane. Vehi-che054 was driving on lane00055, which was a straight lane.
rough an associated search in the conceptual ontology model of the driving scene, it was learned that the right lane was also a straight lane, and the lane line was a white dotted line; the prolog rule of legality is expressed as follows: legalToRight (target, true): targetVehicle (target), isOnLane (target, Lane), hasRightLine (Lane, Line), and hasLineType (Line, "dotted_white").   e next section of the target vehicle to travel was the intersection, and the distance to the intersection was greater than 30 m, so the effectiveness of changing lanes was satisfied; thus, the lane change did not affect the current target of the target vehicle; the prolog rule of reasonableness is as follows: reasonableToRight (target, true): targetVehicle (target), currentRoadState (target. "ApprJunction"), isOnSegment (target, Seg), connectToJunction (Seg, Junc), intersection (Junc), connectToStopLine (Seg, SL), and distToStopLine (SL, DL), DL ≥ 30. e autonomous vehicle was driving in the right back region of the target vehicle, and the TTC and TIV values were both less than the corresponding acceleration threshold but greater than the corresponding parking threshold; the prolog rule of safety is as follows: safeToRight (target, true): targetVehicle (target), has-RightObstacle (target, null), hasRightFrontObstacle (target, null), and hasRightBackObstacle (target, egovehicle). e final prolog rule is as follows: canChangeToRight (target, true): safeToRight (target, true), reasonableToRight (target, true), and legalTo-Right (target, true).
After obtaining the driving intention of the target vehicle, the prediction reference baseline (the blue curve in Figure 19(a)) was fitted, and the historical trajectory was transformed into Frenet coordinates based on the prediction reference baseline and then fed to the LSTM network input; the predicted trajectory is shown by the green dotted line in Figure 19(a). e trajectory prediction results of vehicle054 could effectively reduce the reaction time of autonomous vehicle while avoiding collisions caused by vehicle054 cutting in.
In Scenario 2, vehicle121 was in the preintersection scenario. e input conditions of the scenario evaluation module are shown in Figure 20(b). e current speed of vehicle121 was 24 km/h, and the lane it traveled was lane000103, which was a right-turn lane. In this scenario, the main factor affecting the target vehicle's driving intention was the traffic light. e prolog rule of legality is as follows: legalToTurnRight (ego, acc): targetVehicle (target), currentRoadState (target, "ApprJunction"), isOnSegment (target, Seg), connectToJunction (Seg, Junc), intersection (Junc), hasTrafficLight (Junc, TR),  e experimental results of Scenarios 1 and 2 verify that the proposed LSTM network can effectively combine the prior and posterior knowledge in the driving scene, and the lane change behavior can be predicted before the variation of vehicle kinemics. e proposed LSTM network can be iteratively adapted to a driving scenario without manual annotation during the network training, which significantly reduces the training complexity and solves the sparse data problem. Due to the unique combination of knowledge reasoning, the proposed prediction model can make a more accurate and reasonable estimation of future trajectories of  surrounding vehicles than the existing models in different environments.

Conclusions
is paper combines a maneuver-based and learning-based trajectory prediction models and proposes an improved trajectory prediction model based on the LSTM neural network driven by driving knowledge. In order to achieve better use of a prior driving knowledge in driving scenarios and solve the problem of the combinatorial explosion caused by a large number of conditional attributes, the multisource and heterogeneous information of the driving scenario is modeled based on ontology, and a driving knowledge base, including the driving experience and traffic rules, is constructed. en, the conditional attributes that affect driving intentions are classified and analyzed from the perspectives of safety, legitimacy, and reasonableness, and situation parameters in the horizontal and vertical directions are generated by the deterministic scene evaluation method. Finally, using the obtained situation parameters and the driving knowledge base, the driving intention is inferred based on the prolog online reasoning system. In order to make the prediction results effectively combine the posterior knowledge and solve the problem of insufficient adaptability of the existing learning-based prediction models, this paper converts the driving intention of the target vehicle into a prediction reference baseline, and the Frenet coordinates based on prediction reference baseline are used as a coordinate frame for the LSTM neural network. e prior driving knowledge existing in the driving scene can be used to adjust the predicted trajectory in the form of the prediction reference baseline without increasing the network complexity but ensuring efficient operation of the proposed model on an embedded platform.  e proposed prediction model was verified by simulations and experiments. e simulation results showed that the prediction reference baseline could effectively adjust the output of the LSTM neural network, making the predicted trajectory meet the constraints of the prior knowledge in a driving scenario. e real-world-experiment results show that the proposed prediction model can significantly reduce the computing performance requirements while ensuring real-time performance on the embedded platform. Also due to the full combination of prior and posterior knowledge in the driving scene, the target vehicle's lane-changing behavior can be predicted on average 2.05 s (for LCL) or 2.71 s (for LCR) in advance, and the precision can be improved by 12.5% for long-term predictions and is more robust, flexible, and adaptive in complex traffic scenarios.
Even though the proposed model has advantages in trajectory prediction, there are still some limitations, such as that indexes of scenario assessment are not comprehensive enough. In order to overcome this limitation, in future work, the evaluation index will be considered from the perspective of human-vehicle interaction and multivehicle interaction. Furthermore, data from a more complex scenario will be collected and used to verify the proposed prediction model.

Data Availability
e data used to support the findings of this study are available from the corresponding author upon request.

Conflicts of Interest
e authors declare no conflicts of interest.