Reliability and Security Analysis of Artificial Intelligence-Based Self-Driving Technologies in Saudi Arabia: A Case Study of Openpilot

Saudi Arabia has an ambitious vision that embraces artiﬁcial intelligence (AI) technologies at a mass scale in new cities such as Neom. Self-driving has recently become one of the most important AI applications due to the advancement of sensors and AI algorithms. Given that safety is vital to the success of self-driving cars, existing infrastructures (e.g., roads and traﬃc signs) should be compatible with self-driving technologies. However, self-driving technologies have not been thoroughly examined in Saudi Arabia with regard to the country’s infrastructure and traﬃc. Therefore, this paper highlights the main areas of improvement in available self-driving technologies in Saudi Arabia. This analysis can help governments understand the current limitations of such technologies so that they can regulate them and enhance infrastructures to prepare for the mass adoption of self-driving cars. It can also help car manufacturers and developers improve self-driving algorithms to overcome their existing limitations, which will ultimately improve the safety and experience of driving.


Introduction
Cars remain the main method of transportation both globally and in the Kingdom of Saudi Arabia (KSA). However, traffic safety statistics show that more efforts are required to improve car safety. According to a World Health Organization (WHO) report, the yearly fatality rate per 100,000 people caused by traffic accidents in the KSA has increased from 17.4 to 27.4 since the last decade [1,2]. Such a rate puts KSA the worst among the G20 countries in terms of deaths resulted from traffic accidents [3]. Such accidents also cost the kingdom around 5.6 billion dollars (4.3% of its Gross Domestic Product (GDP)) annually [4]. To reduce such a shocking rate of accidents, it is important to focus on the driver aspects because human error is the main cause of road traffic accidents (over 90%), according to the U.S. National Highway Traffic Safety Administration [5] and the European Union [6]. In KSA, also driver behavior such as speeding, driver distractions, noncompliance to traffic rules, and sudden lane change is the main cause of traffic accidents according to different studies [7][8][9].
To improve safety, there have been many recent advancements in traffic violation detection technologies and stricter punishments in KSA to decrease such dangerous driving behaviors [10]. However, it is important to implement proactive measures that improve driver control. Hence, self-driving cars can also improve traffic safety in the kingdom since many existing self-driving technologies show significant safety benefits. For example, in lane-keeping, an essential task in driving, technologies such as lane centering has already demonstrated success in reducing traffic crash fatalities and injuries in both left-and right-side driving in different parts of the world [11][12][13]. In addition, many studies have shown that automatic braking (i.e., adaptive cruise control (ACC)) is very effective in reducing collisions [14][15][16]. Due to these benefits, many studies have estimated that market demand for self-driving cars will grow by 63.1% between 2021 and 2030 [17] and that market value will reach 60 million USD by 2024 [18].
However, self-driving cars can also cause fatalities if not implemented correctly and tested thoroughly. For example, in 2018, a self-driving Uber car hit and killed a woman because it failed to recognize jaywalking pedestrians [19]. A Tesla driver was also killed in 2018 in a deadly car crash because the Tesla's Autopilot system failed to detect a side concrete lane divider and slammed into it [20]. Infrastructure should be also prepared for self-driving cars. is is especially important for solutions like Waymo, an autonomous driving technology development company that focuses on certain selected premapped areas in the United States. us, there is a need to investigate the reliability of such systems in different conditions. In this paper, I analyze Comma.ai Openpilot (OP), a self-driving technology available in KSA, to examine the reliability and compatibility of such technologies with Saudi infrastructure. I also provide an extensive dataset of real drives in KSA. is can help automakers and developers improve self-driving algorithms by overcoming these algorithms' existing limitations and addressing the KSA infrastructure challenges. It can also help governments understand the current limitations of such technologies so that they can regulate them and enhance infrastructures to prepare for the mass adoption of self-driving cars.

Related Work
Self-driving technologies require thorough reliability and security analysis to ensure the safety of passengers. With the increased reliance on machine learning (ML) algorithms for self-driving systems such as OP, new attacks are emerging to exploit such systems. For example, a real-world DNN-based attack on LKASes was explained in [21]. Hence, many efforts were made to improve the reliability and security of complex self-driving systems. For instance, testing system resiliency is a fault injection framework based on system-theoretic process analysis [22]. In this framework, the system is tested under different identified hazardous scenarios created by a strategic software fault injection. In addition, TensorFI is an injection tool for hardware and software faults in Tensor-Flow programs. TensorFI evaluates the resilience of 12 ML programs, including DNNs used in the autonomous vehicle domain [23]. Moreover, Moss et al. developed an opensource framework to learn the underlying probability distributions of the noise disturbances from driving data in low and high-fidelity simulators [24]. is work extends a reinforcement learning approach known as adaptive stress testing to be more data efficient in detecting probable failures in simulations.
Security of such systems must also be investigated, and many efforts have been made in this area. Moukahal and Zulkernine presented security vulnerability metrics for connected cars [25]. Vulnerability assessments help software testers identify which components need to be prioritized to reduce risk and thereby secure the car. e proposed metrics were applied to OP to show how to measure the car's software. Another approach was also taken by Moukahal et al. in Vulnerability-oriented fuzz (VulFuzz) testing to direct and prioritize the fuzz testing toward the most vulnerable components in connected cars [26]. To increase test performance and avoid dropped test cases, they used an input structureaware mutation technique to bypass car software systems' input formats. Furthermore, Chen et al. proposed a learning framework to detect novel scenarios (due to adversarial attacks, for example) in a task-aware manner [27]. ey use network saliency to provide the learning architecture with information of the input areas that are relevant to the decision-making and learn an association between the saliency map and the predicted output to detect the novelty of the input. Jiang et al. also proposed an Intrusion Detection System (IDS), named Road context-aware IDS (RAIDS) that validates corresponding frames observed on the in-car network and detects anomalous frames that substantially deviate from road context. is IDS can protect the self-driving systems from out-of-context road pictures that can be inserted maliciously into the vision model to cause inaccurate driving decisions accordingly. In addition, Dasgupta et al. proposed a Global Navigation Satellite System (GNSS) spoofing attack detection method using the long short-term memory (LSTM) model [28]. e recurrent neural network model LSTM is used to predict the distance traveled between two consecutive locations of an autonomous car.
OP in particular has previously been assessed in terms of the probability of accidents under a base distribution governing standard traffic behavior [29]. In that study, the authors performed their experiments in a black-box simulation to quantify system risk resulting from failure to make correct decisions in normal conditions, such as direct sunlight. Researchers have also examined the security of the system. Moreover, Liang et al. provided a quantitative analysis of adversarial attack impact (leveraging road patches) on system planning and control [30]. ey considered an end-to-end approach that addresses adversarial attacks that focus on the entire advanced driver assistance system (ADAS) or autonomous driving pipeline, from sensing and perception to planning, navigation, and control.
Datasets used on most works that examine OP relied on an available dataset produced by Comma.ai called Com-ma2k19. is dataset includes 33 hours of driving in California, US in 2019 [31]. It was collected using EONs (i.e., the first generation of Comma devices) that have sensors similar to those of any modern smartphone, including a road-facing camera, phone GPS, thermometers, and 9-axis IMU. In addition, the EON captures raw GNSS measurements and all CAN data sent by the car. is dataset can be used to train vision models or self-driving algorithms. However, it cannot be used to study KSA roads due to the differences in infrastructure and driving behavior. It also does not include a wide view of the roads, which can be useful in applications such as recognizing road signs. erefore, the aforementioned works cannot be used to analyze OP, especially in the context of KSA due to the following reasons: Hence, this paper provides a reliability and security analysis of the latest version of OP using two different hardware in two different cars. Moreover, the datasets used in the analysis are collected from KSA based on real drives in different areas of KSA. e importance of this work is that it does not only focus on the evaluating vision model through simulators but also covers different aspects such as data security, environmental conditions like sunlight and temperature. Another key contribution of this work is that it provides a dataset of real unique drives. is will help developers to better design and tune vision models in selfdriving technologies to improve their effectiveness in KSA.

Background
Self-driving cars, also known as driverless cars, autonomous vehicles, or robo-cars, are cars that are capable of sensing their environments and moving safely with little or no human intervention. is paper focuses on passenger selfdriving car solutions, which are classified into six levels based on the extent of their driving automation (i.e., the amount of human intervention required) [33]. As shown in Table 1, the Society of Automotive Engineers (SAEs) classifies driving automation into six levels ranging from 0 (fully manual) to 5 (fully autonomous). e SAE levels have been widely adopted in many regions, including the United States [39] and Europe [40]. Table 1

Comma.ai Openpilot
Openpilot (OP) is a Level 2 open source self-driving solution developed by Comma.ai, a company founded by George Hotz, a security hacker known for developing the early iOS and PlayStation 3 privilege escalation exploits [42]. It is compliant with the industry standard MISRA C, which includes software development guidelines for the C programming language developed by the Motor Industry Software Reliability Association (MISRA) that aim to facilitate code safety, security, portability, and reliability in the context of embedded systems. As shown in Table 2 according to Consumer Reports, OP is currently the most advanced ADAS of the many available systems [43]. OP supports a growing list of cars equipped with necessary sensors and integrates with cars' stock sensors to enhance the stock lateral and longitudinal control. More specifically, OP improves the automated lane keep assist system (LKAS), which uses the camera(s) mounted on the windshield. In addition to the car sensors, OP also uses its AI-based vision model to provide accurate lateral control. OP also uses some additional sensors from systems such as blind-spot monitoring for safe automatic lane changing [44]. For longitudinal control, OP requires car sensors from systems like ACC and forward collision warning to provide additional and more accurate functionalities.
OP works by intercepting and modifying (i.e., man-inthe-middle) the commands sent from the LKAS camera or ACC sensors inside hidden Controller Area Network (CAN) buses in the car network. Hence, OP's commands should match the original commands issued by cars to avoid any technical errors. Most modern cars use several CAN buses to allow the various modules of the car to communicate with one another. One CAN bus is exposed to the OBD2 port, which is often located under the steering wheel. OP originally used a neural network in the vision model to detect the lanes captured by the camera(s) and issue steering commands accordingly, although as of version 0.8, OP does not use ML for longitudinal control [45], instead relying only on the stock system commands. To ensure compatibility with every supported car, the commands issued by OP regenerate the original commands issued by the car, with some minor changes to masquerade as though they were issued from the car stock systems. Despite some efforts by developers, OP currently does not yet fully support the FlexRay networks found in European brands such as Audi, BMW, and Mercedes [46].
OP can only be installed on hardware purchased from Comma.ai that is then installed in supported cars. e updated full list of supported cars can be found on the system's page [47]. As of December 2021, Comma.ai sells two types of devices: the Comma ree and the Comma Two, which cost, with a car harness, $2,199 [48] and $1,099 [49], respectively. ere are other cheaper unofficial hardware made by the community members, such as [50], which are outside the scope of this paper. e official kit includes the following items, which are connected inside the car as shown on the setup page [51]: (i) Comma Device ( ree or Two). is device is an Android-based (Two) or Linux-based ( ree) phone enclosed in a case that includes fans and infrared cameras that monitor the driver's attention, even at night. Compared with Comma Two, Comma ree has an additional front wide-angle camera for detecting cars on side lines. Both devices are designed to always be placed on the windshield because they have auto-shutoff to protect the car battery and can tolerate high heat from the sun since they do not have batteries. ese devices are the brain of OP, as they are responsible for real-time capturing, analyzing, and issuing of commands that control the car as well as monitoring the driver's attention.
is is a universal interface to the car to connect the Comma device to the car. It is designed to be almost invisible in the car. e car harness includes: (i) Harness Connector. is car-specific T-harness connects the car stock LKAS camera(s) to the harness box in order to intercept the messages coming from the LKAS camera(s).  [36] is an after-market solution that is described in detail in the next section.

Constant attention and temporary control
Level 3: conditional automation e driver is not responsible for driving the car when the specified automated driving features are engaged (i.e., the car can assume full control and operation during select parts of a journey when certain operating conditions are met) but the driver must reassume responsibility for driving when the features request it (e.g., the car is unable to safely make decisions). For instance, a car that can manage itself on a highway trip, excluding city driving and exits, can be classified as Level 3. is level of automation usually requires advanced sensors, hardware backups, and software to keep passengers safe.
Currently, there are no cars at this level, but many car manufacturers (e.g., Tesla) are developing cars that have such features. OP also has the potential to support Level 3 automation in the future. (ii) Comma Power v2: this uses the car's OBD-II port to power the harness box (which also powers the Comma device) and provides ECU fingerprinting capabilities to identify every car. (iii) Harness Box: this connects with the Comma device (using a USB-C cable), harness connector, and Comma Power v2 (using an RJ45 cable) in order to connect them all so the Comma device can read the LKAS messages and send steering commands. It also provides automatic switching to stock in situations where the Comma device is not working.
Compared with other systems, as shown in Table 3, OP is better with regard to some characteristics (highlighted in green) and falls short in terms of others (highlighted in red). Note that, Tesla cars are currently not available in KSA, but Tesla Autopilot was added to the comparison because it is currently one of the top self-driving systems and the most widely used system, with over two billion miles driven [52]. A unique feature of OP is that it can be installed for only $1,199 to $2,199 on a wide range of affordable supported cars. is provides flexibility because consumers can use the self-driving capability without being restricted to certain brands, styles, or budgets. Moreover, since it is open source, people can contribute by improving the system or porting unsupported cars. In fact, most of OP's supported cars were added to the system by people who analyzed their cars and altered OP to work with them. Moreover, since OP is based on AI technologies such as deep learning and neural Car does not require driver intervention (or the car can even operate entirely without a driver) when the automated driving features are engaged and the driver is not expected to take over driving when the automated driving features are in use. However, at this level, the automated driving features can only operate with some constraints and will not engage unless certain conditions are met. ese constraints may include restrictions to a specific geographical location (i.e., geofencing) or restrictions to certain speeds. A Level 4 car will likely still include driver controls such as a steering wheel and gas and brake pedals for instances in which a human may be required to assume control.
Currently, no Level 4 production cars are available to consumers, although some car makers (e.g., Honda) have announced plans to create products with Level 4 features in the near future [37]. ere are also some local Waymo driverless taxis currently operating in specific areas of the United States [38].
No attention and control necessary No attention and control required networks, it can make accurate decisions on previously unseen roads, and the more it is used, the more accurate it becomes when making decisions. is makes OP's lane centering much better than that of stock systems, which tend to zigzag aggressively (and only after crossing the side lanes), as shown in Figure 1. Even though OP is better than most stock self-driving systems, it is also continuously improving due to over-the-air updates, which can occur anytime and anywhere because the system is always connected to the Internet through Wi-Fi or cellular networks. Another unique feature of OP is that it actively tracks the driver's attention (day and night) using infrared cameras, rather than requiring drivers to periodically put their hands on the wheel. OP also supports dashcam functionality, and mapped recordings of drives are stored for up to one week for free in a web portal called Cabana, which also includes logs of all drive data recorded by car sensors. Drives are uploaded using Wi-Fi or over cellular networks.
Other features that OP has in common with other ADASs include automatic lane changing, cut-in detection, lead car following, LKAS, and ACC. However, the accuracy of these features in OP depends on the car, as some manufacturers place restrictions on the torque (i.e., steering angle) and minimum speeds allowed for the self-driving systems to engage. For example, Toyota cars are known to have high torque, which OP can exploit to take sharp curves, whereas Mazda cars have limited torque and lateral control and can only function above 45 km/h. Nevertheless, as shown in Figure 1 and Tables 2-3, OP generally remains the best ADAS.
Despite these positive features of OP, the system also has some drawbacks. First, it supports only limited self-driving functionalities (i.e., LKAS, automatic lane changing, and ACC) and is missing certain features that exist in other systems, such as pedestrian detection, automatic parking, and traffic sign recognition, among others. In addition, installing and maintaining OP can be challenging for users who are not tech-savvy, as these processes require running some wires and computer commands. Moreover, some car dealerships can void the car warranty if they notice that OP is installed because they believe it can cause some electrical issues. Hence, having OP installed can be inconvenient for users who prefer to perform regular maintenance at dealerships. Other issues that exist in all ADASs include the inability to make decisions as a human would in severe weather conditions (e.g., snow and heavy rain) because they rely on image recognition to read lanes, which can be difficult in harsh weather. Finally, since Comma devices are mounted on the windshield, they can increase vulnerability to car break-ins in some areas.

Analysis of Comma.ai's Openpilot
is section provides a detailed analysis of OP using data gathered from real drives in KSA of a car equipped with the   Journal of Advanced Transportation system. In this analysis, I used data representing approximately 13,000 km over 550 drives on different roads in KSA (mainly in the province of Makkah) using four different cars, hardware, and versions of OP. ese data are used in the following subsections to analyze the system's reliability, security, and safety, as well as the readiness of KSA roads for such systems.

Security and Safety.
Maintaining the security of OP is important in ensuring that the system does not pose any safety risk when controlling the car. Hence, OP can be accessed only through Secure Shell, generally considered a secure method for remote connections to manage the device. Other exploits available via physical access to the device itself do not pose any risk because the driver is considered a trusted user of the system. Nevertheless, if an intruder were to access the system through a means such as a steering torque interceptor (TI), which is a man-in-the-middle device that can increase the steering limit in the car EPS [53], Comma.ai will ban the account using interceptors from their servers. is is because Comma.ai believes that such devices change the parameters set by manufacturers after extensive testing by a large number of experienced Automotive Safety Integrity Level D (ASIL-D) hardware and software engineers, which cannot be guaranteed in after-market interceptors [54]. Any mistakes related to safety here can result in a car that the driver cannot physically steer. erefore, Comma.ai does not allow such devices to avoid bad publicity. However, it is still possible to use such interceptors with custom forks but without OP's online features.
OP also implements a safety model that consists of the following three rules [55]: (1) e driver must always be paying attention. is is enforced by the very effective driver monitoring system, which ensures that drivers are always paying attention and that their eyes are on the road; otherwise, the system will disengage and cannot be used again without restarting the car. (2) e driver must always be capable of immediately retaking manual control of the car. is is enforced by the safety code, which has a state variable controls-allowed that determines whether control messages can be sent on the CAN bus. e controlsallowed state can be entered by turning on cruise control and canceled by canceling cruise control.
Pressing the brake always immediately cancels the controls-allowed state. In stock OP, the gas pedal will always cancel the controls-allowed state as well, though there are custom forks that do not cancel the controls-allowed state on gas pressing in order to match the car ADAS behavior. (3) e car must not alter its trajectory so quickly that the driver cannot safely react. is is enforced by relying on the safety model implemented on car stock ADASs. us, only CAN messages designed for the car's stock ADAS are allowed, and an extra layer of safety is implemented to limit how these messages can be used.

Privacy.
OP's privacy policy does not comply with any privacy standards. e privacy policy presented on Comma.ai's website indicates that Comma.ai owns all data collected by OP [56]. ese data include personal information such as name, phone number, and e-mail address, as well as other information provided by the user. Comma.ai also owns data collected from the car's CAN bus network, such as car use, operation, and performance, including but not limited to the geolocation of the car, trip durations and routes, parking locations of the car, braking, and acceleration. Comma.ai also owns the driver monitoring recordings captured by the front camera, such as recordings of faces and eyes. However, the user has the option to disable saving front camera recordings in OP's settings. Although Comma.ai devices have microphone(s), they are not currently utilized. e data collected from the OP are used to provide, analyze, and improve OP's performance and security, to improve or modify other existing products and services, and to develop new products and services. Comma.ai may share personal information with service providers and business partners or with other third parties. e collected information can be also provided in response to lawful requests by government agencies. Email addresses can be also used to send advertisements and notifications, which can be stopped by unsubscribing from the mailing list.

Reliability.
Ensuring the reliability of self-driving systems is important to guarantee that they drive as safely as humans. To evaluate that, several tests were carried out using two different cars and two different Comma devices to ensure comprehensive analysis, as follows.

Test1: Comma Two.
In this test, I used the 2020 Mazda CX-9 (not officially supported by Comma.ai) and Comma two devkit. As shown in Table 4, out of 500 drives in my testing, the system was not able to engage (i.e., cannot be turned on or not ready to run) in 25 (5%) due to network issues. is could be related to bugs in the software update (e.g., some cars' fingerprints were replaced, which caused conflicts with other cars) or wiring issues. In some cases, the device cannot start working after the car was parked for a long time. However, in most cases, flashing or restarting the system was able to solve the issue. e remaining valid 475 drives were categorized into different groups based on the time of the day: daytime (298 drives) or night (86 drives).
is categorization is important in accurately determining the reliability of OP's vision model in recognizing the traffic environment at different brightness and glare levels. Drives in twilight periods were excluded from these datasets because they did not have much of an effect on the vision model decisions. In every category, I randomly took 20% (a widely recommended sample size [57]) of the recorded drives to analyze them in more detail. None of the recorded drives is longer than approximately 2 hours. is is due to the Comma Two limited storage, which made it impossible to obtain and analyze the logs of the drives.
Of the time series variables recorded by OP in the drive logs (approximately 3,779), which can be found in [58], I focused my analysis on only the most important variables that affect the reliability of the system and consequently pose safety risks if the driver is not paying attention. e descriptions of the variables used are shown in Table 5. e variable column shows the full path of the variable in the time series list. e source indicates whether this variable is recorded by OP or was read from the car stock system. e recorded variables were plotted using Plotjuggler, a plotting tool customized by Comma.ai to plot OP logs. e plots of the variables used in T1D1 (i.e., one of the six samples taken in the daytime) are shown in Figures 2-6, where the x-axis represents the duration of the drive. Figure 2 shows OP's ability to recognize the left and right lanes where 1 (in the y axis) indicates that the model is 100% confident a lane exists. is is crucial to send commands for centering the car within lanes. Values below 0.5 indicate that the system did not detect the lanes and it will show that on  the Comma device screen. In such cases, it will not send centering commands and the driver may feel that the car is slightly not centered inside the lanes. Figure 3 shows OP's ability to recognize a lead car that is one car away where 1 (in the y axis) indicates that the model is 100% confident a lead car, that is one car away, exists. is is important to keep a safe distance away (brake value should be 1) from the front car and brake accordingly to avoid accidents. Figure 4 shows OP's monitoring of driver awareness (i.e., paying attention to the road) where 1 (in the y axis) indicates that the model is 100% confident the driver's eyes are on the road. is is to ensure that the system warns the driver to keep their eyes on the road. Driver monitoring figures are not present for all test drives if the system did not detect any driver distraction (i.e., the driver awareness value was 1 the whole drive). Figure 5 shows the steering degrees taken by the car. e figure highlights cases where the car needed to steering degrees that exceeds the car turn degree limit set by the car's electric power steering (EPS) (i.e., represented by the steering ratio in the left and right directions). Hence, such    Journal of Advanced Transportation turns that OP could not do, were assisted by the driver. e fewer the turns that exceed the car limit the better. Figure 6 shows the temperatures in Celsius (on the y axis) of the Comma device. is shows cases where OP stops working due to high temperatures. e plots of the other selected drives in Test 1 are provided in the where F (Fail) represents the number of segments in which OP did not operate as a human would have, including when  incorrect decisions were made by OP or the car or when OP could not make decisions due to the uncertainty of the vision model (e.g., due to unclear lanes) in which the value was less than 0.6, and A (All) represents the total number of segments in which OP was engaged in the drive.
(1) Test1 Results and Discussion. Table 6 shows the analysis results of all drives in Test 1. Overall, OP was effective in 82% of cases. is means that it was able to drive as a human would most of the time. For the remaining 18% of the time, the driver needed to take control, which should not be a concern given the system's very effective driver monitoring. I did not see much difference between daytime and nighttime drives in terms of overall success rate. In terms of lead car detection and braking (i.e., ACC), night drives showed better success rates. is is likely due to the combination of OP's vision model and the car's ACC radar. is is likely because the vision model sees less background clutter in the images captured by the camera at night. e background would be mostly black because distant objects will not be visible at night. Another reason could be less traffic at night. However, smoother braking would be better to avoid rearending accidents. is would require enhancing the detection of lead cars from a far distance and slowing the speed   accordingly. In terms of lane recognition, which is crucial for lane centering, I noticed that on roads without lanes, OP was able to detect only the edge of the road as a left or right lane. Hence, the success rates of both lanes, most of the time, did not match. Regarding driver monitoring, I found that OP was able to constantly and very effectively monitor the   driver's awareness even at night (using the infrared front camera). It was able to detect eyes closing, looking left or right, or looking at one's phone. However, I found it to be very cautious and overalerts too quickly for driver activities that do not necessarily indicate driver distraction, such as drinking from large bottles that can block the driver's face. e main limitation of OP was its inability to steer in sharp turns. is is due to the limited torque available for LKAS in the EPS of the testing car, which is approximately 17 degrees. Other cars, like Toyotas, are known to have better torque that allows them to take sharp turns. However, this should not be an issue on highways, where OP is mostly used. In addition, if OP cannot fully steer during any turn, the system always predicts it and alerts the driver to take control (as shown in Figure 40). Regarding thermal management, Comma Two CPU and GPU temperatures are limited to 75 C degrees and try always to lower the temperatures. is is possibly to avoid overheating issues that Comma Two suffers in the early versions of OP.

Test2: Comma ree.
To test OP in different hardware and in other cars, I also tested the system on Comma ree (the latest hardware from Comma.ai as of April 2022) that was installed in the 2019 Hyundai Tucson. is can provide insights into whether the new and improved cameras can provide better accuracy. As of OP v0.8.13, the system does not use the wide camera for controlling the car. However, the high-resolution narrow camera combined with the better processor can provide clearer images that can improve the speed and accuracy of the system decisions. is of course requires more storage that is available in Comma ree, which can store up to 15 hours of drives compared to only 2 hours in Comma Two. For that, in this test, I focused on longer drives (longer than 20 miles) to test how the system performs under load for long periods of time. As shown in Table 7, I analyzed six randomly selected drives as done in Test1. One of the drives (i.e., T2N1) was made in rainy weather, which allowed analyzing the system in different weather conditions. e same variables in Test1 were plotted using Plotjuggler. e plots of T2D1 (i.e., one of the three samples taken in the daytime) are shown in Figures 41-44, where the x-axis represents the duration of the drive. e plots of the other drivers in Test2 are in Figures 45-60.
(1) Test2 Results and Discussion. As shown in Table 8, the overall results of OP in Test2 were better than in Test1. It also never failed to engage. is could be due to better design and processing capabilities in Comma ree. Figure 41 shows that OP in Comma ree was able to recognize the left and right lanes very effectively, where 1 (in the y-axis) indicates that the model is 100% confident a lane exists. As shown in Table 8, after performing the same method in Test1, the SR was very high in both left and right lanes detection despite the heavy rain. is could be related to the better cameras in Comma ree. Figure 42 shows OP's ability to recognize a lead car where 1 (in the y-axis) indicates that the model is 100% confident a lead car that is one car away exists. All logs from Comma ree on the testing car did not show the braking status to check if the car was braking after detecting a lead car. is could be related to the test car, or the OP system, which is known to have inconsistencies in logs for different car makes and models. However, replaying the drive in OP debugging interface and comparing speed before and after detecting a lead car, showed that longitudinal control was also improved but still slows down very aggressively. Driver monitoring, which was accurate in Test1, shows slightly better results. Due to the higher torque limit in the Test2 car (i.e., about 20 degrees), the driver required fewer assisted turns. Although the SR was improved compared to Test1 in steering degrees, as shown in Table 9, it is still not near what Torque interceptors can do with the higher steering torque. In terms of thermal management, Comma ree seems to sustain high GPU and CPU temperatures (about 80 C) more than Comma Two. However, the camera lenses tend to get too hot, which might reduce the device's longevity in the long term.

Test 3: Torque Interceptor (TI).
To overcome some of the limitations in the Test1 used car, I installed TI, which is a man-in-the-middle device that can unlock some features that are disabled in the car EPS (e.g., steering lock-out, stopand-go) [53]. e interceptor costs around $399. As shown in Table 9, the TI enhances the usability of the stock OP system by providing Stop and Go and removing steering lock-out. Higher steering torque was also noticeable, which is almost double the steering degrees. e testing car can now take about 90 degrees in custom forks or with driver assistance without disengaging. Unfortunately, as described earlier, since Openpilot does not allow the use of such devices, I was unable to obtain any logs to measure the    T1D1 T1D2 T1D3 T1D4 T1D5 T1D6 T1D1-T1D6 T1N1 T1N2 T1N1-T1N2 Overall average (%) A � 32 (%) difference.
is is because if the recorded logs reach the Comma.ai servers, Comma.ai will ban the associated account, which is necessary to run the system. Nevertheless, the difference is very noticeable with TI installed compared to the stock OP system. However, with the increase in steering torque, the driver needs to pay attention while the system is engaged at low speeds (steering torque is higher at low speed than at high speed). I have faced a few situations where the car tried to drive too close to the edge of the road very quickly. Such cases could not be examined due to the lack of logs. is should not be an issue because OP is rarely used inside the city at low speeds. However, for this reason, as well as the lack of OP online features such as cloud storage of drives, logs, and OTA updates, I do not recommend using such TIs before such limitations are addressed by Comma.ai and/or the community developers.

Test 4:
Comma ree vs. Comma Two. I compared two identical drives (in terms of route and time as well as the testing car) using Comma Two and Comma ree. e goal is to find if there is any difference in terms of reliability. However, without testing the "big model," a new vision model announced by Comma.ai that will be only available for Comma ree, I did not find any noticeable difference in the self-driving capabilities worthy of investigation. e big model can utilize the HDR images captured by Comma ree to be used in training on the full 1024 × 512 camera frame instead of the cropped 512 × 256 middle frame it currently has. e new model also used frames annotated by the community in the Comma Pencil project to better train the model to detect other cars and objects on the road [59]. erefore, I anticipate the difference between the two hardware will be noticed and can be practically examined in the future.

Overall Results and Discussion.
e goal of this analysis is to comprehensively assess OP reliability in the context of KSA from the driver's point of view to highlight the gaps in the system or KSA infrastructure that can be improved. e above-mentioned tests were performed in different weather conditions, hardware, and is section discusses the common findings and limitations in all tests. For that, I analyzed all disengages in the testing drives to understand why the driver did not trust OP and decided to take control. Two drivers were asked (a male and a female). e reasons can be summarized as follows: is makes OP cannot work immediately after starting the car, and it takes at least 2-5 minutes before it is on and running. Of course, drivers can always disconnect the Comma device and place it away from the sunlight; but this is inconvenient. Also, with the upcoming big model, overheating issues might be noticed, especially in Comma Two, which has limited processing capabilities.

Dataset
One of the main obstacles for developers is the lack of real datasets that can be used for ML training and testing. To the best of our knowledge, there is no detailed dataset that includes extensive logs and video recordings of drives in KSA. is is important to understand how the road edges or lanes look. is can help solve issues related to the region, such as higher speed limits in which OP cannot operate (i.e., above 145 km/h (90 mp/h) as shown in Figure 61), or heat issues. erefore, in this paper, I provide a dataset of more   T2D1  T2D2  T2D3  T2D1-T2D3  T2N1  T2N2  T2N1-T2N2 Overall  Note that, for privacy reasons, I omitted dcamera.hevc, a H.265 encoded video of the driver that can be captured by the front camera(s) of the Comma device. e dataset can be found on the Github page (https://github.com/fsalsubaei/ KSA_Drives). e dataset will be updated regularly to include labels (day, night, sunny, rainy, long, short, etc), logs decoding, more drives, different cars, and regions.

Recommendations
Despite the continuous improvements in OP, there is always room for enhancements to make the driving experience as humanlike as possible and reduce the defects I noticed during my extensive experiments using the system. In this section, I provide general recommendations to address the previously discussed limitations of OP. In terms of lateral control, the lack of steering torque should be addressed by Comma.ai as well as car manufacturers to reach safe torque limits after thorough testing. Alternatively, rather than banning the use of TIs, which made the use of drives with TIs cannot be logged and improved, Comma.ai should work with TI developers to ensure that such devices are rigorously tested according to industry standards. Comma.ai should also allow logging to continuously monitor the TI drives to detect and resolve any safety concerns as soon as they are discovered. is would eliminate the limitations discussed before, especially in car brands where ADASs functionalities are limited.
Moreover, since lateral control is related to the current infrastructure in KSA, more contributions are needed from the OP community in KSA to provide more data to train and better tune the system parameters to suit the current infrastructure, specifically, to detect the current road speed and upcoming turns so that the car can slow down in order to take sharp turns. Even though Comma.ai is working on big model, which can hopefully improve the detection of road edges and driving on lane less roads, cities should also contribute to marking lanes and road signs because this will not only help with OP equipped cars but also all cars with stock ADAS systems. is is a common obstacle for selfdriving technologies worldwide, even in developed countries like the U.S. [63]. Hence, collaboration is required among all stakeholders, such as the ministry of transportation, technology companies, and car manufacturers to agree on standards that ensure the readiness of the infrastructure for safe mass self-driving adoption. Such collaboration should also consider the fact that there is less tendency for selfdriving cars adoption in rural areas, and individuals with longer commutes tend to prefer self-driving cars. erefore, distant houses are recommended to increase the urban spread and car miles traveled, which will consequently increase traffic safety.  Journal of Advanced Transportation Longitudinal control can also be better tuned to adapt to driving behavior in KSA, where tailgating is very common, specifically, to detect stopped cars and to reduce the distance from the lead car to avoid cut-in lanes and hard braking, which can result in rear-end accidents. For this reason, it is important that Comma.ai provides detailed and consistent logs and documentation for nontechnical users to attract new users, which will ultimately enhance the model's accuracy by providing more training data, which will benefit all users. Finally, to overcome the thermal issues, until Comma.ai produces white color devices, I recommend painting the Comma devices with white color because this reduces the issue of overheating under direct sunlight. Comma.ai should also invest more in developing better cooling mechanisms and materials to keep up with car industry parts that can sustain high temperatures.

Conclusion and Future Work
Self-driving technologies have developed rapidly in recent years. Such technologies and their application in KSA have not been thoroughly studied in the past. us, this paper analyzed the security, privacy, safety, and reliability of OP, the most advanced self-driving technology available in KSA. e analysis showed that OP is very effective despite some limitations related to equipped cars (e.g., steering lock-out) and the environment (e.g., no clear lanes). In addition, despite the secure implementation of OP, more attention is required from Comma.ai to improve its privacy policy to ensure that no sensitive information, such as GPS coordinates and drivers' identities, is used unnecessarily.
Future work includes increasing the size of the testing sample by including the OP upcoming big model, more variables, different cities, and different cars and comparing the other systems available in the Saudi market to OP. Such work could help early adopters of such advanced systems to learn about their effectiveness in KSA.
is is important because many people, even tech-savvy users, are unaware of such systems or think they are unnecessary. Technology and automobile companies might find this work important in understanding the current self-driving landscape and how it can be improved to increase revenue and save lives. Governments and municipalities can also benefit from this work to set legislation and improve current infrastructure to better facilitate such technologies. I believe the large-scale adoption of self-driving technologies can help detect current areas for improvement to enhance safety and reduce the high number of traffic accident fatalities in KSA.

Data Availability
Test data (more than 500GB) are available on the cited Github repository.

Conflicts of Interest
e authors declare that they have no conflicts of interest.