Statistical Approaches Used in Studies Evaluating the Reliability of Autonomous Vehicles Based on Disengagements and Reaction Times

Year 2024, Volume: 8 Issue: 3, 279 - 287, 30.09.2024

Ferhan Baş Kaman , Hülya Olmuş

https://doi.org/10.30939/ijastech..1480056

Abstract

The rapid development of autonomous vehicle (AV) technology highlights the critical importance of enhancing the reliability of these vehicles. Due to the need to test the reliability of AVs, since 2014, the California Department of Motor Vehicles has permitted autonomous vehicle manufacturers to establish an AV Testing program, enabling them to test automated systems on the transportation network. With this, studies on the reliability of AVs have increased rapidly. The most emphasized issues regarding the reliability of AVs have been disengagements, accidents, and reaction times. In this study, disengagements and reaction times are categorized and explained in detail according to the data type, company, period, and statistical method. The data used in the studies cover the years 2014-2020. When examining studies on the reliability of AVs, until 2018, inferences were generally made using real data and descriptive statistics, particularly with methods such as correlation analysis and calculation of disengagements per mile, which investigates the relationship between distance traveled and disengagements. However, since 2018, machine learning has gained importance in evaluating AV reliability. It has been observed that regression, classification, and decision trees were frequently used during this period. Techniques such as deep transfer learning, text mining, and natural language processing also stand out. Furthermore, Software Reliability Growth Models are used to measure software reliability, playing an essential role in evaluating, analyzing, and improving the performance and reliability of AVs. This study aims to reveal the development and diversity of the statistical methods used to determine AV reliability. Additionally, this study aims to guide and provide insights to researchers in the field about the statistical approaches they can utilize.

Keywords

Autonomous vehicle reliability, Disengagements, Reaction times, Statistical methods

References

• [1] SAE International. Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles; SAE J3016_202104; 2021. https://www.sae.org/standards/content/j3016_202104
• [2] Koopman, P., Wagner, M. Autonomous vehicle safety: An interdisciplinary challenge. IEEE Intell. Transp. Syst. Mag.. 2017, 9(1), 90–96. https://doi.org/10.1109/MITS.2016.2583491
• [3] Burton, S., Habli, I., Lawton, T., McDermid, J.,Morgan, P., Porter, Z. Mind the gaps: assuring the safety of autonomous systems from an engineering, ethical, and legal perspective. Artificial Intelligence. 2020, 279, 103201. https://doi.org/10.1016/j.artint.2019.103201
• [4] Kalra, N., Paddock, S.M. Driving to safety: how many miles of driving would it take to demonstrate autonomous vehicle reliability? Transportation Research Part A: Policy and Practice. 2016, 94, 182–193.
• [5] California Department of Motor Vehicles (CA DMV). Article 3.7–Autonomous Vehicles. Title 13, Division 1, Par. 227. 2016. https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/testing
• [6] Lv, C., Cao, D., Zhao, Y., Auger, D.J., Sullman, M., Wang, Dutka, H. L. M., Skrypchuk, L., Mouzakitis, A. Analysis of autopilot disengagements occurring during autonomous vehicle testing. IEEE∕CAA Journal of Automatica Sinica. 2018, 5(1), 58–68. http://dx.doi.org/10.1109/JAS.2017.7510745
• [7] Dixit, V.V., Chand, S., Nair, D.J. Autonomous vehicles: disengagements, accidents and reaction times. PLoS ONE. 2016, 11(12): e0168054. https://doi.org/10.1371/journal.pone.0168054
• [8] Wood, A. Software Reliability Growth Models. Tandem, Technical Report 96.1, Tandem Computers, 1996; Cupartino, CA.
• [9] Merkel, R. Software reliability growth models predict autonomous vehicle disengagement events. arXiv: 1812.08901.2018. https://doi.org/10.48550/arXiv.1812.08901
• [10] Banerjee, S.S., Jha, S., Cyriac, J., Kalbarczyk, Z.T., Iyer, R.K. Hands off the wheel in autonomous vehicles? A systems perspective on over a million miles of field data. In 2018 48th Annual IEEE∕IFIP International Conference on Dependable Systems and Networks (DSN). 2018, 586–597. https://doi.org/10.1109/DSN.2018.00066
• [11] Leveson, N. Engineering a safer world: Systems thinking applied to safety. MIT press; 2011.
• [12] Favarò, F., Eurich, S., Nader, N. Autonomous vehicles disengagements: trends, triggers, and regulatory limitations. Accident Analysis & Prevention. 2018, 110, 136–148. https://doi.org/10.1016/j.aap.2017.11.001
• [13] Zhao, X., Robu,V., Flynn,D., Salako, K., Strigini, L. Assessing the safety and reliability of autonomous vehicles from road testing. In 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE). 2019, 13–23. https://doi.org/10.1109/ISSRE.2019.00012
• [14] Brocklehurst, S., Chan, P. Y., Littlewood, B., Snell, J. (1990). Recalibrating software reliability models. IEEE Transactions on Software Engineering. 1990, 16(4), 458–470. http://dx.doi.org/10.1109/32.54297
• [15] Brocklehurst S., Littlewood, B. Techniques for prediction analysis and recalibration. in Handbook of Software Reliability Eng., M. Lyu, Ed. McGraw-Hill & IEEE Computer Society Press; 1996.
• [16] Wang, S. and Li, Z. Exploring causes and effects of automated vehicle disengagement using statistical modeling and classification tree based on field test data. Accident Anal. Prevention. 2019, 129, 44-54. https://doi.org/10.1016/j.aap.2019.04.015
• [17] Weng, J., Meng, Q. Decision tree-based model for estimation of work zone capacity. Transportation Research Record: Journal of the Transportation Research Board. 2011, 2257, 40–50. http://dx.doi.org/10.3141/2257-05
• [18] Boggs, A. M., Arvin, R., Khattak, A.J. Exploring the who, what, when, where, and why of automated vehicle disengagements. Accident Anal. Prevention. 2020, 136, 105406. https://doi.org/10.1016/j.aap.2019.105406
• [19] Wali, B., Khattak, A.J., Khattak, A.J. A heterogeneity based case-control analysis of motorcyclist’s injury crashes: evidence from motorcycle crash causation study. Accident Analysis & Prevention. 2018, 119, 202–214. http://dx.doi.org/10.1016/j.aap.2018.07.024
• [20] Azimi, G., Asgari, H., Rahimi, A., Jin, X.. Investigation of heterogeneity in severity analysis for large truck crashes. 98th Annual Meeting of the Transportation Research Board. 2019, Washington, D.C. United States.
• [21] Zhang, Y., X. J. Yang, X. J., Zhou, F. Disengagement Cause-and-Effect Relationships Extraction Using an NLP Pipeline. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(11), 21430-21439, https://doi.org/10.1109/TITS.2022.3186248
• [22] Khattak, Z. H., Fontaine, M.D., Smith, B. L. Exploratory Investigation of Disengagements and Crashes in Autonomous Vehicles Under Mixed Traffic: An Endogenous Switching Regime Framework. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(12), 7485-7495. https://doi.org/10.1109/TITS.2020.3003527
• [23] Min, J., Hong, Y., King, C. B., Meeker W. Q. Reliability analysis of artificial intelligence systems using recurrent events data from autonomous vehicles. Journal of the Royal Statistical Society: Series C (Applied Statistics). 2022, 1–30. http://dx.doi.org/10.1111/rssc.12564
• [24] Du, N., Kim, J., Zhou, F., Pulver, E., Tilbury, D.M., Robert, L.P., Pradhan, A.K., Yang, X.J. Evaluating effects of cognitive load, takeover request lead time, and traffic density on drivers’ takeover performance in conditionally automated driving. In Proceedings of the 12th International Conference on Automotive User Interfaces and Interactive Vehicular Applications; 2020; Washington DC, USA. http://dx.doi.org/10.1145/3409120
• [25] Gold, C., Körber, M., Lechner, D., Bengler, K. Taking over control from highly automated vehicles in complex traffic situations: The role of traffic density. Human Factors: The-Journal of the Human Factors and Ergonomics Society. 2016, 58(4), 642–652. http://dx.doi.org/10.1177/0018720816634226
• [26] Du, N., Zhou, F., Pulver, E. M., Tilbury, D.M., Robert, L.P., Pradhan, A.K., Yang, X.J. Examining the effects of emotional valence and arousal on takeover performance in conditionally automated driving. Transportation Research Part C: Emerging Technologies. 2020, 112, 78-87. https://doi.org/10.1016/j.trc.2020.01.006
• [27] Clark, H., Feng, J. Age differences in the takeover of vehicle control and engagement in non-driving-related activities in simulated driving with conditional automation. Accident Analysis & Prevention. 2017, 106, 468–479. http://dx.doi.org/10.1016/j.aap.2016.08.027
• [28] Wandtner, B., Schömig, N., Schmidt, G. Effects of non-driving related task modalities on takeover performance in highly automated driving. Human Factors: The-Journal of the Human Factors and Ergonomics Society. 2018, 60(6), 870–881. https://doi.org/10.1177/0018720818768199
• [29] Dogan, E., Honnêt, V., Masfrand, S., Guillaume, A. Effects of non-driving-related tasks on takeover performance in different takeover situations in conditionally automated driving. Transportation Research Part F: Traffic Psychology and Behavior. 2019, 62, 494-504. https://doi.org/10.1016/j.trf.2019.02.010
• [30] Hu, W., Zhang, T., Zhang, Y., Chan, A.H.S. Non-driving-related tasks and drivers’ takeover time: A meta-analysis. Transportation Research Part F: Traffic Psychology and Behavior. 2024, 103, 623-637. https://doi.org/10.1016/j.trf.2024.05.012
• [31] Hecker, S., Dai, D., Van Gool, L. Failure prediction for autonomous driving. In 2018 IEEE Intelligent Vehicles Symposium (IV). 2018. https://doi.org/10.48550/arXiv.1805.01811 [32] California Department of Motor Vehicles (CA DMV). Disengagement reports. 2024.