Statistical Approaches Used in Studies Evaluating the Reliability of Autonomous Vehicles Based on Disengagements and Reaction Times

Abstract

The rapid development of autonomous vehicle (AV) technology highlights the critical importance of enhancing the reliability of these vehicles. Due to the need to test the reliability of AVs, since 2014, the California Department of Motor Vehicles has permitted autonomous vehicle manufacturers to establish an AV Testing program, enabling them to test automated systems on the transportation network. With this, studies on the reliability of AVs have increased rapidly. The most emphasized issues regarding the reliability of AVs have been disengagements, accidents, and reaction times. In this study, disengagements and reaction times are categorized and explained in detail according to the data type, company, period, and statistical method. The data used in the studies cover the years 2014-2020. When examining studies on the reliability of AVs, until 2018, inferences were generally made using real data and descriptive statistics, particularly with methods such as correlation analysis and calculation of disengagements per mile, which investigates the relationship between distance traveled and disengagements. However, since 2018, machine learning has gained importance in evaluating AV reliability. It has been observed that regression, classification, and decision trees were frequently used during this period. Techniques such as deep transfer learning, text mining, and natural language processing also stand out. Furthermore, Software Reliability Growth Models are used to measure software reliability, playing an essential role in evaluating, analyzing, and improving the performance and reliability of AVs. This study aims to reveal the development and diversity of the statistical methods used to determine AV reliability. Additionally, this study aims to guide and provide insights to researchers in the field about the statistical approaches they can utilize.

Keywords

• Autonomous vehicle reliability
• Disengagements
• Reaction times
• Statistical methods

References

1. [1] SAE International. Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles; SAE J3016_202104; 2021. https://www.sae.org/standards/content/j3016_202104
2. [2] Koopman, P., Wagner, M. Autonomous vehicle safety: An interdisciplinary challenge. IEEE Intell. Transp. Syst. Mag.. 2017, 9(1), 90–96. https://doi.org/10.1109/MITS.2016.2583491
3. [3] Burton, S., Habli, I., Lawton, T., McDermid, J.,Morgan, P., Porter, Z. Mind the gaps: assuring the safety of autonomous systems from an engineering, ethical, and legal perspective. Artificial Intelligence. 2020, 279, 103201. https://doi.org/10.1016/j.artint.2019.103201
4. [4] Kalra, N., Paddock, S.M. Driving to safety: how many miles of driving would it take to demonstrate autonomous vehicle reliability? Transportation Research Part A: Policy and Practice. 2016, 94, 182–193.
5. [5] California Department of Motor Vehicles (CA DMV). Article 3.7–Autonomous Vehicles. Title 13, Division 1, Par. 227. 2016. https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/testing
6. [6] Lv, C., Cao, D., Zhao, Y., Auger, D.J., Sullman, M., Wang, Dutka, H. L. M., Skrypchuk, L., Mouzakitis, A. Analysis of autopilot disengagements occurring during autonomous vehicle testing. IEEE∕CAA Journal of Automatica Sinica. 2018, 5(1), 58–68. http://dx.doi.org/10.1109/JAS.2017.7510745
7. [7] Dixit, V.V., Chand, S., Nair, D.J. Autonomous vehicles: disengagements, accidents and reaction times. PLoS ONE. 2016, 11(12): e0168054. https://doi.org/10.1371/journal.pone.0168054
8. [8] Wood, A. Software Reliability Growth Models. Tandem, Technical Report 96.1, Tandem Computers, 1996; Cupartino, CA.

9. [9] Merkel, R. Software reliability growth models predict autonomous vehicle disengagement events. arXiv: 1812.08901.2018. https://doi.org/10.48550/arXiv.1812.08901
10. [10] Banerjee, S.S., Jha, S., Cyriac, J., Kalbarczyk, Z.T., Iyer, R.K. Hands off the wheel in autonomous vehicles? A systems perspective on over a million miles of field data. In 2018 48th Annual IEEE∕IFIP International Conference on Dependable Systems and Networks (DSN). 2018, 586–597. https://doi.org/10.1109/DSN.2018.00066
11. [11] Leveson, N. Engineering a safer world: Systems thinking applied to safety. MIT press; 2011.
12. [12] Favarò, F., Eurich, S., Nader, N. Autonomous vehicles disengagements: trends, triggers, and regulatory limitations. Accident Analysis & Prevention. 2018, 110, 136–148. https://doi.org/10.1016/j.aap.2017.11.001
13. [13] Zhao, X., Robu,V., Flynn,D., Salako, K., Strigini, L. Assessing the safety and reliability of autonomous vehicles from road testing. In 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE). 2019, 13–23. https://doi.org/10.1109/ISSRE.2019.00012
14. [14] Brocklehurst, S., Chan, P. Y., Littlewood, B., Snell, J. (1990). Recalibrating software reliability models. IEEE Transactions on Software Engineering. 1990, 16(4), 458–470. http://dx.doi.org/10.1109/32.54297
15. [15] Brocklehurst S., Littlewood, B. Techniques for prediction analysis and recalibration. in Handbook of Software Reliability Eng., M. Lyu, Ed. McGraw-Hill & IEEE Computer Society Press; 1996.
16. [16] Wang, S. and Li, Z. Exploring causes and effects of automated vehicle disengagement using statistical modeling and classification tree based on field test data. Accident Anal. Prevention. 2019, 129, 44-54. https://doi.org/10.1016/j.aap.2019.04.015
17. [17] Weng, J., Meng, Q. Decision tree-based model for estimation of work zone capacity. Transportation Research Record: Journal of the Transportation Research Board. 2011, 2257, 40–50. http://dx.doi.org/10.3141/2257-05
18. [18] Boggs, A. M., Arvin, R., Khattak, A.J. Exploring the who, what, when, where, and why of automated vehicle disengagements. Accident Anal. Prevention. 2020, 136, 105406. https://doi.org/10.1016/j.aap.2019.105406
19. [19] Wali, B., Khattak, A.J., Khattak, A.J. A heterogeneity based case-control analysis of motorcyclist’s injury crashes: evidence from motorcycle crash causation study. Accident Analysis & Prevention. 2018, 119, 202–214. http://dx.doi.org/10.1016/j.aap.2018.07.024
20. [20] Azimi, G., Asgari, H., Rahimi, A., Jin, X.. Investigation of heterogeneity in severity analysis for large truck crashes. 98th Annual Meeting of the Transportation Research Board. 2019, Washington, D.C. United States.
21. [21] Zhang, Y., X. J. Yang, X. J., Zhou, F. Disengagement Cause-and-Effect Relationships Extraction Using an NLP Pipeline. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(11), 21430-21439, https://doi.org/10.1109/TITS.2022.3186248
22. [22] Khattak, Z. H., Fontaine, M.D., Smith, B. L. Exploratory Investigation of Disengagements and Crashes in Autonomous Vehicles Under Mixed Traffic: An Endogenous Switching Regime Framework. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(12), 7485-7495. https://doi.org/10.1109/TITS.2020.3003527
23. [23] Min, J., Hong, Y., King, C. B., Meeker W. Q. Reliability analysis of artificial intelligence systems using recurrent events data from autonomous vehicles. Journal of the Royal Statistical Society: Series C (Applied Statistics). 2022, 1–30. http://dx.doi.org/10.1111/rssc.12564
24. [24] Du, N., Kim, J., Zhou, F., Pulver, E., Tilbury, D.M., Robert, L.P., Pradhan, A.K., Yang, X.J. Evaluating effects of cognitive load, takeover request lead time, and traffic density on drivers’ takeover performance in conditionally automated driving. In Proceedings of the 12th International Conference on Automotive User Interfaces and Interactive Vehicular Applications; 2020; Washington DC, USA. http://dx.doi.org/10.1145/3409120
25. [25] Gold, C., Körber, M., Lechner, D., Bengler, K. Taking over control from highly automated vehicles in complex traffic situations: The role of traffic density. Human Factors: The-Journal of the Human Factors and Ergonomics Society. 2016, 58(4), 642–652. http://dx.doi.org/10.1177/0018720816634226
26. [26] Du, N., Zhou, F., Pulver, E. M., Tilbury, D.M., Robert, L.P., Pradhan, A.K., Yang, X.J. Examining the effects of emotional valence and arousal on takeover performance in conditionally automated driving. Transportation Research Part C: Emerging Technologies. 2020, 112, 78-87. https://doi.org/10.1016/j.trc.2020.01.006
27. [27] Clark, H., Feng, J. Age differences in the takeover of vehicle control and engagement in non-driving-related activities in simulated driving with conditional automation. Accident Analysis & Prevention. 2017, 106, 468–479. http://dx.doi.org/10.1016/j.aap.2016.08.027
28. [28] Wandtner, B., Schömig, N., Schmidt, G. Effects of non-driving related task modalities on takeover performance in highly automated driving. Human Factors: The-Journal of the Human Factors and Ergonomics Society. 2018, 60(6), 870–881. https://doi.org/10.1177/0018720818768199
29. [29] Dogan, E., Honnêt, V., Masfrand, S., Guillaume, A. Effects of non-driving-related tasks on takeover performance in different takeover situations in conditionally automated driving. Transportation Research Part F: Traffic Psychology and Behavior. 2019, 62, 494-504. https://doi.org/10.1016/j.trf.2019.02.010
30. [30] Hu, W., Zhang, T., Zhang, Y., Chan, A.H.S. Non-driving-related tasks and drivers’ takeover time: A meta-analysis. Transportation Research Part F: Traffic Psychology and Behavior. 2024, 103, 623-637. https://doi.org/10.1016/j.trf.2024.05.012
31. [31] Hecker, S., Dai, D., Van Gool, L. Failure prediction for autonomous driving. In 2018 IEEE Intelligent Vehicles Symposium (IV). 2018. https://doi.org/10.48550/arXiv.1805.01811 [32] California Department of Motor Vehicles (CA DMV). Disengagement reports. 2024.