Enhancing Autonomous Vehicle Safety Through Chaid Modeling: Influential Factors, Seasonal Variations, and Systematic Limitations

Abstract

The growing prominence of artificial intelligence has driven transformative innovations across sectors, with autonomous vehicles representing a salient manifestation of this technological shift. The reliability of autonomous vehicles plays a crucial role in determining their societal acceptance and large-scale deployment. Within this context, disengagement data serve as an objective indicator of system reliability. A rigorous analysis of disengagement data is essential for evaluating the real-world performance and operational reliability of autonomous vehicles. Such data circumstances necessitate human intervention, thereby revealing system vulnerabilities and opportunities for improvement. Consequently, precise and transparent disengagement analyses are vital for advancing AV technology and strengthening safety. This study investigates the determinants of disengagements and contrasts human-initiated with system-initiated events. Drawing on 17,406 reports (2021–2023), CHAID models identified key triggers including environmental context, system limitations, and operational conditions. The study identified key determinants, including planning inconsistencies, detection failures, and hardware malfunctions, and revealed clear seasonal variations, with disengagements peaking in summer and autumn and declining in winter and spring. Validated CHAID models demonstrated high accuracy, underscoring the importance of comprehensive training and testing across diverse conditions to enhance effectiveness and safety.

Keywords

• Reliability
• Autonomous Vehicles
• Quantitative Decision-Making Methods
• Artificial Intelligence

References

1. SAE International, “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles,” SAE J3016_202104, Apr. 2021. [Online]. Available: https://www.sae.org/standards/content/j3016_202104/
2. P. Koopman and M. Wagner, “Autonomous vehicle safety: An interdisciplinary challenge,” IEEE Intell. Transp. Syst. Mag., vol. 9, no. 1, pp. 90–96, Jan. 2017.
3. S. Burton et al., “Mind the gaps: assuring the safety of autonomous systems from an engineering, ethical, and legal perspective,” Artif. Intell., vol. 279, Feb. 2020, Art. no. 103201.
4. N. Kalra and S. M. Paddock, “Driving to safety: how many miles of driving would it take to demonstrate autonomous vehicle reliability?” Transp. Res. Part A Policy Pract., vol. 94, pp. 182–193, Dec. 2016.
5. California Department of Motor Vehicles, “Article 3.7–Autonomous Vehicles. Title 13, Division 1, Par. 227,” Jul. 2019. [Online]. Available: https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/testing/
6. V. V. Dixit, S. Chand, and D. J. Nair, “Autonomous vehicles: disengagements, accidents and reaction times,” PLoS ONE, vol. 11, no. 12, Dec. 2016, Art. no. e0168054.
7. C. Lv et al., “Analysis of autopilot disengagements occurring during autonomous vehicle testing,” IEEE/CAA J. Autom. Sinica, vol. 5, no. 1, pp. 58–68, Jan. 2018.
8. F. Favarò, S. Eurich, and N. Nader, “Autonomous vehicles disengagements: trends, triggers, and regulatory limitations,” Accid. Anal. Prev., vol. 110, pp. 136–148, Jan. 2018.

9. S. Wang and Z. Li, “Exploring causes and effects of automated vehicle disengagement using statistical modeling and classification tree based on field test data,” Accid. Anal. Prev., vol. 129, pp. 44–54, Aug. 2019.
10. A. M. Boggs, R. Arvin, and A. J. Khattak, “Exploring the who, what, when, where, and why of automated vehicle disengagements,” Accid. Anal. Prev., vol. 136, Mar. 2020, Art. no. 105406.
11. S. S. Banerjee, S. Jha, J. Cyriac, Z. T. Kalbarczyk, and R. K. Iyer, “Hands off the wheel in autonomous vehicles? A systems perspective on over a million miles of field data,” in Proc. 48th Annu. IEEE/IFIP Int. Conf. Dependable Syst. Netw. (DSN), Luxembourg, 2018, pp. 586–597.
12. Z. H. Khattak, M. D. Fontaine, and B. L. Smith, “Exploratory investigation of disengagements and crashes in autonomous vehicles under mixed traffic: An endogenous switching regime framework,” IEEE Trans. Intell. Transp. Syst., vol. 22, no. 12, pp. 7485–7495, Dec. 2021.
13. Y. Zhang, X. J. Yang, and F. Zhou, “Disengagement cause-and-effect relationships extraction using an NLP pipeline,” IEEE Trans. Intell. Transp. Syst., vol. 23, no. 11, pp. 21430–21439, Nov. 2022.
14. F. B. Kaman and H. Olmuş, “Statistical approaches used in studies evaluating the reliability of autonomous vehicles based on disengagements and reaction times,” Int. J. Automot. Sci. Technol., vol. 8, no. 3, pp. 279–287, 2024.
15. A. Yucel, A. Dag, A. Oztekin, and M. Carpenter, “A novel text analytic methodology for classification of product and service reviews,” J. Bus. Res., vol. 151, pp. 287–297, Nov. 2022.
16. G. V. Kass, “An exploratory technique for investigating large quantities of categorical data,” J. R. Stat. Soc. Ser. C (Appl. Stat.), vol. 29, no. 2, pp. 119–127, 1980.
17. R. Kohavi, “A study of cross-validation and bootstrap for accuracy estimation and model selection,” in Proc. Int. Joint Conf. Artif. Intell., vol. 14, no. 12, pp. 1137–1143, 1995.
18. J. Han, M. Kamber, and J. Pei, Data Mining: Concepts and Techniques, 3rd ed. Elsevier, 2011.