A Comparative Robustness Analysis of YOLOV8 and RT-DETR in Autonomous Driving

Abstract

Robust 2D object detection is a cornerstone of perception systems in autonomous vehicles (AVs). While Convolutional Neural Networks (CNNs) like the YOLO series have long dominated real-time detection, the emergence of Vision Transformers (ViTs) introduces new paradigms in handling global context and occlusions. This paper presents a comprehensive comparative analysis of two state-of-the-art architectures: YOLOv8 (representing the peak of efficient CNN design) and RT-DETR-L (a real-time detection transformer). We evaluate these models on the extensive Waymo Open Dataset, focusing on critical safety metrics including Mean Average Precision (mAP) at varying Intersection over Union (IoU) thresholds (0.5, 0.5:0.95) and object scales (small, medium, large). Furthermore, we conduct a fine-grained robustness analysis across diverse environmental conditions, specifically varying weather (fair vs. rain) and time-of-day (day vs. night). Our methodology provides a rigorous framework for understanding the trade-offs between the inductive bias of CNNs and the global attention mechanisms of Transformers in safety-critical driving scenarios.

Keywords

• Autonomous vehicles
• Object detection
• YOLOv8
• RT-DETR
• Robustness analysis

References

1. Alimov, M., & Meiramkhanov, T. (2024). Domain Generalization in Autonomous Driving: Evaluating YOLOv8s, RT-DETR, and YOLO-NAS with the ROAD-Almaty Dataset. arXiv:2412.12349. https://arxiv.org/abs/2412.12349
2. Bazi, Y., Bashir, N., Melgani, F., Al Rahhal, M. M., Zuair, M. A., & Al-Hichri, H. (2021). Vision transformers for remote sensing image classification. Remote Sensing, 13(3), 516.
3. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020). End-to-end object detection with transformers. In Proceedings of the European Conference on Computer Vision (pp. 213–229). Springer.
4. Cheng, Q., Ling, J., Du, Y., & Zhao, Q. (2025). Toward comprehensive traffic scene understanding: a benchmark and detector for traffic object detection in smart city surveillance. Geo-spatial Information Science, 1-24.
5. Das, P., Jain, C., & Gola, K. K. (2026). Surveillance to Self-Driving: A Comprehensive Review of Object Detection and Tracking Paradigms. Iran Journal of Computer Science. https://link.springer.com/article/10.1007/s42044-025-00387-w
6. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., ... & Houlsby, N. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
7. Geiger, A., Lenz, P., & Urtasun, R. (2012). Are we ready for autonomous driving? The KITTI vision benchmark suite. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 3354–3361). IEEE.
8. Jocher, G., Chaurasia, A., & Qiu, J. (2023). Ultralytics YOLOv8 (Version 8.0.0) [Software]. Available from https://github.com/ultralytics/ultralytics

9. Jun, E. L. T., Tham, M. L., & Kwan, B. H. (2024). A Comparative Analysis of RT-DETR and YOLOv8 for Urban Zone Aerial Object Detection. IEEE International Conference on Intelligent Systems. https://ieeexplore.ieee.org/document/10649836
10. Lv, W., Zhao, Y., Xu, S., Wei, J., Cheng, G., Cui, B., & Zhang, Y. (2023). DETRs beat YOLOs on real-time object detection. arXiv preprint arXiv:2304.08069.
11. Parekh, A., & Bauer, M. (2024). Comparative Analysis of YOLOv8 and RT-DETR for Real-Time Object Detection in Advanced Driver Assistance Systems. Springer International Conference on Computational Science. https://link.springer.com/chapter/10.1007/978-3-031-94962-3_3
12. Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 779–788). IEEE.
13. Rong, Y. M., Ding, T. J., Kang, L. Y., Ariannejad, M., & Anzhou, L. (2025). EV2M-YOLOv8: Enhancing AI Computer Vision with EfficientNetV2 and Multi-head Self-attention for Low-complexity Agricultural Crop, Chilli Maturity Detection. Signal, Image and Video Processing, 19(12), 1057.
14. Schoder, S. (2023). First Qualitative Observations on Deep Learning Vision Models YOLO and DETR for Automated Driving in Austria. arXiv:2312.12314. https://arxiv.org/abs/2312.12314
15. Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, S., ... & Anguelov, D. (2020). Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 2446–2454). IEEE.
16. Turkcan, M. K., Narasimhan, S., Zang, C., & Je, G. H. (2024). Constellation Dataset: Benchmarking High-Altitude Object Detection for Urban Intersections. arXiv:2404.16944. https://arxiv.org/abs/2404.16944
17. Wan, Y., Wang, H., Lu, L., Lan, X., Xu, F., & Li, S. (2024). An Improved Real-Time Detection Transformer Model for the Intelligent Survey of Traffic Safety Facilities. Sustainability, 16(23), 10172. https://www.mdpi.com/2071-1050/16/23/10172
18. Zhu, X., Su, W., Lu, L., Li, B., Wang, X., & Dai, J. (2020). Deformable Detr: Deformable Transformers for End-to-End Object Detection. arXiv: 2010.04159. https://arxiv.org/abs/2010.04159.