Depth-Aware Multi-Task Deep Learning Approach for Railway Scene Perception

Abstract

Reliable real-time sensing in complex scenes is crucial for safe and autonomous operation in intelligent rail systems. This study proposes a multi-task and depth-aware sensing system that combines semantic segmentation, object detection, and stereo-based depth estimation for railway environments. The proposed system uses YOLOv8–YOLOv12 object detection models and UNet, DeepLabV3+, FPN, and PSPNet-based segmentation architectures. All learning-based segmentation and detection models were trained on the RailSem19 dataset. Experimental results showed that YOLOv9 and YOLOv11 provided stable performance in object detection, while DeepLabV3+ achieved better results in semantic segmentation. To improve system performance, two multi-task fusion approaches were investigated. The first is a segmentation-based geometric inference approach. The second is a dual-stream fusion architecture combining DeepLabV3+ and YOLOv11. The results showed that the dual-stream structure provides higher accuracy and better spatial consistency in railway scenes with occlusions, complex infrastructure, and moving objects. With the ZED2 stereo camera, 2D detection is supported by 3D spatial analysis. An accuracy of ±0.35 m is achieved in tasks such as rail intrusion detection and distance-dependent risk assessment. Overall, the proposed multimodal detection architecture offers a robust and viable solution for railway safety and autonomous monitoring applications.

Keywords

• Railway Detection
• Semantic Segmentation
• Object Detection
• Multi-Task Fusion
• Stereo Depth Estimation

Demiryolu Sahne Algılaması için Derinlik Duyarlı Çok Görevli Derin Öğrenme Yaklaşımı

Abstract

Karmaşık sahnelerde güvenilir gerçek zamanlı algılama, akıllı demiryolu sistemlerinde güvenli ve otonom çalışma için kritik öneme sahiptir. Bu çalışma, demiryolu ortamları için anlamsal bölütleme, nesne tespiti ve stereo tabanlı derinlik kestirimini birleştiren çok görevli ve derinlik farkındalıklı bir algılama sistemi önermektedir. Önerilen sistemde YOLOv8–YOLOv12 nesne tespit modelleri ile UNet, DeepLabV3+, FPN ve PSPNet tabanlı bölütleme mimarileri kullanılmıştır. Tüm modeller RailSem19 veri kümesi üzerinde eğitilmiş ve karşılaştırmalı olarak değerlendirilmiştir. Deneysel sonuçlar, YOLOv9 ve YOLOv11 modellerinin nesne tespitinde kararlı performans sağladığını, DeepLabV3+ modelinin ise anlamsal bölütlemede daha başarılı sonuçlar verdiğini göstermiştir. Sistem performansını artırmak amacıyla iki farklı çok görevli füzyon yaklaşımı incelenmiştir. Bunlardan ilki bölütleme tabanlı geometrik çıkarım yaklaşımıdır. İkincisi ise DeepLabV3+ ve YOLOv11 modellerini birleştiren çift akışlı füzyon mimarisidir. Sonuçlar, çift akışlı yapının örtüşme, karmaşık altyapı ve hareketli nesneler içeren demiryolu sahnelerinde daha yüksek doğruluk ve daha iyi mekânsal tutarlılık sağladığını göstermiştir. ZED2 stereo kamera ile 2D tespit, 3D mekânsal analiz ile desteklenmiştir. Ray ihlali tespiti ve mesafe tabanlı risk değerlendirmesi gibi görevlerde ±0.35 m doğruluk elde edilmiştir. Genel olarak önerilen çok modlu algılama mimarisi, demiryolu güvenliği ve otonom izleme uygulamaları için güçlü ve uygulanabilir bir çözüm sunmaktadır.

Keywords

• Demiryolu Algılama
• Anlamsal Segmentasyon
• Nesne Tespiti
• Çok Görevli Füzyon
• Stereo Derinlik Kestirimi

References

1. Sengaphone, P., De Leon, J. M., Augusto, G. L., Gonzaga, J. A., Chua, J. A., Lim, L. A. G., ... & Naguib, R. (2024, May). Implementation of single shot multibox detector (SSD) algorithm for object detection. In 2024 9th International Conference on Business and Industrial Research (ICBIR) (pp. 1-6). IEEE.
2. Du, J. (2018, April). Understanding of object detection based on CNN family and YOLO. In Journal of Physics: Conference Series (Vol. 1004, p. 012029). IOP Publishing.
3. Diwan, T., Anirudh, G., & Tembhurne, J. V. (2023). Object detection using YOLO: challenges, architectural successors, datasets and applications. Multimedia Tools and Applications, 82(6), 9243-9275.
4. Bochkovskiy, A., Wang, C. Y., & Liao, H. Y. M. (2020). Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934.
5. Khanam, R., & Hussain, M. (2024). What is YOLOv5: A deep look into the internal features of the popular object detector. arXiv preprint arXiv:2407.20892.
6. Wang, C. Y., Bochkovskiy, A., & Liao, H. Y. (2022). YOLOv7: Trainable bagof-freebies sets new state-of-the-art for real-time object detectors. arXiv preprint arXiv:2207.02696. https://doi.org/10.48550/arXiv.2207.02696
7. Varghese, R., & Sambath, M. (2024, April). Yolov8: A novel object detection algorithm with enhanced performance and robustness. In 2024 International conference on advances in data engineering and intelligent computing systems (ADICS) (pp. 1-6). IEEE.
8. Sevi, M., & Aydın, İ. (2023). Detection of foreign objects around the railway line with yolov8. Computer Science, (IDAP-2023), 19-23.

9. Wang, C. Y., Yeh, I. H., & Mark Liao, H. Y. (2024, September). Yolov9: Learning what you want to learn using programmable gradient information. In European conference on computer vision (pp. 1-21). Cham: Springer Nature Switzerland.
10. Khanam, R., & Hussain, M. (2024). Yolov11: An overview of the key architectural enhancements. arXiv preprint arXiv:2410.17725.
11. Alif, M. A. R. (2024). Yolov11 for vehicle detection: Advancements, performance, and applications in intelligent transportation systems. arXiv preprint arXiv:2410.22898.
12. Tian, Y., Ye, Q., & Doermann, D. (2025). Yolov12: Attention-centric real-time object detectors. arXiv preprint arXiv:2502.12524.
13. Ren, S., He, K., Girshick, R., & Sun, J. (2016). Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence, 39(6), 1137-1149.
14. Guindel, C., Martin, D., & Armingol, J. M. (2018). Fast joint object detection and viewpoint estimation for traffic scene understanding. IEEE Intelligent Transportation Systems Magazine, 10(4), 74-86.
15. Pang, B., Zhang, Q., Chai, M., & Wang, H. (2023, September). An efficient network for obstacle detection in rail transit based on multi-task learning. In 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC) (pp. 1795-1800). IEEE.
16. Ronneberger, O., Fischer, P., & Brox, T. (2015, October). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention (pp. 234-241). Cham: Springer international publishing.
17. Sailaja, K. L., Kumar, P. R., Nikhitha, G., & Siddhartha, V. A. (2024, May). Detection of Railway Tracks from Satellite Images Using U-Net Architecture. In 2024 5th International Conference for Emerging Technology (INCET) (pp. 1-5). IEEE.
18. Zhou, X., Shi, P., Sheil, B., & Suryasentana, S. (2024). Knowledge-based U-Net and transfer learning for automatic boundary segmentation. Advanced Engineering Informatics, 59, 102243.
19. Ye, W., Ren, J., Li, C., Liu, W., Zhang, Z., & Lu, C. (2024). Intelligent Detection of Surface Defects in High‐Speed Railway Ballastless Track Based on Self‐Attention and Transfer Learning. Structural Control and Health Monitoring, 2024(1), 2967927.
20. Liu, Y., Bai, X., Wang, J., Li, G., Li, J., & Lv, Z. (2024). Image semantic segmentation approach based on DeepLabV3 plus network with an attention mechanism. Engineering Applications of Artificial Intelligence, 127, 107260.
21. Zeng, Q., Zhang, L., Wang, Y., Luo, X., & Chen, Y. (2024). DeepLab-Rail: semantic segmentation network for railway scenes based on encoder-decoder structure. Journal of Electronic Imaging, 33(4), 043038-043038.
22. Weng, Y., Yang, J., Zhang, C., He, J., Peng, C., Jia, L., & Xiang, H. (2025). An improved DeepLabv3+ railway track extraction algorithm based on densely connected and attention mechanisms. Scientific Reports, 15(1), 2556.
23. Netrakar, M., Upare, D., Tejas, C., Baddi, T. Y., & Benni, R. (2024, November). A Comparative Analysis of U-Net, PSPNet, AND FPNet: Deep Learning Techniques for Image Segmentation. In 2024 IEEE Conference on Engineering Informatics (ICEI) (pp. 1-10). IEEE.
24. Zhang, C., Zhao, J., & Feng, Y. (2023, May). Research on semantic segmentation based on improved PSPNet. In 2023 International Conference on Intelligent Perception and Computer Vision (CIPCV) (pp. 1-6). IEEE.
25. Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., & Luo, P. (2021). SegFormer: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 34, 12077-12090.
26. Zendel, O., Murschitz, M., Zeilinger, M., Steininger, D., Abbasi, S., & Beleznai, C. (2019). Railsem19: A dataset for semantic rail scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (pp. 0-0).
27. Zhang, L., Yang, Q., Liu, X., & Guan, H. (2021). Rethinking hard-parameter sharing in multi-task learning. arXiv preprint arXiv:2107.11359. https://doi.org/10.48550/arXiv.2107.11359.
28. Sun, X., Panda, R., Feris, R., & Saenko, K. (2019). AdaShare: Learning what to share for efficient deep multi-task learning. arXiv preprint arXiv:1911.12423.
29. Chai, H., Yin, Z., Ding, Y., Liu, L., Fang, B., & Liao, Q. (2023). A model-agnostic approach to mitigate gradient interference for multi-task learning. IEEE Transactions on Cybernetics, 53(12), 7810–7823.
30. Shi, G., Li, Q., Zhang, W., Chen, J., & Wu, X. M. (2023). Recon: Reducing conflicting gradients from the root for multi-task learning. arXiv preprint arXiv:2302.11289.
31. Padberg, T., Heikkonen, J., & Kanth, R. (2024). Study on Stereo AI Based Zed-2i Camera. In Á. Rocha, C. Ferrás, J. Hochstetter Diez, & M. Diéguez Rebolledo (Eds.), Information Technology and Systems (pp. 46–56). Springer, Cham. https://doi.org/10.1007/978-3-031-54256-5_5
32. Broekman, A., & Gräbe, P. J. (2021). RailEnV-PASMVS: A perfectly accurate, synthetic, path-traced dataset featuring a virtual railway environment for multi-view stereopsis training and reconstruction applications. Data in Brief, 38, Article 107411. https://doi.org/10.1016/j.dib.2021.107411
33. D’Amico, G., Marinoni, M., Nesti, F., Rossolini, G., Buttazzo, G., & Sabina, S. (2023). Trainsim: A railway simulation framework for LiDAR and camera dataset generation. IEEE Transactions on Intelligent Transportation Systems. https://doi.org/10.48550/arXiv.2302.14486
34. Zhukov, A., Rivero, A., Benois-Pineau, J., Zemmari, A., & Mosbah, M. (2024). A hybrid system for defect detection on rail lines through the fusion of object and context information. Sensors, 24(4), 1171. https://doi.org/10.3390/s24041171
35. Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016). The Cityscapes dataset for semantic urban scene understanding. arXiv preprint arXiv:1604.01685. https://doi.org/10.48550/arXiv.1604.01685
36. Geiger, A., Lenz, P., & Urtasun, R. (2012). Vision meets robotics: The KITTI dataset. International Journal of Robotics Research, 32(11), 1231–1237. https://doi.org/10.1177/0278364913491297