Multimodal Failure Detection in Robot-Object Interaction

Abstract

Unsafe situations might arise for service robots in everyday manipulation settings due to operational, sensory or environmental factors. Ensuring safety is crucial for these settings. In order to detect these situations, onboard continuous execution monitoring and failure detection procedures are needed. To address these issues, we present a multimodal failure monitoring and detection system to detect manipulation failures. Rather than relying only on a single sensor modality, we consider integration of different modalities to get better detection performance in different failure cases. In our system, high level proprioceptive, auditory and visual predicates are extracted by processing each modality separately. Then, the extracted predicates are fused altogether. Experiments on our humanoid robot for tabletop manipulation scenarios indicate that the contributions of modalities are complementary of each other. Multimodal fusion-based failure detection outperforms the unimodal detection with 86% success rate for pick and 95% success rates for place and push actions.

Keywords

• Cognitive robots
• Failure detection
• Multi-modal perception

Project Number

115E368

Robot-Nesne Etkileşiminde Çok Kipli Hata Sezme

Abstract

Hizmet robotları için gündelik etkileşimlerinde yürütme, sensör veya ortamsal faktörlerden dolayı emniyetsiz durumlar oluşabilir. Bu tür durumlarda emniyetin sağlanması kritik öneme sahiptir. Bu durumların sezilebilmesi için sürekli eylem gözetimi ve hata sezme bileşenlerine ihtiyaç duyulmaktadır. Bu amaçla, bu çalışmada nesne etkileşim hatalarının sezilmesi için çok kipli bir eylem gözetimi ve hata sezme sistemi sunulmuştur. Tek bir sensör kipine bağımlı kalmak yerine, farklı tipte sensörlerden alınan gözlemler tümleştirilerek farklı hata senaryoları için hata sezme başarımı arttırılmıştır. Önerilen sistemde iç algı, işitsel algı ve görsel algı kipleri birbirlerinden bağımsız olarak işlenerek semantik yüklemler elde edilmiş ve bu yüklemler hata sezme için birleştirilmiştir. İnsansı robotumuz ile masa üstünde yapılan deney sonuçlarına göre sensör verilerinin hata sezmeye katkılarının tamamlayıcı olduğu gözlenmiştir. Çok kipli sensör füzyonuyla hata sezme, tutma eylemi için %86, bırakma ve itme eylemleri için %95 oranında hata sezme başarımı ile tek kipli hata sezmeden daha başarılı sonuçlar üretmiştir.

Keywords

• Bilişsel robotlar
• hata sezme
• çok kipli algılama

Supporting Institution

Tübitak

Project Number

115E368

References

1. Adam S., Larsen M., Jensen K., and Schultz U. P., 2014. Towards Rule-based Dynamic Safety Monitoring for Mobile Robots. Simulation, Modeling, and Programming for Autonomous Robots. Springer, 207–218.
2. Aldoma A., Marton Z.C., Tombari F., Wohlkinger W., Potthast C., Zeisl B., Rusu R. B., Gedikli S., and Vincze M., 2012. Point cloud library. IEEE Robotics & Automation Magazine, 1070(12).
3. Bouguerra A., Karlsson L., and Saffiotti A., 2007. Handling uncertainty in semantic-knowledge based execution monitoring. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 437–443.
4. Brigham E. O., Pastor J. R., Apostol T. M., Rodríguez M., Martínez M. R., Edwards C. H., Edwards D. E. H., 1998. The fast Fourier transform and its applications, Prentice Hall.
5. Doherty P., Kvarnström J., and Heintz F., 2009. A temporal logic-based planning and execution monitoring framework for unmanned aircraft systems. Autonomous Agents and Multi-Agent Systems, 19(3), 332–377.
6. Freeman W. T. and Roth M., 1994. Orientation histograms for hand gesture recognition. Mitsubishi Electric Research Laboratories Teknik Raporu.
7. Fritz C., 2005. Execution monitoring – a survey. University of Toronto Teknik Raporu.
8. Fourlas G. K., Karkanis S., Karras G. C., and Kyriakopoulos K. J., 2014. Model based actuator fault diagnosis for a mobile robot. IEEE International Conference on Industrial Technology (ICIT), 79–84.

9. Goel P., Dedeoglu G., Roumeliotis S. I., and Sukhatme G. S., 2000. Fault detection and identification in a mobile robot using multiple model estimation and neural network. IEEE International Conference on Robotics and Automation (ICRA), 2302–2309.
10. HARK: http://www.hark.jp/, 12.09.2022.
11. Häussermann K., Zweigle O., and Levi P., 2015. A novel framework for anomaly detection of robot behaviors. Journal of Intelligent & Robotic Systems, 77(2), 361–375.
12. Hinterstoisser S., Cagniart C., Ilic S., Sturm P., Navab N., Fua P., and Lepetit V., 2012. Gradient response maps for real-time detection of textureless objects. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(5), 876–888.
13. Hovland G. E., and McCarragher, B. J., 1998. Hidden markov models as a process monitor in robotic assembly. The International Journal of Robotics Research, 17(2), 153–168.
14. Inceoglu A., Ince G., Yaslan Y., Sariel S., 2018-1. Comparative Assessment of Sensing Modalities on Manipulation Failure Detection. IEEE International Conference on Robotics and Automation (ICRA) Workshop on Multimodal Robot Perception.
15. Inceoglu A., Ince G., Yaslan Y., Sariel S., 2018-2. Failure Detection Using Proprioceptive, Auditory and Visual Modalities. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
16. Inceoglu A., Koc C., Kanat B. O., Ersen M., and Sariel S., 2018-3. Continuous visual world modeling for autonomous robot manipulation. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 49(1), 192 - 205.
17. ISO 10218-2:2011, 2011. Robots and Robotic Devices – Safety Requirements for Industrial Robots – Part 2: Robot Systems and Integration. ISO, Geneva, Switzerland.
18. ISO 10218:2011, 2011. Robots and Robotic Devices – Safety Requirements for Industrial Robots – Part 1: Robots. ISO, Geneva, Switzerland.
19. Kapotoglu M., Koc C., Sariel S., and Ince G., 2014. Action monitoring in cognitive robots. The 22nd Signal Processing and Communications Applications Conference (SIU), 2154–2157.
20. Kirchner D., and Geihs K., 2014. Qualitative bayesian failure diagnosis for robot systems. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
21. Kvarnström J., Heintz F., and Doherty P., 2008. A temporal logic-based planning and execution monitoring system. ICAPS, 198–205.
22. Logan B., 2000. Mel frequency cepstral coefficients for music modeling. International Symposium on Music Information Retrieval (ISMIR).
23. Mendoza J. P., Veloso M., and Simmons R., 2012. Mobile robot fault detection based on redundant information statistics. IEEE/RSJ International Conference on Intelligent Robots and Systems.
24. Mendoza J. P., Veloso M., and Simmons R., 2015. Plan execution monitoring through detection of unmet expectations about action outcomes. IEEE International Conference on Robotics and Automation (ICRA), 3247–3252.
25. Mendoza J. P., Veloso M., Simmons R., 2014. Focused optimization for online detection of anomalous regions. IEEE International Conference on Robotics and Automation (ICRA), 3358–3363.
26. Micalizio R., 2013. Action failure recovery via model-based diagnosis and conformant planning. Computational Intelligence, 29(2), 233–280.
27. Orendt E. M. and Henrich D., 2015. Design of robust robot programs: Deviation detection and classification using entity-based resources. IEEE International Conference on Robotics and Biomimetics (ROBIO), 1704–1710.
28. Pearson K., 1901. On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11), 559–572.
29. Pettersson O., 2005. Execution monitoring in robotics: A survey. Robotics and Autonomous Systems, 53(2), 73–88.
30. Pettersson O., Karlsson L., and Saffiotti A., 2007. Model-free execution monitoring in behavior-based robotics. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 37(4), 890–901.
31. ROS, 2022: http://www.ros.org/, 12.09.2022.
32. Rubinstein R., 1999. The cross-entropy method for combinatorial and continuous optimization. Methodology and Computing in Applied Probability, 1(2), 127–190.
33. Saied M., Lussier B., Fantoni I., Francis C., Shraim H., and Sanahuja G., 2015. Fault diagnosis and fault-tolerant control strategy for rotor failure in an octorotor. IEEE International Conference on Robotics and Automation (ICRA).
34. Sassi I., Gouin A., and Thiriet J., 2016. A Bayesian Network for Diagnosis of Networked Mobile Robots. Risk, Reliability and Safety: Innovating Theory and Practice (ESREL).
35. Schleyer G. and Russell R. A., 2011. Disturbance and failure classification in walking robots. Australasian Conference on Robotics and Automation, 1–8.
36. Stavrou D., Eliades D. G., Panayiotou C. G., and Polycarpou M., 2015. Fault detection for service mobile robots using model-based method. Autonomous Robots 40(1), 383–394.