Simulation of Evolutionary Reinforcement Learning-Based Self-Balancing Throwable One-Legged Robot with a Reaction Wheel

Abstract

In this paper, the One-Legged robot is designed to stabilize itself and stand upright at the desired location after being thrown from a different heights. The 5-DOF planar underactuated main body is driven by Reaction wheels, and adaptive Cartesian impedance control has been implemented to effectively manage hard impacts. Evolutionary Reinforcement Learning based AI Agent have been used to adapt to different launch conditions, such as varying speed and altitude. The learning process was performed in real-time using the Matlab simulation program, which models the system dynamics of the robot. The graphical results of the simulation confirm that, with the assistance of the AI agent, the dynamic robot has successfully maintained its stability without tipping over after the launch and has been able to make the desired correction.

Keywords

• Self balance
• ERL Learning
• Simscape Multibody Model

Tepki Tekerleği ile Evrimsel Takviyeli Öğrenme Tabanlı Kendini Dengeleyen Fırlatılabilir Tek Bacaklı Robotun Simülasyonu

Öz

Bu çalışmada, Tek Bacaklı robot farklı yüksekliklerden fırlatıldıktan sonra kendini dengeleyerek istenilen konumda dik duracak şekilde tasarlanmıştır. 5 serbestlik dereceli (5-DOF) düzlemsel ve aktüatörsüz ana gövde, Reaksiyon Tekerlekleri ile tahrik edilmekte olup, sert darbelere etkili bir şekilde tepki verebilmek için adaptif Kartezyen empedans kontrolü uygulanmıştır. Farklı fırlatma koşullarına (hız ve irtifa değişimi gibi) uyum sağlamak amacıyla Evrimsel Pekiştirmeli Öğrenme tabanlı bir yapay zeka ajanı kullanılmıştır. Öğrenme süreci, robotun sistem dinamiklerini modelleyen Matlab simülasyon programı kullanılarak gerçek zamanlı olarak gerçekleştirilmiştir. Simülasyonun grafiksel sonuçları, yapay zeka ajanının yardımıyla dinamik robotun fırlatma sonrası devrilmeden dengesini başarıyla koruduğunu ve istenilen düzeltmeyi yapabildiğini doğrulamaktadır.

Anahtar Kelimeler

• ERL Learning
• Simscape Multibody Model
• Dengelenme Öğrenimi

Kaynakça

1. [1] C. Rui, I. V. Kolmanovsky, and N. H. McClamroch, “Nonlinear attitude and shape control of spacecraft with articulated appendages and reaction wheels,” IEEE Trans Automat Contr, vol. 45, no. 8, pp. 1455–1469, Aug. 2000, doi: 10.1109/9.871754.
2. [2] G. Shengmin and C. Hao, “A comparative design of satellite attitude control system with reaction wheel,” in Proceedings - First NASA/ESA Conference on Adaptive Hardware and Systems, AHS 2006, 2006, pp. 359–362. doi: 10.1109/AHS.2006.2.
3. [3] Y. Yang, “Spacecraft Attitude and Reaction Wheel Desaturation Combined Control Method,” IEEE Trans Aerosp Electron Syst, vol. 53, no. 1, pp. 286–295, Feb. 2017, doi: 10.1109/TAES.2017.2650158.
4. [4] P. Zhang, Z. Wu, H. Dong, M. Tan, and J. Yu, “Reaction-Wheel-Based Roll Stabilization for a Robotic Fish Using Neural Network Sliding Mode Control,” IEEE/ASME Transactions on Mechatronics, vol. 25, no. 4, pp. 1904–1911, Aug. 2020, doi: 10.1109/TMECH.2020.2992038.
5. [5] M. Gajamohan, M. Merz, I. Thommen, and R. D’Andrea, “The Cubli: A cube that can jump up and balance,” in IEEE International Conference on Intelligent Robots and Systems, 2012, pp. 3722–3727. doi: 10.1109/IROS.2012.6385896.
6. [6] M. Muehlebach, G. Mohanarajah, and R. D’Andrea, “Nonlinear analysis and control of a reaction wheel-based 3D inverted pendulum,” in Proceedings of the IEEE Conference on Decision and Control, Institute of Electrical and Electronics Engineers Inc., 2013, pp. 1283–1288. doi: 10.1109/CDC.2013.6760059.
7. [7] T. L. Brown and J. P. Schmiedeler, “Reaction Wheel Actuation for Improving Planar Biped Walking Efficiency,” IEEE Transactions on Robotics, vol. 32, no. 5, pp. 1290–1297, Oct. 2016, doi: 10.1109/TRO.2016.2593484.
8. [8] S. I. Han and J. M. Lee, “Balancing and velocity control of a unicycle robot based on the dynamic model,” IEEE Transactions on Industrial Electronics, vol. 62, no. 1, pp. 405–413, Jan. 2015, doi: 10.1109/TIE.2014.2327562.

9. [9] J. Trentin, S. da Silva, J. Ribeiro, and H. Schaub, “Inverted Pendulum Nonlinear Controllers Using Two Reaction Wheels: Design and Implementation,” IEEE Access, vol. PP, p. 1, May 2020, doi: 10.1109/ACCESS.2020.2988800.
10. [10] M. Kim, J. Kim, and D. Yun, “Compact Posture Control System for Jumping Robot Using an Air Reaction Wheel,” IEEE/ASME Transactions on Mechatronics, 2024, doi: 10.1109/TMECH.2024.3398636.
11. [11] M. Zabihi and A. Alasty, “Dynamic stability and control of a novel handspringing robot,” Mech Mach Theory, vol. 137, pp. 154–171, Jul. 2019, doi: 10.1016/j.mechmachtheory.2019.03.018.
12. [12] S. Haoran, X. Yong, L. Jiali, J. Xinyang, and Y. Jie, “Marsbot: A monopod robot capable of achieving three-dimensional dynamic and stable jumping,” Proc Inst Mech Eng C J Mech Eng Sci, vol. 236, no. 1, pp. 552–565, Jan. 2022, doi: 10.1177/09544062211024311.
13. [13] A. Anzai, T. Doi, K. Hashida, X. Chen, L. Han, and K. Hashimoto, “Monopod robot prototype with reaction wheel for hopping and posture stabilisation,” 2021. [Online]. Available: http://creativecommons.org/licenses/by-nc-nd/4.0/
14. [14] T. L. Brown and J. P. Schmiedeler, “Reaction Wheel Actuation for Improving Planar Biped Walking Efficiency,” IEEE Transactions on Robotics, vol. 32, no. 5, pp. 1290–1297, Oct. 2016, doi: 10.1109/TRO.2016.2593484.
15. [15] F. Roscia, A. Cumerlotti, A. Del Prete, C. Semini, and M. Focchi, “Orientation Control System: Enhancing Aerial Maneuvers for Quadruped Robots,” Sensors, vol. 23, no. 3, Feb. 2023, doi: 10.3390/s23031234.
16. [16] Q. Zhu, X. Wu, Q. Lin, and W.-N. Chen, “Two-Stage Evolutionary Reinforcement Learning for Enhancing Exploration and Exploitation,” 2024. [Online]. Available: www.aaai.org
17. [17] L. Deng and S. Liu, “Incorporating Q-learning and gradient search scheme into JAYA algorithm for global optimization,” Artif Intell Rev, vol. 56, pp. 3705–3748, Dec. 2023, doi: 10.1007/s10462-023-10613-1.
18. [18] L. Deng and S. Liu, “Snow ablation optimizer: A novel metaheuristic technique for numerical optimization and engineering design,” Expert Syst Appl, vol. 225, Sep. 2023, doi: 10.1016/j.eswa.2023.120069.
19. [19] S. Zhang, H. Zhang, and Y. Fu, “Leg Locomotion Adaption for Quadruped Robots with Ground Compliance Estimation,” Appl Bionics Biomech, vol. 2020, 2020, doi: 10.1155/2020/8854411.