Q Learning Based PSO Algorithm Application for Inverse Kinematics of 7-DOF Robot Manipulator

Abstract

Solving inverse kinematics problems is one of the fundamental challenges in serial robot manipulators. In this study, a learning-based algorithm was developed to minimize the complexity of solving the inverse kinematics problem for a 7-degree-of-freedom serial manipulator. The parameters of the Particle Swarm Optimization algorithm, modified with Q-learning, a reinforcement learning technique, are updated depending on the states. This approach aimed to increase the efficiency of the algorithm in finding solutions. In the simulation studies, two different end positions of the robot, measured in meters, were used to compare the performance of the proposed algorithm. The location error of the proposed algorithm was statistically compared, and meaningful results were obtained regarding the reliability of the outcomes through Wilcoxon analysis. The simulation results demonstrated that the reinforcement learning-based particle swarm optimization algorithm can be effectively used for inverse kinematics solutions in serial robot manipulators.

Keywords

• Robot kinematics
• Reinforcement learning
• Q learning
• PSO

References

1. [1] F. Özüdoğru, “Endüstriyel Robot Kolu Modelinin Hedef Konum Eklem Açilarinin Yapıcı Sinir Ağı Ile Kestirimi Ve Kontrollü Yörünge Uygulamasi,” Yüksek Lisans, Elektrik Elektronik Mühendisliği, Tokat, 2020.
2. [2] E. Düzgün, “Paralel ve Hibrit Manipülatörlerin Ileri Kinematik Çözümü Için Yeni Metotlar Geliştirilmesi,” Doktora, Fen Bilimleri Enstitüsü, Bursa, 2023.
3. [3] S. Dereli and R. Köker, “A meta-heuristic proposal for inverse kinematics solution of 7-DOF serial robotic manipulator: quantum behaved particle swarm algorithm,” Artif Intell Rev, vol. 53, pp. 949–964, 2020.
4. [4] F. Aysal, İ. Çelik, E. Cengiz, and Y. Oğuz, “A comparison of multi-layer perceptron and inverse kinematic for RRR robotic arm,” Politeknik Dergisi, vol. 27, no. 1, pp. 121–131, 2023.
5. [5] S. Hwang, H. Kim, Y. Choi, K. Shin, and C. Han, “Design Optimization Method for 7 DOF Robot Manipulator Using Performance Indices,” International Journal of Precision Engineering and Manufacturing, vol. 18, no. 3, pp. 293–299, 2017.
6. [6] A. Avaei, L. van der Spaa, L. Peternel, and J. Kober, “An incremental inverse reinforcement learning approach for motion planning with separated path and velocity preferences,” Robotics, vol. 12, no. 2, 2023.
7. [7] S. Dereli and R. Köker, “Simulation based calculation of the inverse kinematics solution of 7-DOF robot manipulator using artificial bee colony algorithm,” SN Appl Sci, vol. 2, no. 1, p. 27, 2020.
8. [8] S. Baressi Šegota, N. Anđelić, M. Šercer, and H. Meštrić, “Dynamics Modeling of Industrial Robotic Manipulators: A Machine Learning Approach Based on Synthetic Data,” Mathematics, vol. 10, no. 7, p. 1174, 2022.

9. [9] Z. Bingül and S. Küçük, Robot Kinematiği. Umuttepe Yayınları, 2019.
10. [10] H. Danaci, L. A. Nguyen, T. L. Harman, and M. Pagan, “Inverse Kinematics for Serial Robot Manipulators by Particle Swarm Optimization and POSIX Threads Implementation,” Applied Sciences, vol. 13, 2023.
11. [11] C. J. Watkins and P. Dayan, “Q-learning,” Mach Learn, vol. 8, pp. 279–292, 1992.
12. [12] J. Peng and R. J. Williams, “Incremental Multi-Step Q-Learning,” 1996.
13. [13] İ. Tunç and M. Söylemez, “Fuzzy logic and deep Q learning based control for traffic lights,” Alexandria Engineering Journal, vol. 67, pp. 343–359, 2023.
14. [14] M. E. Çimen, Z. Garip, Y. Yalçın, M. Kutlu, and A. F. Boz, “Self Adaptive Methods for Learning Rate Parameter of Q-Learning Algorithm,” Journal of Intelligent Systems: Theory and Applications, vol. 6, no. 2, pp. 191–198, 2023.
15. [15] A. O. Köroğlu, A. E. Edem, S. N. Akmeşe, Ö. Elmas, I. Tunc, and M. T. Soylemez, “Agent-Based Route Planning with Deep Q Learning,” in 13th International Conference on Electrical and Electronics Engineering (ELECO), 2021, pp. 403–407.
16. [16] A. Wang, H., Emmerich, M., & Plaat, “Monte Carlo Q-learning for General Game Playing,” arXiv preprint arXiv:1802.05944.
17. [17] F. Candan, S. Emir, M. Doğan, and T. Kumbasar, “Takviyeli Q-Öğrenme Yöntemiyle Labirent Problemi Çözümü Labyrinth Problem Solution with Reinforcement Q-Learning Method,” in TOK2018 Otomatik Kontrol Ulusal Toplantısı, 2048.
18. [18] Y. Liu, H. Lu, S. Cheng, and Y. Shi, “An adaptive online parameter control algorithm for particle swarm optimization based on reinforcement learning,” in IEEE congress on evolutionary computation (CEC), 2019, pp. 815–822.
19. [19] M. Çimen, “Hibrit ve Kaotik Metasezgisel Arama Algoritmalari Kullanarak Model Öngörülü Kontrol Yapıları Tasarımı,” Doktora, Sakarya Uygulamalı Bilimler Üniversitesi, 2022.
20. [20] A. F. Boz and M. E. Çimen, “An interface design for controlling dead time systems using PSO, CS and FA algorithms,” in 8th International Advanced Technologies Symposium (IATS’17), 19-22 October, 2017.
21. [21] A. F. Boz and M. E. Çimen, “PID Controller Design Using Improved FireFly Algorithm,” in 8th International Advanced Technologies Symposium (IATS’17), 19-22 October, 2017.
22. [22] M. E. Çimen and A. F. Boz, “Parameter identification of a non-minimum phase second order system with time delay using relay test and PSO, CS, FA algorithms,” Journal of the Faculty of Engineering and Architecture of Gazi University, vol. 34, no. 1, pp. 461–477, 2019, doi: 10.17341/gazimmfd.416507.
23. [23] Z. B. Garip, M. E. Cimen, D. Karayel, and A. L. I. F. Boz, “The chaos-based whale optimization algorithms global optimization,” vol. 0, no. 1, pp. 51–63, 2019.
24. [24] A. Akgül, Y. Karaca, Pala MA, M. Çimen, A. Boz, and M. Yıldız, “Chaos Theory, Advanced Metaheuristic Algorithms and Their Newfangled Deep Learning Architecture Optimization Applications: A Review,” Fractals, vol. 32, no. 3, 2024.
25. [25] A. Hossain and Z. Yılmaz Acar, “Comparison of New and Old Optimization Algorithms for Traveling Salesman Problem on Small, Medium, and Large-scale Benchmark Instances,” Bitlis Eren Üniversitesi Fen Bilimleri Dergisi, vol. 13, no. 1, pp. 216–231, 2024.
26. [26] M. E. Cimen, Z. Garip, A. F. Boz, and D. Karayel, “Firefly Algorithm and Particle Swarm Optimization for photovoltaic parameters identification based on single model,” ISMSIT 2018 - 2nd International Symposium on Multidisciplinary Studies and Innovative Technologies, Proceedings, 2018, doi: 10.1109/ISMSIT.2018.8567288.
27. [27] A. Karthikeyan, M. E. Cimen, A. Akgul, A. F. Boz, and K. Rajagopal, “Persistence and coexistence of infinite attractors in a fractal Josephson junction resonator with unharmonic current phase relation considering feedback flux effect,” Nonlinear Dyn, vol. 103, no. 2, pp. 1979–1998, 2021, doi: 10.1007/s11071-020-06159-4.
28. [28] K. Rajagopal et al., “A family of circulant megastable chaotic oscillators, its application for the detection of a feeble signal and PID controller for time-delay systems by using chaotic SCA algorithm,” Chaos Solitons Fractals, vol. 148, no. May, p. 110992, 2021, doi: 10.1016/j.chaos.2021.110992.
29. [29] S. A. Celtek and S. Kul, “Parameter Extraction of PV Solar Cells Using Metaheuristic Methods,” Bitlis Eren Üniversitesi Fen Bilimleri Dergisi, vol. 12, no. 4, pp. 1041–1053, 2023.
30. [30] M. Çimen, Z. Garip, E. M, and A. Boz, “Fuzzy Logic PID Design using Genetic Algorithm under Overshoot Constrained Conditions for Heat Exchanger Control,” Journal of the Institute of Science and Technology, vol. 12, no. 1, pp. 164–181, 2022.
31. [31] H. Geçmez and H. Deveci, “Optimization of Hybrid Composite Laminates with Various Materials using the GA/GPSA Hybrid Algorithm for Maximum Dimensional Stability,” Bitlis Eren Üniversitesi Fen Bilimleri Dergisi, vol. 13, no. 1, pp. 107–133, 2024.
32. [32] M. E. Çimen and A. F. Boz, “PSO, CS ve FA Algoritmalarıyla Ortak Emiterli BJT’li Yükselteç Tasarımı,” Cumhuriyet Üniversitesi Fen Edebiyat Fakültesi Fen Bilimleri Dergisi, vol. 38, no. 1, pp. 119–130, 2017.
33. [33] R. Eberhart and J. Kennedy, “Particle swarm optimization,” in Proceedings of the IEEE international conference on neural networks, 1995, pp. 1942–1948.
34. [34] X. S. Yang, “Firefly algorithms for multimodal optimization,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 5792 LNCS, pp. 169–178, 2009, doi: 10.1007/978-3-642-04944-6_14.
35. [35] M. Cimen and Y. Yalçın, “A novel hybrid firefly–whale optimization algorithm and its application to optimization of MPC parameters,” Soft comput, vol. 26, no. 4, pp. 1845–1872, 2022.
36. [36] X. S. Yang and S. Deb, “Cuckoo search via Lévy flights,” 2009 World Congress on Nature and Biologically Inspired Computing, NABIC 2009 - Proceedings, pp. 210–214, 2009, doi: 10.1109/NABIC.2009.5393690.
37. [37] S. Mirjalili, S. Mirjalili, and A. Lewis, “Grey wolf optimizer,” Advances in Engineering Software, vol. 69, pp. 46–61, 2014.
38. [38] M. Çimen, Z. Garip, and A. Boz, “Chaotic flower pollination algorithm based optimal PID controller design for a buck converter,” Analog Integr Circuits Signal Process, 2021.
39. [39] S. Mirjalili and A. Lewis, “The whale optimization algorithm,” Advances in engineering software, vol. 95, pp. 51–67, 2016.
40. [40] M. Juneja and S. K. Nagar, “Particle swarm optimization algorithm and its parameters: A review,” in International Conference on Control, Computing, Communication and Materials (ICCCCM), 2016.
41. [41] R. C. Eberhart and Y. Shi, “Comparing inertia weights and constriction factors in particle swarm optimization,” in Proceedings of the 2000 congress on evolutionary computation. CEC00, 2000, pp. 84–88.
42. [42] A. Ratnaweera, S. K. Halgamuge, and H. C. Watson, “Self-organizing hierarchical particle swarm optimizer with time-varying acceleration coefficients,” IEEE Transactions on evolutionary computation, vol. 8, no. 3, pp. 240–255, 2004.
43. [43] H. M. Cui and Q. B. Zhu, “Convergence analysis and parameter selection in particle swarm optimization,” Jisuanji Gongcheng yu Yingyong (Computer Engineering and Applications), vol. 42, no. 23, pp. 89–91, 2007.
44. [44] C. Guimin, J. Jianyuan, and H. Qi, “Study on the strategy of decreasing inertia weight in particle swarm optimization algorithm,” Journal-Xian Jiaotong University, vol. 40, no. 1, p. 53, 2006.
45. [45] Tanabe R and Fukunaga A, “Success-history based parameter adaptation for differential evolution,” in IEEE Congress on Evolutionary Computation. IEEE, 2013, pp. 71–78.
46. [46] Z. Liu and T. Nishi, “Multipopulation ensemble particle swarm optimizer for engineering design problems,” Math Probl Eng, 2020.
47. [47] Tatsis VA and Parsopoulos KE, “Grid-based parameter adaptation in particle swarm optimization,” in 2th Metaheuristics International Conference (MIC 2017), 2017, pp. 217–226.
48. [48] F. Olivas, F. Valdez, O. Castillo, and P. Melin, “Dynamic parameter adaptation in particle swarm optimization using interval type2 fuzzy logic,” Soft Computing, vol. 20, no. 3, pp. 1057–1070, 2016.
49. [49] P. Melin, F. Olivas, O. Castillo, F. Valdez, J. Soria, and M. Valdez, “Optimal design of fuzzy classification systems using pso with dynamic parameter adaptation through fuzzy logic,” Expert Syst Appl, vol. 40, no. 8, pp. 3196–3206, 2013.
50. [50] S. Yin et al., “Reinforcement-learning-based parameter adaptation method for particle swarm optimization,” Complex & Intelligent Systems, vol. 9, no. 5, pp. 5585–5609, 2023.
51. [51] Y. Xu and D. Pi, “A reinforcement learning-based communication topology in particle swarm optimization,” Neural Comput Appl, pp. 10007–10032, 2020.
52. [52] C. Lee and M. Ziegler, “Geometric approach in solving inverse kinematics of PUMA robots,” IEEE Trans Aerosp Electron Syst, vol. 6, pp. 695–706, 1984.
53. [53] R. Köker, C. Öz, T. Çakar, and H. Ekiz, “A study of neural network based inverse kinematics solution for a three-joint robot,” Rob Auton Syst, vol. 49, no. 3–4, pp. 227–234, 2004.
54. [54] S. Alavandar and M. J. Nigam, “Neuro-fuzzy based approach for inverse kinematics solution of industrial robot manipulators,” International Journal of Computers Communications & Contro, vol. 3, no. 3, pp. 224–234, 2008.
55. [55] G. Jin, S. Ma, and Z. Li, “Dynamic simulation modeling of industrial robot kinematics in industry 4.0,” Discrete Dyn Nat Soc, pp. 1–11, 2022.
56. [56] Y. Chen, X. Zhang, Y. Huang, Y. Wu, and J. Ota, “Kinematics optimization of a novel 7-DOF redundant manipulator,” Rob Auton Syst, vol. 163, p. 104377, 2023.
57. [57] S. Baressi Šegota, N. Anđelić, I. Lorencin, M. Saga, and Z. Car, “Path planning optimization of six-degree-of-freedom robotic manipulators using evolutionary algorithms,” International journal of advanced robotic systems, vol. 17, no. 2, 2020.
58. [58] Y. Hou and J. Li, “Learning 6-DoF grasping with dual-agent deep reinforcement learning,” Rob Auton Syst, vol. 166, 2023.
59. [59] S. Müftü and B. Gökçe, “Design and Implementation of an Optimized PID Controller for Two-Limb Robot Arm Control,” Bitlis Eren Üniversitesi Fen Bilimleri Dergisi, vol. 13, no. 1, pp. 192–204, 2024.
60. [60] M. Çimen and A. Boz, “Parameter identification of a non-minimum phase second order system with time delay using relay test and PSO, CS, FA algorithms,” Journal of the Faculty of Engineering and Architecture of Gazi University, vol. 34, no. 1, pp. 461–477, 2019.
61. [61] A. Angiuli, J. P. Fouque, and M. Laurière, “Unified reinforcement Q-learning for mean field game and control problems,” Mathematics of Control, Signals, and Systems, vol. 34, no. 2, pp. 217–271, 2022.
62. [62] C. J. C. H. Watkins, “Learning from delayed rewards,” Ph.D., Dissertation, King’s College UK, 1989.
63. [63] A. S. Posamentier and I. Lehmann, The Glorious Golden Ratio. Prometheus Books, 2011.
64. [64] S. Rao, Engineering optimization: theory and practice. John Wiley & Sons, 2019.