Machine Learning Methods for (Dis-)Assembly Sequence Planning - A Systematic Literature Review

Abstract

This paper presents a systematic literature review on the application of reinforcement learning in the domain of assembly and disassembly sequence planning. The authors conduct a keyword search to identify scientific publications in the desired field in three scientific databases. Web of Science, Scopus and IEEE-Xplore. The analysis covers two core aspects of rein-forcement learning, namely the definition of the reward function and the representation of states. In total 23 publications are identified, and the content of the collected works is presented. An analysis of the selected publications is then carried out in relation to the questions posed in order to be able to make recommendations for the application of reinforcement learning methods for the generation of efficient assembly and demonstration sequences.

Keywords

• reinforcement learning
• assembly sequence planning
• disassembly sequence planning

References

1. Boothroyd, G.; Alting, L. Design for Assembly and Disassembly. CIRP Annals 1992, 41, 625–636, doi:10.1016/S0007-8506(07)63249-1.
2. Fazio, T.L. de; Edsall, A.C.; Gustavson, R.E.; Hernandez, J.; Hutchins, P.M.; Leung, H.-W.; Luby, S.C.; Metzinger, R.W.; Nevins, J.L.; Tung, K.; et al. A Prototype of Feature-Based Design for Assembly. Journal of Mechanical Design 1993, 115, 723–734, doi:10.1115/1.2919261.
3. Wang, K.; Li, X.; Gao, L. Modeling and optimization of multi-objective partial disassembly line balancing problem considering hazard and profit. Journal of Cleaner Production 2019, 211, 115–133, doi:10.1016/j.jclepro.2018.11.114.
4. Homem de Mello, L.S.; Sanderson, A.C. A correct and complete algorithm for the generation of mechanical assembly sequences. In Proceedings / 1989 IEEE International Conference on Robotics and Automation, [May 14 - 19, 1989, Scottsdale, Arizona]. Proceed-ings, 1989 International Conference on Robotics and Automation, Scottsdale, AZ, USA, 14-19 May 1989; IEEE Computer Society Press: Washington, DC, 1989; pp 56–61, ISBN 0-8186-1938-4.
5. Jiménez, P.; Torras, C. An efficient algorithm for searching implicit AND/OR graphs with cycles. Artificial Intelligence 2000, 124, 1–30, doi:10.1016/S0004-3702(00)00063-1.
6. Wang, D.; Shao, X.; Ge, X.; Liu, S. A hybrid technology for assembly sequence planning of reflector panels. AA 2017, 37, 442–451, doi:10.1108/AA-12-2016-171.
7. Tseng, H.-E.; Chang, C.-C.; Lee, S.-C.; Huang, Y.-M. A Block-based genetic algorithm for disassembly sequence planning. Expert Systems with Applications 2018, 96, 492–505, doi:10.1016/j.eswa.2017.11.004.
8. Hong, D.S.; Cho, H.S. A neural-network-based computational scheme for generating optimized robotic assembly sequences. Engineer-ing Applications of Artificial Intelligence 1995, 8, 129–145, doi:10.1016/0952-1976(94)00068-X.

9. Sutton, R.S.; Barto, A. Reinforcement learning: An introduction, Second edition; The MIT Press: Cambridge, Massachusetts, London, England, 2018, ISBN 9780262352703.
10. Arulkumaran, K.; Deisenroth, M.P.; Brundage, M.; Bharath, A.A. Deep Reinforcement Learning: A Brief Survey. IEEE Signal Pro-cess. Mag. 2017, 34, 26–38, doi:10.1109/msp.2017.2743240.
11. Goodfellow, I.; Courville, A.; Bengio, Y. Deep learning; The MIT Press: Cambridge, Massachusetts, 2016, ISBN 9780262337373.
12. D. Silver. Lectures on reinforcement learning, 2015.
13. Feurer, M.; Hutter, F. Hyperparameter Optimization. In Automated machine learning: Methods, systems, challenges; Hutter, F., Kotthoff, L., Vanschoren, J., Eds.; Springer: Cham, Switzerland, 2019; pp 3–33, ISBN 9783030053178.
14. A. D. Laud. Theory and application of reward shaping in reinforcement learning, 2004.
15. Aria, M.; Cuccurullo, C. bibliometrix : An R-tool for comprehensive science mapping analysis. Journal of Informetrics 2017, 11, 959–975, doi:10.1016/j.joi.2017.08.007.
16. Neves, M.; Vieira, M.; Neto, P. A study on a Q-Learning algorithm application to a manufacturing assembly problem. Journal of Manufacturing Systems 2021, 59, 426–440, doi:10.1016/j.jmsy.2021.02.014.
17. Neves, M.; Neto, P. Deep reinforcement learning applied to an assembly sequence planning problem with user preferences. Int J Adv Manuf Technol 2022, 122, 4235–4245, doi:10.1007/s00170-022-09877-8.
18. Giorgio, A. de; Maffei, A.; Onori, M.; Wang, L. Towards online reinforced learning of assembly sequence planning with interactive guidance systems for industry 4.0 adaptive manufacturing. Journal of Manufacturing Systems 2021, 60, 22–34, doi:10.1016/j.jmsy.2021.05.001.
19. Hayashi, K.; Ohsaki, M.; Kotera, M. Assembly Sequence Optimization of Spatial Trusses Using Graph Embedding and Reinforcement Learning. Journal of the International Association for Shell and Spatial Structures 2022, 63, 232–240, doi:10.20898/j.iass.2022.016.
20. Kitz, K.; Thomas, U. Neural dynamic assembly sequence planning. In 2021 IEEE 17th International Conference on Automation Science and Engineering (CASE). 2021 IEEE 17th International Conference on Automation Science and Engineering (CASE), Lyon, France, 23–27 Aug. 2021; IEEE, 2021; pp 2063–2068, ISBN 978-1-6654-1873-7.
21. Zhao, M.; Guo, X.; Zhang, X.; Fang, Y.; Ou, Y. ASPW-DRL: assembly sequence planning for workpieces via a deep reinforcement learning approach. AA 2019, 40, 65–75, doi:10.1108/AA-11-2018-0211.
22. Antonelli, D.; Zeng, Q.; Aliev, K.; Liu, X. Robust assembly sequence generation in a Human-Robot Collaborative workcell by rein-forcement learning. FME Transactions 2021, 49, 851–858, doi:10.5937/FME2104851A.
23. Antonelli, D.; Aliev, K. Robust assembly task assignment in Human Robot Collaboration as a Markov Decision Process problem. Procedia CIRP 2022, 112, 174–179, doi:10.1016/j.procir.2022.09.068.
24. Yin, J.; Chen, M.; Zhang, T. Optimization Algorithm for Cooperative Assembly Sequence of Truss Structure Based on Reinforcement Learning. In Intelligent Robotics and Applications; Liu, X.-J., Nie, Z., Yu, J., Xie, F., Song, R., Eds.; Springer International Publish-ing: Cham, 2021; pp 474–484, ISBN 978-3-030-89091-9.
25. Alessio, A.; Aliev, K.; Antonelli, D. Robust Adversarial Reinforcement Learning for Optimal Assembly Sequence Definition in a Cobot Workcell. In Advances in Manufacturing III; Trojanowska, J., Kujawińska, A., Machado, J., Pavlenko, I., Eds.; Springer Inter-national Publishing: Cham, 2022; pp 25–34, ISBN 978-3-030-99309-2.
26. dos Santos, S.R.B.; Givigi, S.N.; Nascimento, C.L. Autonomous construction of structures in a dynamic environment using Rein-forcement Learning. In 2013 IEEE International Systems Conference (SysCon). 2013 7th Annual IEEE Systems Conference (Sys-Con), Orlando, FL, 15–18 Apr. 2013; IEEE, 2013; pp 452–459, ISBN 978-1-4673-3107-4.
27. Watanabe, K.; Inada, S. Search algorithm of the assembly sequence of products by using past learning results. International Journal of Production Economics 2020, 226, 107615, doi:10.1016/j.ijpe.2020.107615.
28. Winter, J. de; Beir, A. de; El Makrini, I.; van de Perre, G.; Nowé, A.; Vanderborght, B. Accelerating Interactive Reinforcement Learn-ing by Human Advice for an Assembly Task by a Cobot. Robotics 2019, 8, 104, doi:10.3390/ROBOTICS8040104.
29. Winter, J. de; EI Makrini, I.; van de Perre, G.; Nowé, A.; Verstraten, T.; Vanderborght, B. Autonomous assembly planning of demon-strated skills with reinforcement learning in simulation. Auton Robot 2021, 45, 1097–1110, doi:10.1007/s10514-021-10020-x.
30. Cebulla, A.; Asfour, T.; Kröger, T. Efficient Multi-Objective Assembly Sequence Planning via Knowledge Transfer between Similar Assemblies. In 2023 IEEE 19th International Conference on Automation Science and Engineering (CASE). 2023 IEEE 19th Interna-tional Conference on Automation Science and Engineering (CASE), Auckland, New Zealand, 26–30 Aug. 2023; IEEE, 2023; pp 1–7, ISBN 979-8-3503-2069-5.
31. Guo, K.; Liu, R.; Duan, G.; Liu, J.; Cao, P. Research on dynamic decision-making for product assembly sequence based on Connector-Linked Model and deep reinforcement learning. Journal of Manufacturing Systems 2023, 71, 451–473, doi:10.1016/j.jmsy.2023.09.015.
32. Bi, Z.; Guo, X.; Wang, J.; Qin, S.; Qi, L.; Zhao, J. A Q-Learning-based Selective Disassembly Sequence Planning Method. In 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC). 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Prague, Czech Republic, 09–12 Oct. 2022; IEEE, 2022; pp 3216–3221, ISBN 978-1-6654-5258-8.
33. Chen, Z.; Li, L.; Zhao, F.; Sutherland, J.W.; Yin, F. Disassembly sequence planning for target parts of end-of-life smartphones using Q-learning algorithm. Procedia CIRP 2023, 116, 684–689, doi:10.1016/j.procir.2023.02.115.
34. Yang, C.; Xu, W.; Liu, J.; Yao, B.; Hu, Y. Robotic Disassembly Sequence Planning Considering Robotic Movement State Based on Deep Reinforcement Learning. In 2022 IEEE 25th International Conference on Computer Supported Cooperative Work in Design (CSCWD). Hangzhou, China, 04–06 May 2022; IEEE, 2022; pp 183–189, ISBN 978-1-6654-0527-0.
35. Zhao, X.; Li, C.; Tang, Y.; Cui, J. Reinforcement Learning-Based Selective Disassembly Sequence Planning for the End-of-Life Products With Structure Uncertainty. IEEE Robot. Autom. Lett. 2021, 6, 7807–7814, doi:10.1109/LRA.2021.3098248.
36. Cui, J.; Yang, C.; Zhang, J.; Tian, S.; Liu, J.; Xu, W. Robotic disassembly sequence planning considering parts failure features. IET Collab Intel Manufact 2023, 5, doi:10.1049/cim2.12074.
37. Allagui, A.; Belhadj, I.; Plateaux, R.; Hammadi, M.; Penas, O.; Aifaoui, N. Reinforcement learning for disassembly sequence planning optimization. Computers in Industry 2023, 151, 103992, doi:10.1016/j.compind.2023.103992.
38. Liu, J.; Xu, Z.; Xiong, H.; Lin, Q.; Xu, W.; Zhou, Z. Digital Twin-Driven Robotic Disassembly Sequence Dynamic Planning Under Uncertain Missing Condition. IEEE Trans. Ind. Inf. 2023, 19, 11846–11855, doi:10.1109/TII.2023.3253187.
39. Collins, K.; Palmer, A.J.; Rathmill, K. The Development of a European Benchmark for the Comparison of Assembly Robot Program-ming Systems. In Robot Technology and Applications: Proceedings of the 1st Robotics Europe Conference Brussels, June 27-28, 1984; Rathmill, K., MacConaill, P., O’Leary, P., Browne, J., Eds.; Springer: Berlin, Heidelberg, 1985; pp 187–199, ISBN 978-3-662-02442-3.