State space scalability to enable smart ships with statistical physics and multi-agent-based reinforcement learning

Year 2023, Volume: 3 Issue: 2, 67 - 80, 01.12.2023

Alexander Manohar , David Singer

Abstract

The global marine industry is transitioning to smart ships where navigation, maintenance, and operations are done autonomously. Integrating autonomy into already complex ships presents many challenges, including identifying faults and taking corrective actions. These actions are key components in a Self-Adaptive Health Monitoring (SAHM) system which aims to maintain ship operations. The challenge lies in the failure state space’s extraordinary size which current methods aren’t capable of dealing with. Diagnosis has been achieved for smaller scale systems such as NASA deep space probes, but the complexity of a probe is equivalent to a ship’s single small sub-system. The authors combine recent advancements in statistical physics and multi-agent-based reinforcement learning to address the scale issue and enable crewless vessels. Statistical physics works to extract information about objects through tensor networks, combining physical and logical representations of objects. By combining a sequence of contractions, an ensemble of data about the physical system can be constructed quickly. To demonstrate the proposed method, the algorithm is applied to a modified version of the N-Queens problem which contains operational decision making, geometrical constraints, and a scalable problem. The authors then apply an already proven method to the modified version of the N-Queens problem and compare the results. The tensor network enables agents to handle state space explosion by decoupling the system’s complexity from decision making.

Keywords

Diagnosis, Fault detection, Self-adaptive health monitoring, Tensor networks

Supporting Institution

Office of Naval Research

Project Number

N00014-17-1-2491

Thanks

Thanks to Ms. Kelly Cooper of the Office of Naval Research

References

• [1] Register, L. Global marine technology trends 2030, 2014. <www.0427.co.uk>
• [2] O’Rourke, R. Navy large unmanned surface and undersea vehicles: Background and issues for congress. CRS Report No. R45757, 2021 [Online]. <https://crsreports.congress.gov/product/pdf/R/R45757/44f>
• [3] Kobylinski, L. Smart ships-autonomous or remote controlled? Scientific Journals of the Maritime University of Szczecin, 53:28–34, 2018. <https://doi.org/10.17402/262>
• [4] Evgeniy, Aleksandra, I., Vladimir, B. A. K., Ol’Khovik. Technology level and development trends of autonomous shipping means. pages 421–432. Springer International Publishing, 2021.
• [5] Ang, J. H., Goh, C., Li, Y. Smart design for ships in a smart product through-life and industry 4.0 environment, 2016.
• [6] Ellefsen, A. L., Æsøy, V., Ushakov, S., Zhang, H. A comprehensive survey of prognostics and health management based on deep learning for autonomous ships. IEEE Transactions on Reliability, 68:720–740, 2019. <https://doi.org/10.1109/TR.2019.2907402>
• [7] Dasgupta, A., Doraiswami, R., Azarian, M., Osterman, M., Mathew, S., Pecht, M. The use of canaries for adaptive health management of electronic systems. 2010.
• [8] Gao, Z., Liu, X. An overview on fault diagnosis, prognosis and resilient control for wind turbine systems, 2 2021. <https://doi.org/10.3390/pr9020300>
• [9] Chiang, L. H., Russell, E. L., Braatz, R. D. Fault detection and diagnosis in industrial systems. Springer Science & Business Media, 2000.
• [10] Scheidt, D., Mccubbin, C., Pekala, M., Vick, S., Alger, D. Intelligent control of auxiliary ship systems, 2002. <www.aaai.org>
• [11] Yajnik, S., Jha, N. K. Graceful degradation in algorithm-based fault tolerant multiprocessor systems. IEEE Transactions on Parallel and Distributed Systems, 8:137–153, 1997. <https://doi.org/10.1109/71.577256>
• [12] DARPA. The no manning required ship (nomars) program kicks off, 10 2020.
• [13] Scheit, D. Email from david scheit’s about research at weathergage, 2021.
• [14] Venkatasubramanian, V., Rengaswamy, R., Kavuri, S. N., Yin, K. A review of process fault detection and diagnosis: Part iii: Process history based methods. Computers & Chemical Engineering, 27:327–346, 3 2003. <https://doi.org/10.1016/S0098-1354(02)00162-X>
• [15] Xu, Y., Sun, Y., Wan, J., Liu, X., Song, Z. Industrial big data for fault diagnosis: Taxonomy, review, and applications. IEEE Access, 5:17368–17380, 7 2017. <https://doi.org/10.1109/ACCESS.2017.273194>
• [16] Williams, B. C., Nayak, P. P. A. A model-based approach to reactive self-connguring systems, 1996.
• [17] Gaspar, H. M., Rhodes, D. H., Ross, A. M., Erikstad, S. O. Addressing complexity aspects in conceptual ship design: A systems engineering approach. Journal of Ship Production, 28:145–159, 11 2012. <https://doi.org/10.5957/JSPD.28.4.120015>
• [18] Biamonte, J. Lectures on quantum tensor networks. 12 2019. <http://arxiv.org/abs/1912.10049>
• [19] Okunishi, K., Nishino, T., Ueda, H. Developments in the tensor network – from statistical mechanics to quantum entanglement. 11 2021. <http://arxiv.org/ abs/2111.12223>
• [20] Klishin, A. A., Singer, D. J., Anders, G. V. Avoidance, adjacency, and association in distributed systems design. Journal of Physics: Complexity, 2, 4 2021. <doi:10.1088/2632-072X/abe27f>
• [21] Zhang, K., Yang, Z., Bas¸ar, T. Multi-agent reinforcement learning: A selective overview of theories and algorithms. 11 2019. <http://arxiv.org/abs/1911.10635>
• [22] Friedland, B. Control system design : an introduction to state-space methods. Dover Publications, 2005.
• [23] Hamilton, J. D. State-space models*, 1994.
• [24] Kim, S. D. Characterizing unknown unknowns. 4 2012.
• [25] Lipol, L. S., Haq, J. Risk analysis method:Fmea/fmeca in the organizations. International Journal of Basic & Applied Sciences, 11:74–82, 2011.
• [26] Wang, Y., Liu, M., Bao, Z. Deep learning neural network for power system fault diagnosis. volume 2016-August, 2016. <https://doi.org/10.1109/ChiCC.2016.7554408>
• [27] Abishekraj, N., Prashanna, G. R. J., Suriyaa, M. S., Barathraj, T., Mohanraj, D. Condition based monitoring for fault detection in windmill gear box using artificial neural network. volume 912. IOP Publishing Ltd, 9 2020. <https://doi.org/10.1088/1757-899X/912/3/032061>
• [28] Chan, W. K. V. Interaction metric of emergent behaviors in agent-based simulation. pages 357–368, 2011. <https://doi.org/10.1109/WSC.2011.6147763>
• [29] Johnson, T. L., Genc, S., Bush, S. F. Active probing for diagnosis of emergent faults. IFAC Proceedings Volumes, 42:293–298, 6 2009. <https://doi.org/10.3182/20090610-3-IT-4004.00055>
• [30] Isermann, R., Freyermuth, B. Process fault diagnosis based on process model knowledge-part i: Principles for fault diagnosis with parameter estimation, 1991. <http://asmedigitalcollection.asme.org/dynamicsystems/article-pdf/113/4/620/5555274/620 1.pdf?casa token=-uvY5IbINc8AAAAA: 7oH6rA4tHm2rGdTS3WR-m45keYL9obFoSyEQg5EgY N7Vu0SK0CL8ou-2YhSTyZAbHRLslk>
• [31] Freyermuth, B. R. isermann process fault diagnosis based on process model knowledge-part ii: Case study experiments, 1991. <http://asmedigitalcollection.asme. org/dynamicsystems/article-pdf/113/4/627/5555434/627 1.pdf?casa token=0RwFHWxtWRQAAAAA: aftkGPr6TMTUI68kjzeWTOrZs3ARb l 4Jivq2O2r9nZtlQFbm3zy-2ufJ-pdDkWCGubVc>
• [32] Xu, X., Yan, X., Yang, K., Zhao, J., Sheng, C., Yuan, C. Review of condition monitoring and fault diagnosis for marine power systems. Transportation Safety and Environment, 3, 2021. <https://doi.org/10.1093/tse/tdab005>
• [33] Lee, W. S., Grosh, D. L., Tillman, F. A., Lie, C. H. Fault tree analysis, methods, and applications: A review. IEEE Transactions on Reliability, R-34:194–203,1985. <https://doi.org/10.1109/TR.1985.5222114>
• [34] Toliyat, H. A., Nandi, S., Choi, S., Meshgin-Kelk, H. Electric machines: modeling, condition monitoring, and fault diagnosis. CRC press, 2012.
• [35] Saari, J., Strombergsson, D., Lundberg, J., Thomson,¨ A. Detection and identification of windmill bearing faults using a one-class support vector machine (svm). Measurement: Journal of the International Measurement Confederation, 137:287–301, 4 2019. <https://doi.org/10.1016/j.measurement.2019.01.020>
• [36] Huang, K. Huang - introduction to statistical physics. 2009.
• [37] Klishin, A. A. Statistical physics of design.
• [38] Ran, S.-J., Tirrito, E., Peng, C., Xi, , Luca, C., Gang, T. , Lewenstein, S. M. Lecture notes in physics 964 tensor network contractions methods and applications to quantum many-body systems. <http://www.springer. com/series/5304>
• [39] Peng, C., Lewenstein, M., Ran, S.-J. Review of tensor network contraction approaches review of tensor network contraction approaches emanuele tirrito, 2017. <https://www.researchgate.net/publication/ 319391456>
• [40] Liang, L., Xu, J., Deng, L., Yan, M., Hu, X., Zhang, Z., Li, G., Xie, Y. Fast search of the optimal contraction sequence in tensor networks, 2021. <https://github.com/liangling76/tensor-contraction-sequence-searching>
• [41] Gray, J., Kourtis, S. Hyper-optimized tensor network contraction. 2 2020. <http://arxiv.org/abs/2002. 01935>
• [42] Soong, W. Reinforcement learning, 2018.
• [43] Watkins, C. J. C. H., Dayan, P. Q-learning, 1992.
• [44] Fan, J., Wang, Z., Xie, Y., Yang, Z. A theoretical analysis of deep q-learning. 1 2019. <http://arxiv. org/abs/1901.00137>
• [45] Sonoda, S., Murata, N. Neural network with unbounded activation functions is universal approximator. Applied and Computational Harmonic Analysis, 43:233–268, 9 2017. <https://doi.org/10.1016/J.ACHA.2015.12.005>
• [46] DiGiovanni, A., Zell, E. C. Survey of self-play in reinforcement learning. 7 2021. <http://arxiv.org/ abs/2107.02850>
• [47] Liu, Q., Yu, T., Bai, Y., Jin, C. A sharp analysis of model-based reinforcement learning with self-play, 2021.
• [48] Rivin, I., Vardi, I., Zimmermann, P. The n -queens problem. The American Mathematical Monthly, 101:629–639, 8 1994. <https://doi.org/10.1080/00029890.1994.11997004>
• [49] Sosic, R., Gu, J. A polynomial time algorithm for the n-queens problem. ACM SIGART Bulletin, 1(3):7–11, 1990.
• [50] Bell, J., Stevens, B. A survey of known results and research areas for n-queens. Discrete Mathematics, 309:1– 31, 1 2009. <https://doi.org/10.1016/J.DISC.2007.12.043>