Reinforcement Learning-Based Self-Healing Framework in IoT Sensor Networks

Tunahan Timuçin

doi:10.54287/gujsa.1914900

Reinforcement Learning-Based Self-Healing Framework in IoT Sensor Networks

Abstract

The fast increase of devices in Internet of Things (IoT) networks, which is projected to grow to over 21 billion devices by 2025, will require more sophisticated management paradigms, as current rule-based and reactive frameworks cannot support networks of this scale. Rule-based systems are capable of processing only of predefined failure patterns and when it comes to complex situations like simultaneous multiple failures and cascading failures, they cannot work. The proposed paper suggests a self-healing framework of reinforcement learning (RL) with respect to an IoT sensor network. The original innovation of the framework is that it brings the MAPE-K cycle (Monitor-Analyze-Plan-Execute over Knowledge) framework, the fundamental reference model of autonomous computing, to which a learning element is added to form the MAPE-K+L model. This aspect provides the system to be able to enhance its policy gradually through learning on the past failures. The proposed framework has been tested in a custom Python/Gymnasium simulation framework, with six failure modes (single node failure, sensor drift, gateway failure, concurrent multiple failure, network congestion, cascading failure) in cluster topology networks, using 50 to 500 nodes. The Q-Learning and Deep Q-Network (DQN) agents were fully contrasted with random (RND) and rule-based (RB) baselines. The Q-Learning agent in the multiple failure scenario decreased the mean recovery time (MTTR) by 51.9 and 32.8 percent relative to random selection and rule-based approach respectively (p<0.001, Cohen d=1.424). The DQN agent had the best cumulative reward and the most stable performance in the cascading failure case; scalability experiments proved that DQN can work with a stable performance even in the 500-node networks.

Keywords

References

Adeniyi, O., Sadiq, A. S., Pillai, P., Taheir, M. A., & Kaiwartya, O. (2023). Proactive self-healing approaches in mobile edge computing: A systematic literature review. Computers, 12(3), 63. https://doi.org/10.3390/computers12030063
Albrecht, S. V., Christianos, F., & Schäfer, L. (2024). Multi-Agent Reinforcement Learning: Foundations and Modern Approaches. MIT Press.
Aldrini, J., Chihi, I., & Sidhom, L. (2024). Fault diagnosis and self-healing for smart manufacturing: a review. Journal of Intelligent Manufacturing, 35(6), 2441-2473. https://doi.org/10.1007/s10845-023-02165-6
Alhanaf, A. S., Balik, H. H., & Farsadi, M. (2023). Intelligent fault detection and classification schemes for smart grids based on deep neural networks. Energies, 16(22), 7680. https://doi.org/10.3390/en16227680
Aliu, O. G., Imran, A., Imran, M. A., & Evans, B. (2013). A survey of self organisation in future cellular networks. IEEE Communications Surveys & Tutorials, 15(1), 336-361. https://doi.org/10.1109/SURV.2012.021312.00116
Aminikhanghahi, S., & Cook, D. J. (2017). A survey of methods for time series change point detection. Knowledge and Information Systems, 51(2), 339-367. https://doi.org/10.1007/s10115-016-0987-z
Asghar, M. Z., Nieminen, P., Hämäläinen, S., Ristaniemi, T., Imran, M. A., & Hämäläinen, T. (2017). Towards proactive context-aware self-healing for 5G networks. Computer Networks, 128, 5-13. https://doi.org/10.1016/j.comnet.2017.04.053
Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly detection: A survey. ACM Computing Surveys, 41(3), 15. https://doi.org/10.1145/1541880.1541882

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd Ed.). Lawrence Erlbaum Associates. https://doi.org/10.4324/9780203771587
Devi, S. K., Thenmozhi, R., & Kumar, D. S. (2024). Self-healing IoT sensor networks with isolation forest algorithm for autonomous fault detection and recovery. In: Proceedings of the International Conference on Automation and Computation (AUTOCOM) (14-16 March 2024, pp. 451-456), Dehradun, India. https://doi.org/10.1109/AUTOCOM60220.2024.10486184
Feng, X., Wu, J., Wu, Y., Li, J., & Yang, W. (2023). Blockchain and digital twin empowered trustworthy self-healing for edge-AI enabled industrial Internet of things. Information Sciences, 642, 119169. https://doi.org/10.1016/j.ins.2023.119169
Hagberg, A. A., Schult, D. A., & Swart, P. J. (2008). Exploring network structure, dynamics, and function using NetworkX. In: G. Varoquaux, T. Vaught, & J. Millman (Eds.), Proceedings of the 7th Python in Science Conference (SciPy 2008) (19-24 August 2008, pp. 11-15), Pasadena, California. https://doi.org/10.25080/TCWV9851
Johnphill, O., Sadiq, A. S., Al-Obeidat, F., Al-Khateeb, H., Taheir, M. A., Kaiwartya, O., & Ali, M. (2023). Self-healing in cyber–physical systems using machine learning: A critical analysis of theories and tools. Future Internet, 15(7), 244. https://doi.org/10.3390/fi15070244
Karamthulla, M. J., Arasu Malaiyappan, J. N., & Prakash, S. (2023). AI-powered self-healing systems for fault tolerant platform engineering: Case studies and challenges. Journal of Knowledge Learning and Science Technology, 2(2), 327-338. https://doi.org/10.60087/jklst.vol2.n2.p338
Kephart, J. O., & Chess, D. M. (2003). The vision of autonomic computing. Computer, 36(1), 41-50. https://doi.org/10.1109/MC.2003.1160055
Lei, L., Tan, Y., Zheng, K., Liu, S., Zhang, K., & Shen, X. (2020). Deep reinforcement learning for autonomous internet of things: Model, applications and challenges. IEEE Communications Surveys & Tutorials, 22(3), 1722-1760. https://doi.org/10.1109/COMST.2020.2988367
Manju, & Srivastav, V. K. (2025). Review of self-healing IoT networks based AI-driven fault detection and recovery. International Journal of Applied and Behavioral Sciences, 02(01), 230-244. https://doi.org/10.70388/ijabs250121
Mann, H. B., & Whitney, D. R. (1947). On a test of whether one of two random variables is stochastically larger than the other. The Annals of Mathematical Statistics, 18(1), 50-60. https://doi.org/10.1214/aoms/1177730491
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., & Hassabis, D. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529-533. https://doi.org/10.1038/nature14236
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., & Dormann, N. (2021). Stable-Baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268), 1-8.
Riegler, M., Sametinger, J., & Vierhauser, M. (2023). A distributed MAPE-K framework for self-protective IoT devices. In: Proceedings of the IEEE/ACM 18th Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS) (15-16 May 2023, pp. 202-208), Melbourne, Australia. https://doi.org/10.1109/SEAMS59076.2023.00034
Rosenberger, J., Urlaub, M., Rauterberg, F., Lutz, T., Selig, A., Bühren, M., & Schramm, D. (2022). Deep reinforcement learning multi-agent system for resource allocation in industrial internet of things. Sensors, 22(11), 4099. https://doi.org/10.3390/s22114099
Sinha, S. (2025, October 28). State of IoT 2025: Number of connected IoT devices growing 14% to 21.1 billion globally. IoT Analytics Research. URL: https://iot-analytics.com/number-connected-iot-devices/
Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction (2nd Ed.). MIT Press.
Towers, M., Kwiatkowski, A., Terry, J., Balis, J. U., De Cola, G., Deleu, T., Goulão, M., Kallinteris, A., Krimmel, M., KG, A., Perez-Vicente, R., Pierré, A., Schulhoff, S., Tai, J. J., Tan, H., & Younis, O. G. (2024). Gymnasium: A standard interface for reinforcement learning environments. In: 39th Conference on Neural Information Processing Systems (NeurIPS 2025). https://doi.org/10.48550/arXiv.2407.17032
Vankayalapati, R. K., Pandugula, C., Ganti, V. K. A. T., & Mishra, G. (2022). AI-powered self-healing cloud infrastructures: A paradigm for autonomous fault recovery. Migration Letters, 19(6), 1173-1187.
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S. J., Brett, M., Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E., … van Mulbregt, P. (2020). SciPy 1.0: Fundamental algorithms for scientific computing in Python. Nature Methods, 17(3), 261-272. https://doi.org/10.1038/s41592-019-0686-2
Watkins, C. J. C. H., & Dayan, P. (1992). Q-learning. Machine Learning, 8(3-4), 279-292. https://doi.org/10.1007/BF00992698
Younis, M., & Akkaya, K. (2008). Strategies and techniques for node placement in wireless sensor networks: A survey. Ad Hoc Networks, 6(4), 621-655. https://doi.org/10.1016/j.adhoc.2007.05.003
Zinn, J., Vogel-Heuser, B., & Gruber, M. (2021). Fault-tolerant control of programmable logic controller-based production systems with deep reinforcement learning. Journal of Mechanical Design, 143(7), 072004. https://doi.org/10.1115/1.4050624

Details

Primary Language

English

Subjects

Cyberphysical Systems and Internet of Things, System and Network Security, Network Engineering

Journal Section

Research Article

Authors

Tunahan Timuçin ^*
0000-0003-0332-4118
Türkiye

Publication Date

June 30, 2026

Submission Date

March 24, 2026

Acceptance Date

April 22, 2026

Published in Issue

Year 2026 Volume: 13 Number: 2

DOI

https://doi.org/10.54287/gujsa.1914900

IZ

https://izlik.org/JA27GG68PY

Cite

RIS / Bibtex

APA

Timuçin, T. (2026). Reinforcement Learning-Based Self-Healing Framework in IoT Sensor Networks. Gazi University Journal of Science Part A: Engineering and Innovation, 13(2), 764-783. https://doi.org/10.54287/gujsa.1914900

AMA

1.Timuçin T. Reinforcement Learning-Based Self-Healing Framework in IoT Sensor Networks. GU J Sci, Part A. 2026;13(2):764-783. doi:10.54287/gujsa.1914900

Chicago

Timuçin, Tunahan. 2026. “Reinforcement Learning-Based Self-Healing Framework in IoT Sensor Networks”. Gazi University Journal of Science Part A: Engineering and Innovation 13 (2): 764-83. https://doi.org/10.54287/gujsa.1914900.

EndNote

Timuçin T (June 1, 2026) Reinforcement Learning-Based Self-Healing Framework in IoT Sensor Networks. Gazi University Journal of Science Part A: Engineering and Innovation 13 2 764–783.

IEEE

[1]T. Timuçin, “Reinforcement Learning-Based Self-Healing Framework in IoT Sensor Networks”, GU J Sci, Part A, vol. 13, no. 2, pp. 764–783, June 2026, doi: 10.54287/gujsa.1914900.

ISNAD

Timuçin, Tunahan. “Reinforcement Learning-Based Self-Healing Framework in IoT Sensor Networks”. Gazi University Journal of Science Part A: Engineering and Innovation 13/2 (June 1, 2026): 764-783. https://doi.org/10.54287/gujsa.1914900.

JAMA

1.Timuçin T. Reinforcement Learning-Based Self-Healing Framework in IoT Sensor Networks. GU J Sci, Part A. 2026;13:764–783.

MLA

Timuçin, Tunahan. “Reinforcement Learning-Based Self-Healing Framework in IoT Sensor Networks”. Gazi University Journal of Science Part A: Engineering and Innovation, vol. 13, no. 2, June 2026, pp. 764-83, doi:10.54287/gujsa.1914900.

Vancouver

1.Tunahan Timuçin. Reinforcement Learning-Based Self-Healing Framework in IoT Sensor Networks. GU J Sci, Part A. 2026 Jun. 1;13(2):764-83. doi:10.54287/gujsa.1914900