Reinforcement Learning-Tuned Nonlinear Controller Parameters for Synchronization of Chua and Duffing Chaotic Systems

Abstract

Reinforcement learning has recently emerged as a promising machine learning paradigm for the adaptive control of complex nonlinear systems by enabling online decision-making through continuous interaction with the environment. However, its application to the synchronization of chaotic systems using Lyapunov-based nonlinear controllers remains relatively unexplored, particularly with respect to comparing discrete-action and continuous-action reinforcement learning algorithms. To address this research gap, this study proposes reinforcement learning-assisted Lyapunov-based nonlinear controllers for the synchronization of chaotic systems. Specifically, two independent master–slave synchronization problems are considered, one based on Chua’s circuit and the other based on the Duffing oscillator. In each synchronization problem, Deep Q-Learning (DQN) and Deep Deterministic Policy Gradient (DDPG) are employed to adaptively adjust the nonlinear controller parameters online by maximizing the instantaneous reward at each sampling instant, thereby improving synchronization performance while reducing undesirable switching behavior in the control signals. The proposed approaches are evaluated through numerical simulations using reward, average reward, Integral Absolute Error (IAE), Integral Square Error (ISE), Integral Time Absolute Error (ITAE), Integral Time Square Error (ITSE), and control effort as performance metrics. The simulation results demonstrate that DQN-assisted approach consistently achieves higher reward and average reward values than DDPG-assisted approach for both synchronization problems, indicating a more effective reward-maximizing policy under identical operating conditions. For the Chua master–slave synchronization problem, DQN-assisted nonlinear controller provides superior synchronization performance, whereas for the Duffing master–slave synchronization problem, DDPG-assisted nonlinear controller achieves better performance in terms of the error-based performance indices and control effort. Furthermore, robustness analyses under external disturbances demonstrate that both proposed approaches successfully suppress disturbances and maintain stable synchronization between the corresponding master and slave systems. Overall, the results confirm that integrating Lyapunov-based nonlinear control with reinforcement learning provides an adaptive, robust, and effective framework for the synchronization of nonlinear chaotic systems, offering improved synchronization performance and enhanced practical applicability.

Keywords

• Reinforment learning
• DQN
• DDPG
• Chaos
• Nonlinear control
• Synchronization

References

1. Alfred, D., D. Czarkowski, and J. Teng, 2024 Reinforcement learning-based control of a power electronic converter. Mathematics 12: 671.
2. Alhazmi, K. and S. M. Sarathy, 2023 Online reinforcement learning of controller parameters adaptation law. In 2023 American Control Conference (ACC), pp. 2975–2980, IEEE.
3. Bucci, M. A., O. Semeraro, A. Allauzen, G. Wisniewski, L. Cordier, et al., 2019 Control of chaotic systems by deep reinforcement learning. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences 475.
4. Bujgoi, G. and D. Sendrescu, 2025 Tuning of pid controllers using reinforcement learning for nonlinear system control. Processes 13: 735.
5. Cheng, H., H. Li, Q. Dai, and J. Yang, 2023 A deep reinforcement learning method to control chaos synchronization between two identical chaotic systems. Chaos, Solitons & Fractals 174: 113809.
6. Chun-Cheng, W. and H. Lon-Chen, 2020 Control of chaotic nonlinear systems using rbf network with reinforcement learning. In 2020 8th International Conference on Orange Technology (ICOT), pp. 1–5, IEEE.
7. Çimen, M. E., 2024a Controller design for dc-dc boost converter using pi, state feedback and q learning. Gaziosmanpa¸sa Bilimsel Ara¸stırma Dergisi 13: 30–46.
8. Çimen, M. E., 2024b Pi sliding mode control for cuk converter and their tuning using cheetah optimizer and reinforcement learning. In Proceedings of the Ege 12th International Conference on Applied Sciences.

9. Çimen, M. E., 2026 Reinforcement learning and metaheuristic optimization approach to pid with derivative filter controller design for pemfc-driven sepic converter. Arabian Journal for Science and Engineering pp. 1–32.
10. Çimen, M. E. and Z. Garip, 2024 Controlling a single tank liquid level system with classical control methods and reinforcement learning methods. Kocaeli Journal of Science and Engineering 7: 30–41.
11. Clifton, J. and E. Laber, 2020 Q-learning: Theory and applications. Annual Review of Statistics and Its Application 7: 279–301.
12. Das, D., I. Taralova, T. Slavov, J. J. Loiseau, and M. Pandey, 2025 Reinforcement learning-based pid control of rössler oscillator. In V. International Conference on Electrical, Computer and Energy Technologies (ICECET 2025), IEEE Xplore.
13. Emiroglu, S., A. Akgül, Y. Adıyaman, T. E. Gümü¸s, Y. Uyaroglu, et al., 2022 A new hyperchaotic system from t chaotic system: dynamical analysis, circuit implementation, control and synchronization. Circuit World 48: 265–277.
14. Evdüzen, M. R., 2021 Derin peki¸stirmeli ö˘grenme ile robot kol tork kontrolü .
15. Farebrother, J., M. C. Machado, and M. Bowling, 2018 Generalization and regularization in dqn. arXiv preprint arXiv:1810.00123.
16. Hao, G., Y. Li, Y. Li, L. Jiang, and Z. Zeng, 2024 Lyapunov-based safe reinforcement learning for microgrid energy management. IEEE transactions on neural networks and learning systems 36: 9985–9999.
17. Huang, G.-C., 2025 Efficient parameter search for chaotic dynamical systems using lyapunov-based reinforcement learning. Symmetry 17: 1832.
18. Jia, C., Q. Gong, and X. Huang, 2021 Synchronization control of nonlinear chaotic systems with deep reinforcement learning algorithm. In Advances in Guidance, Navigation and Control: Proceedings of 2020 International Conference on Guidance, Navigation and Control, ICGNC 2020, Tianjin, China, October 23–25, 2020, pp. 1673–1682, Springer.
19. Kang, C., C. Rong, W. Ren, F. Huo, and P. Liu, 2021 Deep deterministic policy gradient based on double network prioritized experience replay. IEEE Access 9: 60296–60308.
20. Kharabian, B. and H. Mirinejad, 2021 Synchronization of rossler chaotic systems via hybrid adaptive backstepping/sliding mode control. Results in Control and Optimization 4: 100020.
21. Lillicrap, T. P., J. J. Hunt, A. Pritzel, N. M. O. Heess, T. Erez, et al., 2020 Continuous control with deep reinforcement learning. US Patent 10,776,692.
22. Lucarelli, G. and M. Borrotti, 2020 A deep q-learning portfolio management framework for the cryptocurrency market. Neural Computing and Applications 32: 17229–17244.
23. Madan, R. N., 1993 Chua’s circuit: a paradigm for chaos, volume 1. World Scientific.
24. Mahmoud, A., Y. Ismaeil, and M. Zohdy, 2020 Continuous lyapunov controller and chaotic non-linear system optimization using deep machine learning. arXiv preprint arXiv:2010.14746 .
25. Mnih, V., K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, et al., 2013 Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 .
26. Momtaz, A., E. Qoreishi, S. Amini, H. Khayami, K. Amini, et al., 2026 Chaos-based pseudo random number generators via quasisynchronized chua’s circuits: a symmetric encryption perspective.
27. Analog Integrated Circuits and Signal Processing 126: 26. OpenAI, 2018 Deep deterministic policy gradient. https://spinningup. openai.com/en/latest/algorithms/ddpg.html [Accessed: 2026-07-22].
28. Öztürk, G., M. E. Çimen, Ü. Çavu¸so˘ glu, O. Eldo˘gan, and D. Karayel, 2025 Secure and efficient data encryption for internet of robotic things via chaos-based ascon. Applied Sciences 15: 10641.
29. Qin, Y., L. Cao, Q. Lu, and Y. Pan, 2023 Reinforcement learningbased optimized backstepping control for strict-feedback nonlinear systems subject to external disturbances. Optimal Control Applications and Methods 44: 2724–2743.
30. Rego, R. C. B. and F. M. U. de Araújo, 2021 Nonlinear control system with reinforcement learning and neural networks based lyapunov functions. IEEE Latin America Transactions 19: 1253– 1260.
31. Saeed, N. A., H. A. Saleh, W. A. El-Ganaini, M. Kamel, and M. S. Mohamed, 2023 On a new three-dimensional chaotic system with adaptive control and chaos synchronization. Shock and Vibration 2023: 1969500.
32. Silver, D., G. Lever, N. Heess, T. Degris, D. Wierstra, et al., 2014 Deterministic policy gradient algorithms. In International conference on machine learning, pp. 387–395, Pmlr.
33. Singh, B., R. Kumar, and V. P. Singh, 2022 Reinforcement learning in robotic applications: a comprehensive survey. Artificial intelligence review 55: 945–990.
34. Smart, W. D. and L. P. Kaelbling, 2000 Practical reinforcement learning in continuous spaces. In ICML, pp. 903–910.
35. Stefanski, A. and T. Kapitaniak, 2000 Using chaos synchronization to estimate the largest lyapunov exponent of nonsmooth systems. Discrete Dynamics in Nature and society 4: 207–215.
36. Tan, X., J. Zhang, and Y. Yang, 2003 Synchronizing chaotic systems using backstepping design. Chaos, Solitons & Fractals 16: 37–45.
37. Tsai, C.-L. and H.-C. Lee, 2018 Optimal control of nonlinear duffing oscillators: Lie-group approaches. Journal of Marine Science and Technology 26: 1.
38. Tufenkci, S., B. B. Alagoz, G. Kavuran, C. Yeroglu, N. Herencsar, et al., 2025 Implementations of td3 and ddpg reinforcement learning techniques for tuning pid controller of trms system. Journal of Systems Science and Systems Engineering pp. 1–20.
39. Turab, N., N. Raghu, S. Choudhury, A. Zkear Abass, S. E. Roslin, et al., 2025 Adaptive sliding mode control for chaotic system synchronization using neural networks. Scientific Reports 15: 37547.
40. Wang, R., J. Deng, and Z. Jing, 2006 Chaos control in duffing system. Chaos, Solitons & Fractals 27: 249–257.
41. Wang, S., D. Jia, and X.Weng, 2018 Deep reinforcement learning for autonomous driving. arXiv preprint arXiv:1811.11329 .
42. Wang, X., P. Shi, C.Wen, and Y. Zhao, 2020 Design of parameterself- tuning controller based on reinforcement learning for tracking noncooperative targets in space. IEEE Transactions on Aerospace and Electronic Systems 56: 4192–4208.
43. Watkins, C. J. and P. Dayan, 1992 Q-learning. Machine learning 8: 279–292.
44. Yang, T., X. F. Li, and H. H. Shao, 2001 Chaotic synchronization using backstepping method with application to the chua’s circuit and lorenz system. In Proceedings of the 2001 American Control Conference.(Cat. No. 01CH37148), volume 3, pp. 2299–2300, IEEE.
45. Yang, T., C.W.Wu, and L. O. Chua, 1997 Cryptography based on chaotic systems. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications 44: 469–472.
46. Yang, X., D. Liu, and D.Wang, 2014a Reinforcement learning for adaptive optimal control of unknown continuous-time nonlinear systems with input constraints. International Journal of Control 87: 553–566.
47. Yang, X., D. Liu, and Q. Wei, 2014b Reinforcement-learning-based controller design for nonaffine nonlinear systems. In International Symposium on Neural Networks, pp. 51–58, Springer.
48. Yau, H.-T., P.-H. Kuo, P.-C. Luan, and Y.-R. Tseng, 2024 Proximal policy optimization-based controller for chaotic systems. International Journal of Robust and Nonlinear Control 34: 586–601.
49. Yıldız, H., F. Güney, ˙I. Tunç, and M. T. Söylemez, 2024 Control of emergency vehicles with deep q-learning. ITU Computer Science AI and Robotics 1: 33–38.