An Approach for DC Motor Speed Control with Off-Policy Reinforcement Learning Method

Abstract

In the literature, interest in automatic control systems that do not require human intervention and perform at the desired level increases day by day. In this study, a Twin Delay Deep Deterministic Policy Gradient (TD3), a reinforcement learning algorithm, automatically controls a DC motor system. A reinforcement learning method is an approach that learns what should be done to reach the goal and observes the results that come out with the interaction of both itself and the environment. The proposed method aims to adjust the voltage value applied to the input of the DC motor in order to reach output with single input and single output structure to the desired speed.

Keywords

• Deep reinforcement learning
• DC motor
• PI controller
• Twin-delayed deep deterministic policy gradient

References

1. R.S. Sutton, "Reinforcement Learning: Past, Present and Future", Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), Vol. 1585, 1998, 195–197.
2. L.P. Kaelbling, M.L. Littman, A.W. Moore, "Reinforcement Learning: A Survey", J. Artif. Intell. Res., Vol. 4, 1996, pp. 237–285.
3. R.S. Sutton, A.G. Barto, "Reinforcement Learning: An Introduction", 1998.
4. J. Xue, Q. Gao, W. Ju, "Reinforcement learning for engine idle speed control", 2010 Int. Conf. Meas. Technol. Mechatronics Autom. ICMTMA 2010, Vol. 2, 2010, pp. 1008–1011.
5. E. Uchibe, M. Asada, K. Hosoda, "Behavior coordination for a mobile robot using modular reinforcement learning", IEEE Int. Conf. Intell. Robot. Syst., Vol. 3, 1996, pp. 1329–1336.
6. Z. Linan, Y. Peng, C. Lingling, Z. Xueping, T. Yantao, "Obstacle avoidance of multi mobile robots based on behavior decomposition reinforcement learning", 2007 IEEE Int. Conf. Robot. Biomimetics, ROBIO, 2007, pp. 1018–1023.
7. N.J. Van Eck, M. Van Wezel, "Application of reinforcement learning to the game of Othello", Comput. Oper. Res., Vol. 35, 2008, pp. 1999–2017.
8. C.J.C.H. Watkins, "Learning from delayed rewards", 1989.

9. C.J.C.H. Watkins, P. Dayan, "Q-learning", Mach. Learn. 1992, Vol. 83, 8, 1992, pp. 279–292,
10. V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, M. Riedmiller, " Playing Atari with Deep Reinforcement Learning", 2013.
11. D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, M. Riedmiller, "Deterministic Policy Gradient Algorithms".
12. T.P. Lillicrap, J.J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, D. Wierstra, " Continuous control with deep reinforcement learning", 4th Int. Conf. Learn. Represent. ICLR 2016 - Conf. Track Proc., 2015.
13. S. Fujimoto, H. Hoof, D. Meger, "Addressing Function Approximation Error in Actor-Critic Methods", http://proceedings.mlr.press/v80/fujimoto18a.html, 2018.
14. F. Harashima, S. Kondo, "Design Method For Digital Speed Control System Of Motor Drives", PESC Rec. - IEEE Annu. Power Electron. Spec. Conf., 1982, pp. 289–297.
15. D. Germanton, M. Lehr, "Variable speed DC motor controller apparatus particularly adapted for control of portable-power tools", 1989.
16. Y. Hoshino, "A proposal of Reinforcement Learning System to Use Knowledge effectively", 2003, pp. 1582–1585.
17. S.J. Russell, P. Norvig, "Artificial Intelligence A Modern Approach", 2003.
18. R.S. Sutton, D. Mcallester, S. Singh, Y. Mansour, "Policy gradient methods for reinforcement learning with function approximation", Adv. NEURAL Inf. Process. Syst. 12, Vol. 12, 2000, pp. 1057--1063.
19. H. van Hasselt, A. Guez, D. Silver, "Deep Reinforcement Learning with Double Q-Learning", Proc. AAAI Conf. Artif. Intell. 30, 2016.
20. W.B. Knox, P. Stone, "Reinforcement learning from human reward: Discounting in episodic tasks", Proc. - IEEE Int. Work. Robot Hum. Interact. Commun., 2012, pp. 878–885.
21. University of Michigan: Control Tutorials for MATLAB and Simulink - Motor Speed: System Modeling, https://ctms.engin.umich.edu/CTMS/index.php?example=MotorSpeed§ion=SystemModeling.