Improving Sample Efficiency of Reinforcement Learning Control Using Autoencoders

Abstract

This study presents a novel approach for improving the sample efficiency of reinforcement learning (RL) control of dynamic systems by utilizing autoencoders. The main objective of this research is to investigate the effectiveness of autoencoders in enhancing the learning process and improving the resulting policies in RL control problems. In literature most applications use only autoencoder’s latent space while learning. This approach can cause loss of information, difficulty in interpreting latent space, difficulty in handling dynamic environments and outdated representation. In this study, proposed novel approach overcomes these problems and enhances sample efficiency using both states and their latent space while learning. The methodology consists of two main steps. First, a denoising-contractive autoencoder is developed and implemented for RL control problems, with a specific focus on its applicability to state representation and feature extraction. The second step involves training a Deep Reinforcement Learning algorithm using the augmented states generated by the autoencoder. The algorithm is compared against a baseline Deep Q-Network (DQN) algorithm in the LunarLander environment, where observations from the environment are subject to Gaussian noise.

Keywords

• Reinforcement Learning
• Autoencoders
• Control

References

1. R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press, 2018.
2. V. Mnih, K. Kavukcuoglu, D. Silver, et al., “Humanlevel control through deep reinforcement learning,” nature, vol. 518, no. 7540, pp. 529–533, 2015.
3. D. Silver, J. Schrittwieser, K. Simonyan, et al., “Mastering the game of go without human knowledge,” nature, vol. 550, no. 7676, pp. 354–359, 2017.
4. G. Dulac-Arnold, D. Mankowitz, and T. Hester, Challenges of real-world reinforcement learning, 2019. arXiv: 1904.12901 [cs.LG].
5. R. S. Sutton, “Dyna, an integrated architecture for learning, planning, and reacting,” ACM Sigart Bulletin, vol. 2, no. 4, pp. 160–163, 1991.
6. K. Chua, R. Calandra, R. McAllister, and S. Levine, “Deep reinforcement learning in a handful of trials using probabilistic dynamics models,” Advances in neural information processing systems, vol. 31, 2018.
7. R. Agarwal, C. Liang, D. Schuurmans, and M. Norouzi, “Learning to generalize from sparse and underspecified rewards,” in International conference on machine learning, PMLR, 2019, pp. 130–140.
8. T. Nguyen, T. M. Luu, T. Vu, and C. D. Yoo, “Sample-efficient reinforcement learning representation learning with curiosity contrastive forward dynamics model,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2021, pp. 3471–3477.

9. N. Botteghi, M. Poel, and C. Brune, Unsupervised representation learning in deep reinforcement learning: A review, 2022. arXiv: 2208.14226 [cs.LG].
10. L. Kocsis and C. Szepesvári, “Bandit based montecarlo planning,” in European conference on machine learning, Springer, 2006, pp. 282–293.
11. P.-Y. Oudeyer and F. Kaplan, “What is intrinsic motivation? a typology of computational approaches,” Frontiers in neurorobotics, vol. 1, p. 108, 2007.
12. Y. Burda, H. Edwards, A. Storkey, and O. Klimov, “Exploration by random network distillation,” arXiv preprint arXiv:1810.12894, 2018.
13. A. A. Rusu, N. C. Rabinowitz, G. Desjardins, et al., “Progressive neural networks,” arXiv preprint arXiv:1606.04671, 2016.
14. C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in International conference on machine learning, PMLR, 2017, pp. 1126–1135.
15. A. Nichol, J. Achiam, and J. Schulman, “On first-order meta-learning algorithms,” arXiv preprint arXiv:1803.02999, 2018.
16. T. Hester, M. Vecerik, O. Pietquin, et al., “Deep qlearning from demonstrations,” in Proceedings of the AAAI conference on artificial intelligence, vol. 32, 2018.
17. A. Kumar, A. Zhou, G. Tucker, and S. Levine, “Conservative q-learning for offline reinforcement learning,” Advances in Neural Information Processing Systems, vol. 33, pp. 1179–1191, 2020.
18. M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas, “Reinforcement learning with augmented data,” Advances in neural information processing systems, vol. 33, pp. 19 884–19 895, 2020.
19. M. Watter, J. Springenberg, J. Boedecker, and M. Riedmiller, “Embed to control: A locally linear latent dynamics model for control from raw images,” Advances in neural information processing systems, vol. 28, 2015.
20. D. Ha and J. Schmidhuber, “World models,” arXiv preprint arXiv:1803.10122, 2018.
21. V. Mnih, K. Kavukcuoglu, D. Silver, et al., Playing atari with deep reinforcement learning, 2013. arXiv: 1312. 5602 [cs.LG].
22. I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT Press, 2016.
23. S. Rifai, P. Vincent, X. Muller, X. Glorot, and Y. Bengio, “Contractive auto-encoders: Explicit invariance during feature extraction,” in Proceedings of the 28th international conference on international conference on machine learning, 2011, pp. 833–840.
24. O. Gym, Openai gym, https://www.gymlibrary. dev.
25. F.-q. Chen, Y. Wu, G.-d. Zhao, J.-m. Zhang, M. Zhu, and J. Bai, Contractive de-noising auto-encoder, 2014. arXiv: 1305.4076 [cs.LG].