Human activity recognition and classification using of convolutional neural networks and recurrent neural networks

Abstract

Recently, by using the deep learning models, it has become easier to recognize the human activity with more accuracy than before by categorizing the activities that people are doing daily. Nowadays, with the extensive use of modern smartphones that have sensors, it has become easier to capture the data in raw format that has the movement details in three dimensions (X-Y-Z). In this paper, we utilized the open source WIreless Sensor Data Mining (WISDM) dataset which has six activities that are walking, jogging, standing, sitting, upstairs and downstairs. Each type of those activities consists of values in terms of (X, Y and Z) axes. We employed two types of deep learning algorithms that are Convolutional Neural Network (CNN) and Recurrent Neural Network - Long Short-Term Memory (RNN-LSTM). Our objective is to make a comparison between accuracy and loss after implementing the two models. We discovered that, when using the Convolutional Neural Network (CNN), the accuracy was 81%. However, the accuracy was 91% when using Recurrent Neural Network - Long Short-Term Memory (RNN-LSTM) and applying it on the same database. As a result, the Recurrent Neural Network - Long Short-Term Memory (RNN-LSTM) model outperformed the Convolutional Neural Network (CNN) model. 

Keywords

• Activity Type
• CNN
• Human Activity Recognition
• RNN-LSTM
• WISDM

References

1. C. Snehal and C. Peng-Wen and Z. Jiang and Y. Z. Joy, “Mobile Lifelogger – Recording, Indexing, and Understanding a Mobile User’s Life,” Second International ICST Conference, MobiCASE 2010: Mobile Computing, Applications, and Services: Proceeding of 2nd, 2010, Santa Clara, CA, USA, October 25-28, 2010. Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, 2012. pp. 263-281. DOI: https://doi.org/10.1007/978-3-642-29336-8_15.
2. F. Kilian and R. Daniel and T. Gerhard, “Unsupervised classifier self-calibration through repeated context occurences: is there robustness against sensor displacement to gain?,” 2009 International Symposium on Wearable Computers: Proceeding of 9th, 2009, Linz, Austria, 4-7 Sept. 2009. IEEE, 2009. pp. 77-84. DOI: 10.1109/ISWC.2009.12.
3. W. Pang and P. Huan-Kai and Z. Jiang and Z. Ying, “SensCare: Semi-automatic Activity Summarization System for Elderly Care,” Third International Conference, MobiCASE 2011: Mobile Computing, Applications, and Services: Proceeding of 3rd, 2011, Los Angeles, CA, USA, October 24-27, 2011. Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, 2012. pp. 1-19. DOI: http://dx.doi.org/10.1007/978-3-642-32320-1_1.
4. W. Pang and Z. Jiang and Y. Z. Joy, “MobiSens: A Versatile Mobile Sensing Platform for Real-World Applications,” Mobile Networks and Applications, vol. 18, pp. 60-80, 2013. DOI: https://doi.org/10.1007/s11036-012-0422-y.
5. Z. Ming and T. N. Le and Y. Bo and J. M. Ole and Z. Jiang and W. Pang and Z. Joy, “Convolutional Neural Networks for Human Activity Recognition using Mobile Sensors,” 6th International Conference on Mobile Computing, Applications and Services (MobiCASE): Proceeding of the 6th, 6th International Conference on Mobile Computing, Applications and Services, 6-7 Nov. 2014. IEEE, 2014. pp. 197-205. DOI: 10.4108/icst.mobicase.2014.257786.
6. C. Diane and D. F. Kyle and C. K. Narayanan, “Transfer Learning for Activity Recognition: A Survey,” Knowledge and Information Systems, vol. 36, pp. 537-556, 2013. DOI: https://doi.org/10.1007/s10115-013-0665-3.
7. V. Praneeth and D. Debraj and K. D. Sajal and B. Shekhar, “A-Wristocracy: Deep Learning on Wrist-worn Sensing for Recognition of User Complex Activities,” 2015 IEEE International Conference on Wearable and Implantable Body Sensor Networks (BSN): Proceeding of the 12th, Cambridge, MA, USA, 9-12 June 2015. IEEE, 2015. pp. 1-6. DOI: 10.1109/BSN.2015.7299406.
8. J. O. Francisco and R. Daniel, “Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition,” Sensors, vol. 16, pp. 1-25, 2016. DOI: https://doi.org/10.3390/s16010115.

9. Z. Mi and A. S. Alexander, “A Feature Selection-Based Framework for Human Activity Recognition Using Wearable Multimodal Sensors,” 6th International Conference on Body Area Networks: Proceeding of 6th, 2011, Beijing, China, November 7 - 8, 2011. ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering) Brussels, Belgium, 2011. pp. 92-98. DOI: 10.4108/icst.bodynets.2011.247018.
10. W. P. Schalk and M. Reza, “Human Activity Recognition Using LSTM-RNN Deep Neural Network Architecture,” 2019 IEEE 2nd Wireless Africa Conference (WAC): Proceeding of the 2nd, 2019, Pretoria, South Africa. IEEE, 2019. pp. 1-5. DOI: 10.1109/AFRICA.2019.8843403.
11. C. Wen-Hui and A. B. B. Carlos and T. Chih-Hao, “LSTM-RNNs Combined with Scene Information for Human Activity Recognition,” 2017 IEEE 19th International Conference on e-Health Networking, Applications and Services (Healthcom): Proceeding of 19th, 2017, Dalian, China, 12-15 Oct. 2017. IEEE, 2017. pp. 1-6. DOI: 10.1109/HealthCom.2017.8210846.
12. R. K. Jennifer and M. W. Gary and A. M. Samuel, “Activity Recognition using Cell Phone Accelerometers,” ACM SIGKDD Explorations Newsletter, vol. 12, 2011. Association for Computing Machinery New York, NY, United States, 2011. DOI: https://doi.org/10.1145/1964897.1964918. WISDM (Wireless Sensor Data Mining) Project. Fordham University, Department of Computer and Information Science, http://www.cis.fordham.edu/wisdm/dataset.php [accessed date 10 May 2020]
13. MathWorks, Accelerometer Documentation, Measure linear acceleration along X, Y, and Z axes [online] https://www.mathworks.com/help/supportpkg/android/ref/accelerometer.html [accessed date 20 July 2020]
14. L. Yann and B. Yoshua, The handbook of brain theory and neural networks. MIT Press55 Hayward St. Cambridge, MA, United States, 1998.
15. C. Heeryon and M. Y. Sang, “Divide and Conquer-Based 1D CNN Human Activity Recognition Using Test Data Sharpening,” Sensors, vol. 18, pp. 1-24, 2018. DOI: 10.3390/s18040055.
16. X. Li and X. Si and L. Nie and J. Li and R. Ding and D. Zhan and D. Chu, “Understanding and Improving Deep Neural Network for Activity Recognition,” 11th EAI International Conference on Mobile Multimedia Communications: Proceeding of 11th, 2018, Qingdao, People's Republic of China. EAI, 2018. pp. 1-10. DOI: http://dx.doi.org/10.4108/eai.21-6-2018.2276632.
17. Z. U. Md and T. Jim, “Activity Recognition Using Smartphone Sensors, Robust Features, and Recurrent Neural Network,” 2019 13th International Symposium on Medical Information and Communication Technology (ISMICT): Proceeding of 13th, 2019, Oslo, Norway. IEEE, 2019. pp. 1-6. DOI: 10.1109/ISMICT.2019.8743759.
18. S.U. Park and J.H. Park and M.A. Al-masni and M.A. Al-antari and Md.Z. Uddin and T.-S. Kim, “A Depth Camera-based Human Activity Recognition via Deep Learning Recurrent Neural Network for Health and Social Care Services,” Procedia Computer Science, vol. 100, pp. 78-84, 2016. DOI: https://doi.org/10.1016/j.procs.2016.09.126.