IoT-Based Energy Consumption Prediction Using Transformers

Abstract

With the advancement of various IoT-based systems, the amount of data is steadily increasing. The increase of data on a daily basis is essential for decision-makers to assess current situations and formulate future policies. Among the various types of data, time-series data presents a challenging relationship between current and future dependencies. Time-series prediction aims to forecast future values of target variables by leveraging insights gained from past data points. Recent advancements in deep learning-based algorithms have surpassed traditional machine learning-based algorithms for time-series in IoT systems. In this study, we employ Enc & Dec Transformer, the latest advancements in neural networks for time-series prediction problems. The obtained results were compared with Encoder-only and Decoder-only Transformer blocks as well as well-known recurrent based algorithms, including 1D-CNN, RNN, LSTM, and GRU. To validate our approach, we utilize three different univariate time-series datasets collected on an hourly basis, focusing on energy consumption within IoT systems. Our results demonstrate that our proposed Transformer model outperforms its counterparts, achieving a minimum Mean Squared Error (MSE) of 0.020 on small, 0.008 on medium, and 0.006 on large-sized datasets.

Keywords

• Transformers
• Time-Series
• Prediction
• IoT

References

1. Adhikari, R., & Agrawal, R.K. (2013). An Introductory Study on Time Series Modeling and Forecasting. ArXiv, abs/1302.6613. https://doi.org/10.48550/arXiv.1302.6613
2. Afanasieva, T., & Platov, P. (2019). The Study of Recurrent Neuron Networks based on GRU and LSTM in Time Series Forecasting. In ITISE 2019. Proceedings of papers. Vol 1 (pp. 12). Granada, Spain: International Conference on Time Series and Forecasting. https://itise.ugr.es/ITISE2019_Vol1.pdf
3. Akiba, T., Sano, S., Yanase, T., Ohta, T., & Koyama, M. (2019). Optuna: A Next-generation Hyperparameter Optimization Framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 2623–2631). Association for Computing Machinery. https://doi.org/10.1145/3292500.3330701
4. Box, G. E. P., Jenkins, G. M., Reinsel, G. C., & Ljung, G. M. (2015). Time Series Analysis: Forecasting and Control (5th ed.). Hoboken, NJ: John Wiley & Sons Inc. https://doi.org/10.1111/jtsa.12194
5. Cao, L. (2003). Support vector machines experts for time series forecasting. Neurocomputing, 51, 321-339. https://doi.org/10.1016/S0925-2312(02)00577-5
6. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020). End-to-End Object Detection with Transformers. CoRR, abs/2005.12872. https://doi.org/10.48550/arXiv.2005.12872
7. Cochrane, J. H. (1997). Time Series for Macroeconomics and Finance. Graduate School of Business, University of Chicago. Retrieved from http://www.fsb.miamioh.edu/lij14/672_notes_Cochrane
8. Coulibaly, P., & Baldwin, C. K. (2005). Nonstationary hydrological time series forecasting using nonlinear dynamic methods. Journal of Hydrology, 307(1–4), 164-174. https://doi.org/10.1016/j.jhydrol.2004.10.008

9. Das, A., Kong, W., Sen, R., & Zhou, Y. (2024). A decoder-only foundation model for time-series forecasting. ICML. https://doi.org/10.48550/arXiv.2310.10688
10. Haugsdal, E., Aune, E., & Ruocco, M. (2023). Persistence Initialization: a novel adaptation of the Transformer architecture for time-series prediction. Applied Intelligence, 53, 26781–26796. https://doi.org/10.1007/s10489-023-04927-4
11. Hipel, K. W., & McLeod, I. (1994). Time series modelling of water resources and environmental systems. In Proceedings of the International Conference on Systems, Man and Cybernetics (pp. 1-6). https://doi.org/10.1016/s0167-5648(08)x7026-1
12. Hu, C., Sun, Z., Li, C., Zhang, Y., & Xing, C. (2023). Survey of time-series data generation in IoT. Sensors, 23(15), 6976. https://doi.org/10.3390/s23156976
13. IoT Analytics (2023). state of IoT 2023: number of connected IoT devices growing 16% to 16.7 billion globally. https://iot-analytics.com/number-connected-iot
14. Javaid N., Jul 12, 2019. Implementing an RNN from scratch in Python: towards data science. https://towardsdatascience.com/recurrent-neural-networks-rnns-3f06d7653a85
15. Lara-Benítez, P., Gallego-Ledesma, L., Carranza-García, M., & Luna-Romera, J. M. (2021). Evaluation of the Transformer Architecture for Univariate Time Series Forecasting. In E. Alba et al. (Eds.), Advances in Artificial Intelligence. CAEPIA 2021. Lecture Notes in Computer Science (Vol. 12882). Springer, Cham. https://doi.org/10.1007/978-3-030-85713-4_11
16. Li, S., Jin, X., Xuan, Y., Zhou, X., Chen, W., Wang, Y.-X., & Yan, X. (2019). Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting. CoRR, abs/1907.00235. https://doi.org/10.48550/arXiv.1907.00235
17. Lim, B., Arık, S. Ö., Loeff, N., & Pfister, T. (2021). Temporal Fusion Transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 37(4), 1748–1764. https://doi.org/10.1016/j.ijforecast.2021.03.012
18. Ma, J., Shou, Z., Zareian, A., Mansour, H., Vetro, A., & Chang, S. (2019). CDSA: Cross-Dimensional Self-Attention for Multivariate, Geo-tagged Time Series Imputation. ArXiv, abs/1905.09904. https://doi.org/10.48550/arXiv.1905.09904
19. Markova, M. (2022). Convolutional neural networks for forex time series forecasting. AIP Conference Proceedings, 2459(1), 030024. https://doi.org/10.1063/5.0083533
20. Masum, S., Liu, Y., & Chiverton, J. (2018). Multi-step Time Series Forecasting of Electric Load Using Machine Learning Models. In L. Rutkowski, R. Scherer, M. Korytkowski, W. Pedrycz, R. Tadeusiewicz, & J. M. Zurada (Eds.), Artificial Intelligence and Soft Computing (pp. 148-159). Springer International Publishing. https://doi.org/10.1007/978-3-319-91253-0_15
21. Mo, Y., Wu, Q., Li, X., et al. (2021). Remaining useful life estimation via transformer encoder enhanced by a gated convolutional unit. Journal of Intelligent Manufacturing, 32, 1997–2006. https://doi.org/10.1007/s10845-021-01750-x
22. Mulla, R. (2019). Hourly Energy Consumption. PJM Interconnection LLC in Kaggle.
23. Nie, H., Liu, G., Liu, X., & Wang, Y. (2012). Hybrid of ARIMA and SVMs for Short-Term Load Forecasting. Energy Procedia, 16, 1455-1460. https://doi.org/10.1016/j.egypro.2012.01.229
24. Nor, M. E., Mohd Safuan, H., Md Shab, N. F., Asrul, M., Abdullah, A., Mohamad, N. A. I., & Lee, M. H. (2017). Neural network versus classical time series forecasting models. AIP Conference Proceedings, 1842(1), 030027. https://doi.org/10.1063/1.4982865
25. Ogunfowora, O., & Najjaran, H. (2023). A Transformer-based Framework for Multi-variate Time Series: A Remaining Useful Life Prediction Use Case. https://doi.org/10.48550/arXiv.2308.09884
26. Pashamokhtari, A. (2020). Dynamic inference on IoT network traffic using programmable telemetry and machine learning. In Proceedings of the 2020 19th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN) (pp. 371–372). https://doi.org/10.1109/IPSN48710.2020.00006
27. Raheem, I., Mubarak, N. M., Karri, R. R., et al. (2022). Forecasting of energy consumption by G20 countries using an adjacent accumulation grey model. Scientific Reports, 12, 13417. https://doi.org/10.1038/s41598-022-17505-4
28. Russell, S. J., & Norvig, P. (2020). (4th ed.). Artificial Intelligence: A Modern Approach. Prentice Hall Publishing.
29. Sahoo, B. B., Jha, R., Singh, A., et al. (2019). Long short-term memory (LSTM) recurrent neural network for low-flow hydrological time series forecasting. Acta Geophysica, 67, 1471–1481. https://doi.org/10.1007/s11600-019-00330-1
30. Shapi, M. K. M., Ramli, N. A., & Awalin, L. J. (2021). Energy consumption prediction by using machine learning for smart building: Case study in Malaysia. Developments in the Built Environment, 5, 100037. https://doi.org/10.1016/j.dibe.2020.100037
31. Shekhar, S., Bansode, A., & Salim, A. (2021). A Comparative study of Hyper-Parameter Optimization Tools. In 2021 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE) (pp. 1-6). Brisbane, Australia. https://doi.org/10.1109/CSDE53843.2021.9718485
32. Shi, J., Jain, M., & Narasimhan, G. (2022). Time Series Forecasting (TSF) Using Various Deep Learning Models. arXiv, 2204.11115. https://doi.org/10.48550/arXiv.2204.11115
33. Tealab, A. (2018). Time series forecasting using artificial neural networks methodologies: A systematic review. Future Computing and Informatics Journal, 3(2), 334-340. https://doi.org/10.1016/j.fcij.2018.10.003
34. Torres, J. F., Hadjout, D., Sebaa, A., Martínez-Álvarez, F., & Troncoso, A. (2021). Deep Learning for Time Series Forecasting: A Survey. Big Data, 9(1), 3-21. https://doi.org/10.1089/big.2020.0159
35. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need. CoRR, abs/1706.03762. https://doi.org/10.48550/arXiv.1706.03762
36. Wen, Q., Zhou, T., Zhang, C., Chen, W., Ma, Z., Yan, J., & Sun, L. (2023). Transformers in time series: A survey. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (pp. 759). Macao, P.R. China: International Joint Conferences on Artificial Intelligence. https://doi.org/10.24963/ijcai.2023/759
37. Wu, N., Green, B., Ben, X., & O'Banion, S. (2020). Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case. CoRR, abs/2001.08317. https://doi.org/10.48550/arXiv.2001.08317
38. Zeyer, A., Bahar, P., Irie, K., Schlüter, R., & Ney, H. (2019). A Comparison of Transformer and LSTM Encoder Decoder Models for ASR. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) (pp. 8-15). Singapore. https://doi.org/10.1109/ASRU46091.2019.9004025
39. Zhang, Q., Lipani, A., Kirnap, Ö., & Yilmaz, E. (2019). Self-Attentive Hawkes Processes. CoRR, abs/1907.07561. https://doi.org/10.48550/arXiv.1907.07561
40. Zuo, S., Jiang, H., Li, Z., Zhao, T., & Zha, H. (2020). Transformer Hawkes Process. CoRR, abs/2002.09291. https://doi.org/10.48550/arXiv.2002.09291