IoT-Based Energy Consumption Prediction Using Transformers

Year 2024, Volume: 11 Issue: 2, 304 - 323, 29.06.2024

Abdul Amir Alıoghlı Feyza Yıldırım Okay

https://doi.org/10.54287/gujsa.1438011

Abstract

With the advancement of various IoT-based systems, the amount of data is steadily increasing. The increase of data on a daily basis is essential for decision-makers to assess current situations and formulate future policies. Among the various types of data, time-series data presents a challenging relationship between current and future dependencies. Time-series prediction aims to forecast future values of target variables by leveraging insights gained from past data points. Recent advancements in deep learning-based algorithms have surpassed traditional machine learning-based algorithms for time-series in IoT systems. In this study, we employ Enc & Dec Transformer, the latest advancements in neural networks for time-series prediction problems. The obtained results were compared with Encoder-only and Decoder-only Transformer blocks as well as well-known recurrent based algorithms, including 1D-CNN, RNN, LSTM, and GRU. To validate our approach, we utilize three different univariate time-series datasets collected on an hourly basis, focusing on energy consumption within IoT systems. Our results demonstrate that our proposed Transformer model outperforms its counterparts, achieving a minimum Mean Squared Error (MSE) of 0.020 on small, 0.008 on medium, and 0.006 on large-sized datasets.

Keywords

Transformers, Time-Series, Prediction, IoT

References

• Adhikari, R., & Agrawal, R.K. (2013). An Introductory Study on Time Series Modeling and Forecasting. ArXiv, abs/1302.6613. https://doi.org/10.48550/arXiv.1302.6613
• Afanasieva, T., & Platov, P. (2019). The Study of Recurrent Neuron Networks based on GRU and LSTM in Time Series Forecasting. In ITISE 2019. Proceedings of papers. Vol 1 (pp. 12). Granada, Spain: International Conference on Time Series and Forecasting. https://itise.ugr.es/ITISE2019_Vol1.pdf
• Akiba, T., Sano, S., Yanase, T., Ohta, T., & Koyama, M. (2019). Optuna: A Next-generation Hyperparameter Optimization Framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 2623–2631). Association for Computing Machinery. https://doi.org/10.1145/3292500.3330701
• Box, G. E. P., Jenkins, G. M., Reinsel, G. C., & Ljung, G. M. (2015). Time Series Analysis: Forecasting and Control (5th ed.). Hoboken, NJ: John Wiley & Sons Inc. https://doi.org/10.1111/jtsa.12194
• Cao, L. (2003). Support vector machines experts for time series forecasting. Neurocomputing, 51, 321-339. https://doi.org/10.1016/S0925-2312(02)00577-5
• Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020). End-to-End Object Detection with Transformers. CoRR, abs/2005.12872. https://doi.org/10.48550/arXiv.2005.12872
• Cochrane, J. H. (1997). Time Series for Macroeconomics and Finance. Graduate School of Business, University of Chicago. Retrieved from http://www.fsb.miamioh.edu/lij14/672_notes_Cochrane
• Coulibaly, P., & Baldwin, C. K. (2005). Nonstationary hydrological time series forecasting using nonlinear dynamic methods. Journal of Hydrology, 307(1–4), 164-174. https://doi.org/10.1016/j.jhydrol.2004.10.008
• Das, A., Kong, W., Sen, R., & Zhou, Y. (2024). A decoder-only foundation model for time-series forecasting. ICML. https://doi.org/10.48550/arXiv.2310.10688
• Haugsdal, E., Aune, E., & Ruocco, M. (2023). Persistence Initialization: a novel adaptation of the Transformer architecture for time-series prediction. Applied Intelligence, 53, 26781–26796. https://doi.org/10.1007/s10489-023-04927-4
• Hipel, K. W., & McLeod, I. (1994). Time series modelling of water resources and environmental systems. In Proceedings of the International Conference on Systems, Man and Cybernetics (pp. 1-6). https://doi.org/10.1016/s0167-5648(08)x7026-1
• Hu, C., Sun, Z., Li, C., Zhang, Y., & Xing, C. (2023). Survey of time-series data generation in IoT. Sensors, 23(15), 6976. https://doi.org/10.3390/s23156976
• IoT Analytics (2023). state of IoT 2023: number of connected IoT devices growing 16% to 16.7 billion globally. https://iot-analytics.com/number-connected-iot
• Javaid N., Jul 12, 2019. Implementing an RNN from scratch in Python: towards data science. https://towardsdatascience.com/recurrent-neural-networks-rnns-3f06d7653a85
• Lara-Benítez, P., Gallego-Ledesma, L., Carranza-García, M., & Luna-Romera, J. M. (2021). Evaluation of the Transformer Architecture for Univariate Time Series Forecasting. In E. Alba et al. (Eds.), Advances in Artificial Intelligence. CAEPIA 2021. Lecture Notes in Computer Science (Vol. 12882). Springer, Cham. https://doi.org/10.1007/978-3-030-85713-4_11
• Li, S., Jin, X., Xuan, Y., Zhou, X., Chen, W., Wang, Y.-X., & Yan, X. (2019). Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting. CoRR, abs/1907.00235. https://doi.org/10.48550/arXiv.1907.00235
• Lim, B., Arık, S. Ö., Loeff, N., & Pfister, T. (2021). Temporal Fusion Transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 37(4), 1748–1764. https://doi.org/10.1016/j.ijforecast.2021.03.012
• Ma, J., Shou, Z., Zareian, A., Mansour, H., Vetro, A., & Chang, S. (2019). CDSA: Cross-Dimensional Self-Attention for Multivariate, Geo-tagged Time Series Imputation. ArXiv, abs/1905.09904. https://doi.org/10.48550/arXiv.1905.09904
• Markova, M. (2022). Convolutional neural networks for forex time series forecasting. AIP Conference Proceedings, 2459(1), 030024. https://doi.org/10.1063/5.0083533
• Masum, S., Liu, Y., & Chiverton, J. (2018). Multi-step Time Series Forecasting of Electric Load Using Machine Learning Models. In L. Rutkowski, R. Scherer, M. Korytkowski, W. Pedrycz, R. Tadeusiewicz, & J. M. Zurada (Eds.), Artificial Intelligence and Soft Computing (pp. 148-159). Springer International Publishing. https://doi.org/10.1007/978-3-319-91253-0_15
• Mo, Y., Wu, Q., Li, X., et al. (2021). Remaining useful life estimation via transformer encoder enhanced by a gated convolutional unit. Journal of Intelligent Manufacturing, 32, 1997–2006. https://doi.org/10.1007/s10845-021-01750-x
• Mulla, R. (2019). Hourly Energy Consumption. PJM Interconnection LLC in Kaggle.
• Nie, H., Liu, G., Liu, X., & Wang, Y. (2012). Hybrid of ARIMA and SVMs for Short-Term Load Forecasting. Energy Procedia, 16, 1455-1460. https://doi.org/10.1016/j.egypro.2012.01.229
• Nor, M. E., Mohd Safuan, H., Md Shab, N. F., Asrul, M., Abdullah, A., Mohamad, N. A. I., & Lee, M. H. (2017). Neural network versus classical time series forecasting models. AIP Conference Proceedings, 1842(1), 030027. https://doi.org/10.1063/1.4982865
• Ogunfowora, O., & Najjaran, H. (2023). A Transformer-based Framework for Multi-variate Time Series: A Remaining Useful Life Prediction Use Case. https://doi.org/10.48550/arXiv.2308.09884
• Pashamokhtari, A. (2020). Dynamic inference on IoT network traffic using programmable telemetry and machine learning. In Proceedings of the 2020 19th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN) (pp. 371–372). https://doi.org/10.1109/IPSN48710.2020.00006
• Raheem, I., Mubarak, N. M., Karri, R. R., et al. (2022). Forecasting of energy consumption by G20 countries using an adjacent accumulation grey model. Scientific Reports, 12, 13417. https://doi.org/10.1038/s41598-022-17505-4
• Russell, S. J., & Norvig, P. (2020). (4th ed.). Artificial Intelligence: A Modern Approach. Prentice Hall Publishing.
• Sahoo, B. B., Jha, R., Singh, A., et al. (2019). Long short-term memory (LSTM) recurrent neural network for low-flow hydrological time series forecasting. Acta Geophysica, 67, 1471–1481. https://doi.org/10.1007/s11600-019-00330-1
• Shapi, M. K. M., Ramli, N. A., & Awalin, L. J. (2021). Energy consumption prediction by using machine learning for smart building: Case study in Malaysia. Developments in the Built Environment, 5, 100037. https://doi.org/10.1016/j.dibe.2020.100037
• Shekhar, S., Bansode, A., & Salim, A. (2021). A Comparative study of Hyper-Parameter Optimization Tools. In 2021 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE) (pp. 1-6). Brisbane, Australia. https://doi.org/10.1109/CSDE53843.2021.9718485
• Shi, J., Jain, M., & Narasimhan, G. (2022). Time Series Forecasting (TSF) Using Various Deep Learning Models. arXiv, 2204.11115. https://doi.org/10.48550/arXiv.2204.11115
• Tealab, A. (2018). Time series forecasting using artificial neural networks methodologies: A systematic review. Future Computing and Informatics Journal, 3(2), 334-340. https://doi.org/10.1016/j.fcij.2018.10.003
• Torres, J. F., Hadjout, D., Sebaa, A., Martínez-Álvarez, F., & Troncoso, A. (2021). Deep Learning for Time Series Forecasting: A Survey. Big Data, 9(1), 3-21. https://doi.org/10.1089/big.2020.0159
• Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need. CoRR, abs/1706.03762. https://doi.org/10.48550/arXiv.1706.03762
• Wen, Q., Zhou, T., Zhang, C., Chen, W., Ma, Z., Yan, J., & Sun, L. (2023). Transformers in time series: A survey. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (pp. 759). Macao, P.R. China: International Joint Conferences on Artificial Intelligence. https://doi.org/10.24963/ijcai.2023/759
• Wu, N., Green, B., Ben, X., & O'Banion, S. (2020). Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case. CoRR, abs/2001.08317. https://doi.org/10.48550/arXiv.2001.08317
• Zeyer, A., Bahar, P., Irie, K., Schlüter, R., & Ney, H. (2019). A Comparison of Transformer and LSTM Encoder Decoder Models for ASR. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) (pp. 8-15). Singapore. https://doi.org/10.1109/ASRU46091.2019.9004025
• Zhang, Q., Lipani, A., Kirnap, Ö., & Yilmaz, E. (2019). Self-Attentive Hawkes Processes. CoRR, abs/1907.07561. https://doi.org/10.48550/arXiv.1907.07561
• Zuo, S., Jiang, H., Li, Z., Zhao, T., & Zha, H. (2020). Transformer Hawkes Process. CoRR, abs/2002.09291. https://doi.org/10.48550/arXiv.2002.09291