TY  - JOUR
T1  - IoT-Based Energy Consumption Prediction Using Transformers
AU  - Alıoghlı, Abdul Amir
AU  - Yıldırım Okay, Feyza
PY  - 2024
DA  - June
Y2  - 2024
DO  - 10.54287/gujsa.1438011
JF  - Gazi University Journal of Science Part A: Engineering and Innovation
JO  - GU J Sci, Part A
PB  - Gazi University
WT  - DergiPark
SN  - 2147-9542
SP  - 304
EP  - 323
VL  - 11
IS  - 2
LA  - en
AB  - With the advancement of various IoT-based systems, the amount of data is steadily increasing. The increase of data on a daily basis is essential for decision-makers to assess current situations and formulate future policies. Among the various types of data, time-series data presents a challenging relationship between current and future dependencies. Time-series prediction aims to forecast future values of target variables by leveraging insights gained from past data points. Recent advancements in deep learning-based algorithms have surpassed traditional machine learning-based algorithms for time-series in IoT systems. In this study, we employ Enc & Dec Transformer, the latest advancements in neural networks for time-series prediction problems. The obtained results were compared with Encoder-only and Decoder-only Transformer blocks as well as well-known recurrent based algorithms, including 1D-CNN, RNN, LSTM, and GRU. To validate our approach, we utilize three different univariate time-series datasets collected on an hourly basis, focusing on energy consumption within IoT systems. Our results demonstrate that our proposed Transformer model outperforms its counterparts, achieving a minimum Mean Squared Error (MSE) of 0.020 on small, 0.008 on medium, and 0.006 on large-sized datasets.
KW  - Transformers
KW  - Time-Series
KW  - Prediction
KW  - IoT
CR  - Adhikari, R., & Agrawal, R.K. (2013). An Introductory Study on Time Series Modeling and Forecasting. ArXiv, abs/1302.6613. https://doi.org/10.48550/arXiv.1302.6613
CR  - Afanasieva, T., & Platov, P. (2019). The Study of Recurrent Neuron Networks based on GRU and LSTM in Time Series Forecasting. In ITISE 2019. Proceedings of papers. Vol 1 (pp. 12). Granada, Spain: International Conference on Time Series and Forecasting. https://itise.ugr.es/ITISE2019_Vol1.pdf
CR  - Akiba, T., Sano, S., Yanase, T., Ohta, T., & Koyama, M. (2019). Optuna: A Next-generation Hyperparameter Optimization Framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 2623–2631). Association for Computing Machinery. https://doi.org/10.1145/3292500.3330701
CR  - Box, G. E. P., Jenkins, G. M., Reinsel, G. C., & Ljung, G. M. (2015). Time Series Analysis: Forecasting and Control (5th ed.). Hoboken, NJ: John Wiley & Sons Inc. https://doi.org/10.1111/jtsa.12194
CR  - Cao, L. (2003). Support vector machines experts for time series forecasting. Neurocomputing, 51, 321-339. https://doi.org/10.1016/S0925-2312(02)00577-5
CR  - Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020). End-to-End Object Detection with Transformers. CoRR, abs/2005.12872. https://doi.org/10.48550/arXiv.2005.12872
CR  - Cochrane, J. H. (1997). Time Series for Macroeconomics and Finance. Graduate School of Business, University of Chicago. Retrieved from http://www.fsb.miamioh.edu/lij14/672_notes_Cochrane
CR  - Coulibaly, P., & Baldwin, C. K. (2005). Nonstationary hydrological time series forecasting using nonlinear dynamic methods. Journal of Hydrology, 307(1–4), 164-174. https://doi.org/10.1016/j.jhydrol.2004.10.008
CR  - Das, A., Kong, W., Sen, R., & Zhou, Y. (2024). A decoder-only foundation model for time-series forecasting. ICML. https://doi.org/10.48550/arXiv.2310.10688
CR  - Haugsdal, E., Aune, E., & Ruocco, M. (2023). Persistence Initialization: a novel adaptation of the Transformer architecture for time-series prediction. Applied Intelligence, 53, 26781–26796. https://doi.org/10.1007/s10489-023-04927-4
CR  - Hipel, K. W., & McLeod, I. (1994). Time series modelling of water resources and environmental systems. In Proceedings of the International Conference on Systems, Man and Cybernetics (pp. 1-6). https://doi.org/10.1016/s0167-5648(08)x7026-1
CR  - Hu, C., Sun, Z., Li, C., Zhang, Y., & Xing, C. (2023). Survey of time-series data generation in IoT. Sensors, 23(15), 6976. https://doi.org/10.3390/s23156976
CR  - IoT Analytics (2023). state of IoT 2023: number of connected IoT devices growing 16% to 16.7 billion globally. https://iot-analytics.com/number-connected-iot
CR  - Javaid N., Jul 12, 2019. Implementing an RNN from scratch in Python: towards data science. https://towardsdatascience.com/recurrent-neural-networks-rnns-3f06d7653a85
CR  - Lara-Benítez, P., Gallego-Ledesma, L., Carranza-García, M., & Luna-Romera, J. M. (2021). Evaluation of the Transformer Architecture for Univariate Time Series Forecasting. In E. Alba et al. (Eds.), Advances in Artificial Intelligence. CAEPIA 2021. Lecture Notes in Computer Science (Vol. 12882). Springer, Cham. https://doi.org/10.1007/978-3-030-85713-4_11
CR  - Li, S., Jin, X., Xuan, Y., Zhou, X., Chen, W., Wang, Y.-X., & Yan, X. (2019). Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting. CoRR, abs/1907.00235. https://doi.org/10.48550/arXiv.1907.00235
CR  - Lim, B., Arık, S. Ö., Loeff, N., & Pfister, T. (2021). Temporal Fusion Transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 37(4), 1748–1764. https://doi.org/10.1016/j.ijforecast.2021.03.012
CR  - Ma, J., Shou, Z., Zareian, A., Mansour, H., Vetro, A., & Chang, S. (2019). CDSA: Cross-Dimensional Self-Attention for Multivariate, Geo-tagged Time Series Imputation. ArXiv, abs/1905.09904. https://doi.org/10.48550/arXiv.1905.09904
CR  - Markova, M. (2022). Convolutional neural networks for forex time series forecasting. AIP Conference Proceedings, 2459(1), 030024. https://doi.org/10.1063/5.0083533
CR  - Masum, S., Liu, Y., & Chiverton, J. (2018). Multi-step Time Series Forecasting of Electric Load Using Machine Learning Models. In L. Rutkowski, R. Scherer, M. Korytkowski, W. Pedrycz, R. Tadeusiewicz, & J. M. Zurada (Eds.), Artificial Intelligence and Soft Computing (pp. 148-159). Springer International Publishing. https://doi.org/10.1007/978-3-319-91253-0_15
CR  - Mo, Y., Wu, Q., Li, X., et al. (2021). Remaining useful life estimation via transformer encoder enhanced by a gated convolutional unit. Journal of Intelligent Manufacturing, 32, 1997–2006. https://doi.org/10.1007/s10845-021-01750-x
CR  - Mulla, R. (2019). Hourly Energy Consumption. PJM Interconnection LLC in Kaggle.
CR  - Nie, H., Liu, G., Liu, X., & Wang, Y. (2012). Hybrid of ARIMA and SVMs for Short-Term Load Forecasting. Energy Procedia, 16, 1455-1460. https://doi.org/10.1016/j.egypro.2012.01.229
CR  - Nor, M. E., Mohd Safuan, H., Md Shab, N. F., Asrul, M., Abdullah, A., Mohamad, N. A. I., & Lee, M. H. (2017). Neural network versus classical time series forecasting models. AIP Conference Proceedings, 1842(1), 030027. https://doi.org/10.1063/1.4982865
CR  - Ogunfowora, O., & Najjaran, H. (2023). A Transformer-based Framework for Multi-variate Time Series: A Remaining Useful Life Prediction Use Case. https://doi.org/10.48550/arXiv.2308.09884
CR  - Pashamokhtari, A. (2020). Dynamic inference on IoT network traffic using programmable telemetry and machine learning. In Proceedings of the 2020 19th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN) (pp. 371–372). https://doi.org/10.1109/IPSN48710.2020.00006
CR  - Raheem, I., Mubarak, N. M., Karri, R. R., et al. (2022). Forecasting of energy consumption by G20 countries using an adjacent accumulation grey model. Scientific Reports, 12, 13417. https://doi.org/10.1038/s41598-022-17505-4
CR  - Russell, S. J., & Norvig, P. (2020). (4th ed.). Artificial Intelligence: A Modern Approach. Prentice Hall Publishing.
CR  - Sahoo, B. B., Jha, R., Singh, A., et al. (2019). Long short-term memory (LSTM) recurrent neural network for low-flow hydrological time series forecasting. Acta Geophysica, 67, 1471–1481. https://doi.org/10.1007/s11600-019-00330-1
CR  - Shapi, M. K. M., Ramli, N. A., & Awalin, L. J. (2021). Energy consumption prediction by using machine learning for smart building: Case study in Malaysia. Developments in the Built Environment, 5, 100037. https://doi.org/10.1016/j.dibe.2020.100037
CR  - Shekhar, S., Bansode, A., & Salim, A. (2021). A Comparative study of Hyper-Parameter Optimization Tools. In 2021 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE) (pp. 1-6). Brisbane, Australia. https://doi.org/10.1109/CSDE53843.2021.9718485
CR  - Shi, J., Jain, M., & Narasimhan, G. (2022). Time Series Forecasting (TSF) Using Various Deep Learning Models. arXiv, 2204.11115. https://doi.org/10.48550/arXiv.2204.11115
CR  - Tealab, A. (2018). Time series forecasting using artificial neural networks methodologies: A systematic review. Future Computing and Informatics Journal, 3(2), 334-340. https://doi.org/10.1016/j.fcij.2018.10.003
CR  - Torres, J. F., Hadjout, D., Sebaa, A., Martínez-Álvarez, F., & Troncoso, A. (2021). Deep Learning for Time Series Forecasting: A Survey. Big Data, 9(1), 3-21. https://doi.org/10.1089/big.2020.0159
CR  - Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need. CoRR, abs/1706.03762. https://doi.org/10.48550/arXiv.1706.03762
CR  - Wen, Q., Zhou, T., Zhang, C., Chen, W., Ma, Z., Yan, J., & Sun, L. (2023). Transformers in time series: A survey. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (pp. 759). Macao, P.R. China: International Joint Conferences on Artificial Intelligence. https://doi.org/10.24963/ijcai.2023/759
CR  - Wu, N., Green, B., Ben, X., & O'Banion, S. (2020). Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case. CoRR, abs/2001.08317. https://doi.org/10.48550/arXiv.2001.08317
CR  - Zeyer, A., Bahar, P., Irie, K., Schlüter, R., & Ney, H. (2019). A Comparison of Transformer and LSTM Encoder Decoder Models for ASR. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) (pp. 8-15). Singapore. https://doi.org/10.1109/ASRU46091.2019.9004025
CR  - Zhang, Q., Lipani, A., Kirnap, Ö., & Yilmaz, E. (2019). Self-Attentive Hawkes Processes. CoRR, abs/1907.07561. https://doi.org/10.48550/arXiv.1907.07561
CR  - Zuo, S., Jiang, H., Li, Z., Zhao, T., & Zha, H. (2020). Transformer Hawkes Process. CoRR, abs/2002.09291. https://doi.org/10.48550/arXiv.2002.09291
UR  - https://doi.org/10.54287/gujsa.1438011
L1  - https://dergipark.org.tr/en/download/article-file/3733055
ER  -