Simple and effective descriptive analysis of missing data anomalies in smart home energy consumption readings

Abstract

Smart grids evolution is ramping up in the global energy scenario by offering deregulated markets, demand-side management, prosumer culture, demand response, contingency forecasting, outage management, etc., functionalities. These functionalities help to manage the grid effectively by taking informed decisions timely. Further, the progressive developments in information and communication technologies embedding smartness in the power grids. Especially, smart homes are playing a key role, which possesses the communication between various devices/appliances and collect their functional data in terms of energy consumption readings, timestamp, etc. However, the availability of high-quality data is always desired to achieve superior benefits with respect to all the above-mentioned functionalities. But, the failures of communication networks, metering devices, server station issues, etc., create anomalies in the data collection. Hence, there is a dire need of identifying the ways of analyzing the smart home data to find the irregularities that occurred because of aforesaid failures. Especially, it has been a common problem to see missing data at some particular instants in the overall database captured. In this view, this paper proposes a simple and effective descriptive analysis to find missing data anomalies in smart home energy consumption data. A real-time dataset is used to execute the proposed method. For which, a clear enumeration of missing data is visualized using comprehensive simulation results. This helps to realize the actual problems that are hidden in the energy consumption data.

Keywords

• Data anomalies
• Missing data
• Smart home data analytics
• Tracebase dataset
• Visualization

References

1. [1] Moharm, K. State of the art in big data applications in microgrid: A review. Advanced Engineering Informatics 2019; 42: 100945, DOI: https://doi.org/10.1016/j.aei.2019.100945
2. [2] Wang, Y, Qixin, C. Review of smart meter data analytics: Applications, methodologies, and challenges. IEEE Transactions on Smart Grid 2019; 10: 3125-3148, DOI: https://doi.org/10.1109/TSG.2018.2818167
3. [3] Bhattarai, PB, Paudyal, S, Luo, Y, Manish, M, Cheung, K, Reinaldo, T, Rob, H, Kurt, SM, Rui, Z, Power, Z, Milos, M, Song, Z, Xiaping, Z. Big data analytics in smart grids: State-of-the-art, challenges, opportunities, and future directions. IET Smart Grid 2019; 2: 141-154, DOI: https://doi.org/10.1049/iet-stg.2018.0261
4. [4] Ghorbanian, M, Dolatabadi, SH, Pierluigi, S. Big data issues in smart grids: A survey. IEEE Systems Journal 2019; 13: 4158-4168, DOI: https://doi.org/10.1109/JSYST.2019.2931879
5. [5] Zhang, Y, Tao, H, Bompard, EF. Big data analytics in smart grids: A review. Energy Informatics 2019; 8, DOI: https://doi.org/10.1186/s42162-018-0007-5
6. [6] Wen, L, Zhou K, Shanlin, Y, Li, L. Compression of smart meter big data: A survey. Renewable and Sustainable Energy Reviews 2018; 91: 59-69, DOI: https://doi.org/10.1016/j.rser.2018.03.088
7. [7] Tu, C, Xi, H, Shuai, Z, Fie, J. Big data issues in smart grid – A review. Renewable and Sustainable Energy Reviews 2017; 79: 1099-1107, DOI: https://doi.org/10.1016/j.rser.2017.05.134
8. [8] Jiang, H, Kun, W, Wang, Y, Gao, M, Zhang, Y. Energy big data: A survey. IEEE Access 2016; 4: 3844-3861, DOI: https://doi.org/10.1109/ACCESS.2016.2580581

9. [9] Kaile, Z, Fu, C, Yang, S. Big data driven smart energy management: From big data to big insights. Renewable and Sustainable Energy Reviews 2016; 56: 215-225, DOI: https://doi.org/10.1016/j.rser.2015.11.050
10. [10] Vasilica, SO, Bara, A, George, BT, Maria, IC, Botezatu, AM. Insights into demand-side management with big data analytics in electricity consumers’ behaviour. Computers and Electrical Engineering 2021; 89: 106902, DOI: https://doi.org/10.1016/j.compeleceng.2020.106902
11. [11] Syed, D, Zainab, A, Refaat, SS, Haitham, AR, Bouhali, O. Smart grid big data analytics: Survey of technologies, techniques, and applications. IEEE Access 2020; 9: 59564-59585, DOI: https://doi.org/10.1109/ACCESS.2020.3041178
12. [12] Kumar, SV, Tianyi, W, Aggour, SK, Wang, P, Hart, JP, Yan, W. Big data analysis of massive PMU datasets: A data platform perspective. In: 2021 IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT); 16-18 February 2021: IEEE, Washington, DC, USA: pp. 1-5.
13. [13] Babar, M, Khattak, SA, Jan, MA, Tariq, UM. Energy aware smart city management system using data analytics and Internet of Things. Sustainable Energy Technologies and Assessments 2021; 44: 100992, DOI: https://doi.org/10.1016/j.seta.2021.100992
14. [14] Kumar, DP, Saptarshi, D. Smart grid architecture model for control, optimization and data analytics of future power networks with more renewable energy. Journal of Cleaner Production 2021; 301: 126877, DOI: https://doi.org/10.1016/j.jclepro.2021.126877
15. [15] Mohajeri, M, Ghassemi, A, Gulliver, TA. Fast big data analytics for smart meter data. IEEE Open Journal of the Communications Society 2020; 1: 1864-1871, DOI: https://doi.org/10.1109/OJCOMS.2020.3038590
16. [16] Rossi, B, Chren, S. Smart grids data analysis: A systematic mapping study. IEEE Transactions on Industrial Informatics 2020; 16: 3619-3639, DOI: https://doi.org/10.1109/TII.2019.2954098
17. [17] Ushakova, A, Mikhaylov, JS. Big data to the rescue? Challenges in analysing granular household electricity consumption in the United Kingdom. Energy Research & Social Science 2020; 64: 101428, DOI: https://doi.org/10.1016/j.erss.2020.101428
18. [18] Shang, C, Fengqi, Y. Data analytics and machine learning for smart process manufacturing: Recent advances and perspectives in the big data era. Engineering 2019; 5: 1010-1016, DOI: https://doi.org/10.1016/j.eng.2019.01.019
19. [19] Tom, W, Jin, N, Falch, P, Joshua, T. A big data platform for smart meter data analytics. Computers in Industry 2019; 105: 250-259, DOI: https://doi.org/10.1016/j.compind.2018.12.010
20. [20] Cordon, I, Luengo, J, Garcia, S, Herrera, F, Francisco, C. Smartdata: Data preprocessing to achieve smart data in R. Neurocomputing 2019; 360: 1-13, DOI: https://doi.org/10.1016/j.neucom.2019.06.006
21. [21] Scott, ME. The role of statistics in the era of big data: Crucial, critical and under-valued. Statistics and Probability Letters 2018; 136: 20-24, DOI: https://doi.org/10.1016/j.spl.2018.02.050
22. [22] Tao, H. Big data analytics making the smart grid smarter. IEEE Power & Energy Magazine 2018; 16: 12-16, DOI: https://doi.org/10.1109/MPE.2018.2801440
23. [23] Kang, C, Yi, W, Xue, Y, Mu, G, Liao, R. Big data analytics in China’s electric power industry: Modern information, communication technologies, and millions of smart meters. IEEE Power & Energy Magazine 2018; 16: 54-65, DOI: https://doi.org/10.1109/MPE.2018.2790819
24. [24] Eckroth, J. A course on big data analytics. Journal of Parallel Distrib. Comput. 2018; 118: 166-176, DOI: https://doi.org/10.1016/j.jpdc.2018.02.019
25. [25] Kunjin, C, He, Z, Shan, XW, Jun, H, Li, L, Jinliang, H. Learning-based data analytics: Moving towards transparent power grids. CSEE Journal of Power and Energy Systems 2018; 4: 67-82, DOI: https://doi.org/10.17775/CSEEJPES.2017.01070
26. [26] Munshi, AA, Yasser, ARIM. Big data framework for analytics in smart grids. Electric Power Systems Research 2017; 151: 369-380, DOI: https://doi.org/10.1016/j.epsr.2017.06.006
27. [27] Tarek, A, Dev, S, Visser, L, Murhaf, H, Van, SW. A systematic analysis of meteorological variables for PV output power estimation. Renewable Energy 2020; 153: 12-22, DOI: https://doi.org/10.1016/j.renene.2020.01.150
28. [28] Somu, N, Raman, GMR, Krithi, R. A hybrid model for building energy consumption forecasting using long short term memory networks. Applied Energy 2020; 261: 114131, DOI: https://doi.org/10.1016/j.apenergy.2019.114131
29. [29] Abbas, A, Chowdhury, B. A data-driven approach for providing frequency regulation with aggregated residential HVAC units. In: 2019 North American Power Symposium (NAPS); 13-15 October 2019: IEEE, Wichita, KS, USA: pp. 1-6.
30. [30] Dai, B, Ran, W, Kun, Z, Hao, J, Ping, W. A demand response scheme in smart grid with clustering of residential customers. In: 2019 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm); 21-23 October 2019: IEEE, Beijing, China: pp. 1-6.
31. [31] Huishi, L, Ma, J, Sun, R, Du, Y. A data-driven approach for targeting residential customers for energy efficiency programs. IEEE Transactions on Smart Grid 2019; 11: 1229-1238, DOI: https://doi.org/10.1109/TSG.2019.2933704
32. [32] Chi, S, Sanmukh, RK, Xiong, C, Kannan, R, Prasanna, VK. A cooperative multi-agent deep reinforcement learning framework for real-time residential load scheduling. In: International Conference on Internet of Things Design and Implementation (IoTDI’19); 15-18 April 2019: Association for Computing Machinery (ACM), Montreal, QC, Canada: pp. 59-69.
33. [33] Mammoli, A, Matthew, R, Victor, A, Manel, MR, Chien-fei, C, Joana, MA. A behavior-centered framework for real-time control and load-shedding using aggregated residential energy resources in distribution microgrids. Energy & Buildings 2019; 198: 275-290, DOI: https://doi.org/10.1016/j.enbuild.2019.06.021
34. [34] Awad, A, Peter, B, German, R. A house appliances-level co-simulation framework for smart grid applications. In: Puliafito A, Trivedi K (Eds.), Systems Modeling: Methodologies and Tools, Cham: Springer, 2018, pp. 303-317.
35. [35] Cao, HA, Tri, KW, Karl, A, Nuno, N. A collaborative framework for annotating energy datasets. In: 2015 IEEE International Conference on Big Data (Big Data); 29 October-1 November 2015: IEEE Computer Society, Santa Clara, CA, USA: pp. 2716-2725.
36. [36] Moreno, AFJ, David, LM, Morrow, DJ, Jesús, MdR, Aoife, MF. Load modelling and non-intrusive load monitoring to integrate distributed energy resources in low and medium voltage networks. Renewable Energy 2021; 179: 445-466, DOI: https://doi.org/10.1016/j.renene.2021.07.056
37. [37] Klemenjak, C, Stephen, M, Wilfried, E. Investigating the performance gap between testing on real and denoised aggregates in non-intrusive load monitoring. Energy Informatics 2021; 4: 1-15, DOI: https://doi.org/10.1186/s42162-021-00137-9
38. [38] Huan, C, Yue‑Hsien, W, Chun‑Hung, F. A convolutional autoencoder‑based approach with batch normalization for energy disaggregation. The Journal of Supercomputing 2021; 77: 2961-2978, DOI: https://doi.org/10.1007/s11227-020-03375-y
39. [39] Khurram, HI, Farhan, HM, Muhammad, A, Muhammad, AQ, Nawaz, AM, Chishti, AR. A critical review of state-of-the-art non-intrusive load monitoring datasets. Electric Power Systems Research 2021; 192: 106921, DOI: https://doi.org/10.1016/j.epsr.2020.106921
40. [40] Ifeoluwa, OO, Obaid, M, Zhang, J, Dyanand, SR. Solving the fair electric load shedding problem in developing countries. Autonomous Agents and Multi-Agent Systems 2020; 34: 1-35, DOI: https://doi.org/10.1007/s10458-019-09428-8
41. [41] Artem, S, Malof, MJ, Bohao, H, Bradbury, K. Estimating residential building energy consumption using overhead imagery. Applied Energy 2020; 280: 116018, DOI: https://doi.org/10.1016/j.apenergy.2020.116018
42. [42] Gopinath, R, Mukesh, K, Chandra, CPJ, Kota, S. Energy management using non-intrusive load monitoring techniques – State-of-the-art and future research directions. Sustainable Cities and Society 2020; 62: 102411, DOI: https://doi.org/10.1016/j.scs.2020.102411
43. [43] Radu-Casian, M, Hurtig, D, Charlie, O. End-to-end anytime solution for appliance recognition based on high-resolution current sensing with few-shot learning. Internet of Things 2020; 11: 100263, DOI: https://doi.org/10.1016/j.iot.2020.100263
44. [44] Langevin, A, Ghyslain, G, Cheriet, M. Crosstalk suppression in semi-intrusive load monitoring systems using Hall Effect sensors. IEEE Transactions on Smart Grid 2020; 11: 5019-5027, DOI: https://doi.org/10.1109/TSG.2020.3002668
45. [45] Himeur, Y, Abdullah, A, Faycal, B, Abbes, A. Building power consumption datasets: Survey, taxonomy and future directions. Energy & Buildings 2020; 227: 110404, DOI: https://doi.org/10.1016/j.enbuild.2020.110404
46. [46] Chinthaka, D, Stephen, M, Bajić, IV. Residential power forecasting using load identification and graph spectral clustering. IEEE Transactions on Circuits and Systems II: Express Briefs 2019; 66: 1900-1904, DOI: https://doi.org/10.1109/TCSII.2019.2891704
47. [47] Nalmpantis, C, Dimitris, V. Machine learning approaches for non-intrusive load monitoring: from qualitative to quantitative comparation. Artif Intell Rev 2019; 52: 217-243, DOI: https://doi.org/10.1007/s10462-018-9613-7
48. [48] Jana, H, Andreas, R. A study on the impact of data sampling rates on load signature event detection. Energy Inform 2019; 2: 1-12, DOI: https://doi.org/10.1186/s42162-019-0096-9
49. [49] Lorena, MR, Adriana, RCG. Competitive autoassociative neural networks for electrical appliance identification for non-Intrusive load monitoring. IEEE Access 2019; 7: 111746-111755, DOI: https://doi.org/10.1109/ACCESS.2019.2934019
50. [50] Rashid, H, Pushpendra, S, Stankovic, V, Lina, S. Can non-intrusive load monitoring be used for identifying an appliance’s anomalous behaviour?. Applied Energy 2019; 238: 796-805, DOI: https://doi.org/10.1016/j.apenergy.2019.01.061
51. [51] Christoph, K, Andreas, R, Pereira, L, Stephen, M, Mario, B, Elmenreich, W. Electricity consumption data sets: Pitfalls and opportunities. In: Proceedings of the 6th ACM International Conference on Systems of Energy-Efficient Buildings, Cities, and Transportation (BuildSys’19); 13-14 November 2019: Association for Computing Machinery (ACM), New York, NY, USA: pp. 159-162.
52. [52] Henriet, S, Umut, Ş, Fuentes, B, Gaël, R. A generative model for non-Intrusive load monitoring in commercial buildings. Energy & Buildings 2018; 177: 268-278, DOI: https://doi.org/10.1016/j.enbuild.2018.07.060
53. [53] Dinesh, C, Shirantha, W, Liyanage, Y, Mervyn, PBE, Roshan, IG, Ekanayake, J. Non-intrusive load monitoring under residential solar power influx. Applied Energy 2017; 205: 1068-1080, DOI: http://dx.doi.org/10.1016/j.apenergy.2017.08.094
54. [54] Saeed, SH, Kodjo, A, Kelouwani, S, Alben, C. Non-intrusive load monitoring through home energy management systems: A comprehensive review. Renewable and Sustainable Energy Reviews 2017; 79: 1266-1274, DOI: http://dx.doi.org/10.1016/j.rser.2017.05.096
55. [55] Bonfigli, R, Emanuele, P, Marco, F, Severini, M, Stefano, S, Francesco, P. Non-intrusive load monitoring by using active and reactive power in additive Factorial Hidden Markov Models. Applied Energy 2017; 208: 1590-1607, DOI: http://dx.doi.org/10.1016/j.apenergy.2017.08.203
56. [56] Antonio, G, Molina, R, Alejandro, M, Sergio, V, Carlos, A. Residential end-uses disaggregation and demand response evaluation using integral transforms. J. Mod. Power Syst. Clean Energy 2017; 5: 91-104, DOI: 10.1007/s40565-016-0258-8
57. [57] Chinthaka, D, Buddhika, WN, Indika, GR, Mervyn, PBE, Janaka, E, Janaka, VW. Residential appliance identification based on spectral information of low frequency smart meter measurements. IEEE Transactions on Smart Grid 2016; 7: 2781-2792, DOI: https://doi.org/10.1109/TSG.2015.2484258
58. [58] Pedro, B, Brito, A, Hyggo, A. A Technique to provide differential privacy for appliance usage in smart metering. Information Sciences 2016; 370-371: 355-367, DOI: http://dx.doi.org/10.1016/j.ins.2016.08.011
59. [59] Alaa, A, Felix, R, Andreas, R, Englert, F, Daniel, B, Doreen, B, Gottron, C, Ralf, S. SMARTENERGY.KOM: An intelligent system for energy saving in smart home. In: 39th Annual IEEE Conference on Local Computer Networks Workshops; 8-11 September 2014: IEEE, Edmonton, AB, Canada: pp. 685-692.
60. [60] Frank, E, Till, S, Sebastian, K, Andreas, R, Ralf, S. How to auto-configure your smart home? High-resolution power measurements to the rescue. In: Proceedings of the Fourth International Conference on Future Energy Systems (e-Energy’13); 21-24 May 2013: Association for Computing Machinery (ACM), New York, NY, USA: pp. 215-224.
61. [61] Andreas, R, Paul, B, Burgstahler, D, Matthias, H, Hristo, C, Werner, M, Ralf, S. On the accuracy of appliance identification based on distributed load metering data. In: 2012 Sustainable Internet and ICT for Sustainability (SustainIT); 4-5 October 2012: IEEE, Pisa, Italy: pp. 1-9.
62. [62] Andreas, R, Dominic, B, Manzil, Z, Ralf, S. Electric appliance classification based on distributed high resolution current sensing. In: 37th Annual IEEE Conference on Local Computer Networks - Workshops; 22-25 October 2012: IEEE, Clearwater, FL, USA: pp. 999-1005.
63. [63] Andreas, R, Dominic, B, Parag, SM, Manzil, Z, Ralf, S. SmartMeter.KOM: A low-cost wireless sensor for distributed power metering. In: 2011 IEEE 36th Conference on Local Computer Networks; 4-7 October 2011: IEEE, Bonn, Germany: pp. 1032-1039.