Automated Sensor Data Validation and Correction with Long Short-Term Memory Recurring Neural Network Model

Year 2021, Volume: 8 Issue: 1, 43 - 57, 29.03.2021

Abraham Sudharson Ponraj , Vigneswaran T Christy Jackson J

Abstract

Proper irrigation planning by matching reference evapotranspiration (ETo) with active crop growth requirement leads to an improved water usage efficiency and thereby improving the crop yield. ETo is primarily influenced by the following weather parameters the air temperature, relative humidity, wind speed and solar radiation. To make the ETo estimation system fault tolerant it is important to validate the real time data from the weather station, since the sensors used in these weather stations are prone to error due to influence of various environmental factors. A Recurring Neural Network (RNN) based Data Validation and Correction (DVC) algorithm was proposed to identify the faulty data and to correct them. Long Short-Term Memory (LSTM) RNN model is used to forecast the weather data such as temperature, solar radiation, wind speed and relative humidity. It uses statistical significance test to identify faulty data and isolate them. Then the DVC approach corrects the faulty data by replacing them by LSTM forecasted data. The performance evaluation of this approach showed better forecasting ability when compared with Seasonal Autoregressive Integrated Moving Average (SARIMA) based DVC and thereby improving overall performance of the DVC approach.

Keywords

Data Validation, LSTM, SARIMA, Reference Evapotranspiration ETo, Weather Data

References

• Allen, R. G., Pereira, L. S., Raes, D., & Smith, M. (1998). Crop Evapotranspiration, guidelines for computing crop water requirements, FAO Irrigation and Drainage Paper No. 56. Food and Agriculture Organization of the United Nations, Rome, 300p.
• Box, G. E., Jenkins, G. M., & Reinsel, G. C. (1994). Time Series Analysis, Forecasting and Control. Englewood Clifs, 598p.
• Box, G. E. P., Jenkins, G. M., Reinsel, G. C., & Ljung, G. M. (2015). Time series analysis: forecasting and control, 5th Edition. John Wiley & Sons, 712p.
• Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly detection: A survey. ACM Computing Surveys (CSUR), 41(3), Article No:15, 1-58.
• Chatfield, C. (2000). Time-Series Forecasting. CRC Press, 280p.
• Chauhan, S., & Vig, L. (2015). Anomaly detection in ECG time signals via deep long short-term memory networks. In: Proceedings of the IEEE International Conference on Data Science and Advanced Analytics (DSAA 2015), 1-7.
• Cobaner, M. (2011). Evapotranspiration estimation by two different neuro-fuzzy inference systems. Journal of Hydrology, 398(3-4), 292-302.
• Fischer, G., Nachtergaele, F. O., Prieler, S., Teixeira, E., Tóth, G., van Velthuizen, H., Verelst, L., & Wiberg, D. (2012). Global Agro-ecological Zones (GAEZ v3.0) - Model Documentation. (IIASA: Laxenburg, Austria; FAO: Rome, Italy) pure.iiasa.ac.at/13290
• Gardner, Jr, E. S. (2006). Exponential smoothing: The state of the art-Part II. International Journal of Forecasting, 22(4), 637-666.
• Goh, J., Adepu, S., Tan, M., & Lee, Z. S. (2017). Anomaly Detection in Cyber Physical Systems Using Recurrent Neural Networks In: Proceedings of the 18th International Symposium on High Assurance Systems Engineering (HASE 2017), 140-145.
• Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780.
• Husein, M., & Chung, I. Y. (2019). Day-ahead solar irradiance forecasting for microgrids using a long short-term memory recurrent neural network: A deep learning approach. Energies, 12(10), 1856, 1-22.
• Kim, S., Hong, S., Joh, M., & Song, S. K. (2017). DeepRain: ConvLSTM Network for Precipitation Prediction using Multichannel Radar Data. In: Proceedings of the 7th International Workshop on Climate Informatics. arxiv.org/abs/1711.02316
• Lee, S., & Kim, H. K. (2018). ADSaS: Comprehensive Real-time Anomaly Detection System. 19th World International Conference on Information Security and Application (WISA 2018), Revised Selected Papers, 29-41, Springer.
• Malhotra, P., Vig, L., Shroff, G., & Agarwal, P. (2015). Long Short Term Memory Networks for Anomaly Detection in Time Series. In: Proceedings of the European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning. (ESANN 2015), vol. 89, 89-94.
• Martin, D. L., & Gilley, J. (1993). Irrigation Water Requirements, Chapter 2, Part 623 of the National Engineering Handbook, United States Department of Agriculture - Soil Conservation Service.
• Mondal, P., Shit, L., & Goswami, S. (2014). Study of effectiveness of time series modeling (ARIMA) in forecasting stock prices. International Journal of Computer Science, Engineering and Applications, 4(2), 13-29.
• O'Reilly, C., Gluhak, A., Imran, M. A., & Rajasegarar, S. (2014). Anomaly detection in wireless sensor networks in a non-stationary environment. IEEE Communications Surveys & Tutorials, 16(3), 1413-1432.
• OECD (2008). Environmental outlook to 2030, The Organisation for Economic Co-operation and Development, 1-10.
• Ponraj, A. S., & Vigneswaran, T. (2019). Daily evapotranspiration prediction using gradient boost regression model for irrigation planning. The Journal of Supercomputing, 76, 5732-5744.
• Roesch, I., & Günther, T. (2019). Visualization of Neural Network Predictions for Weather Forecasting. Computer Graphics Forum, 38(1), 209-220.
• Salman, A. G., Kanigoro, B., & Heryadi, Y. (2015). Weather forecasting using deep learning techniques. In: Proceedings of the International Conference on Advanced Computer Science and Information System (ICACSIS 2015), 281-285.
• Sedgwick, P. (2014). Understanding statistical hypothesis testing. BMJ, 348.
• Sharma, A. B., Golubchik, L., & Govindan, R. (2010). Sensor faults: Detection methods and prevalence in real-world datasets. ACM Transactions on Sensor Networks (TOSN), 6(3), Article No:23, 1-39.
• Simmons, L. F. (1986). M-competition-A closer look at NAIVE2 and median APE: A note. International Journal of Forecasting, 2(4), 457-460.
• Thiyagarajan, K., Kodagoda, S., Van Nguyen, L., & Ranasinghe, R. (2018). Sensor failure detection and faulty data accommodation approach for instrumented wastewater infrastructures. IEEE Access, 6, 56562-56574.
• Tiwari, D., & Dinar, A. (2002). Balancing future food demand and water supply: The role of economic incentives in irrigated agriculture. Quarterly, Journal of International Agriculture, 41(1), 77-97.
• Tomar, A. S. & Ranade, D. H. (2001). Pan coefficient determination for evapotranspiration at Indore, Madhya Pradesh. Indian J. Soil Conserv., 29, 173-175.
• Watson, I., & Burnett A. D. (2017). Hydrology: An Environmental Approach, Routledge, 722p.
• Venugopal, P., & Vigneswaran, T. (2019). State-of-Health Estimation of Li-ion Batteries in Electric Vehicle Using IndRNN under Variable Load Condition. Energies, 12(22), 4338, 1-29.