On the Prediction of Chaotic Time Series using Neural Networks

Abstract

Prediction techniques have the challenge of guaranteeing large horizons for chaotic time series. For instance, this paper shows that the majority of techniques can predict one step ahead with relatively low root-mean-square error (RMSE) and Symmetric Mean Absolute Percentage Error (SMAPE). However, some techniques based on neural networks can predict more steps with similar RMSE and SMAPE values. In this manner, this work provides a summary of prediction techniques, including the type of chaotic time series, predicted steps ahead, and the prediction error. Among those techniques, the echo state network (ESN), long short-term memory, artificial neural network and convolutional neural network are compared with similar conditions to predict up to ten steps ahead of Lorenz-chaotic time series. The comparison among these prediction techniques include RMSE and SMAPE values, training and testing times, and required memory in each case. Finally, considering RMSE and SMAPE, with relatively few neurons in the reservoir, the performance comparison shows that an ESN is a good technique to predict five to fifteen steps ahead using thirty neurons and taking the lowest time for the tracking and testing cases.

Keywords

• Chaotic time series
• Neural network
• Echo state network
• Long short-term memory
• RMSE
• Prediction technique

References

1. Alemu, M. N., 2018 A fuzzy model for chaotic time series prediction. International Journal of Innovative Computing Information and Control 14: 1767–1786.
2. Ardalani-Farsa, M. and S. Zolfaghari, 2010 Chaotic time series prediction with residual analysis method using hybrid elmannarx neural networks. Neurocomputing 73: 2540–2553.
3. Chandra, R., Y.-S. Ong, and C.-K. Goh, 2017 Co-evolutionary multitask learning with predictive recurrence for multi-step chaotic time series prediction. Neurocomputing 243: 21–34.
4. Chandra, R. and M. Zhang, 2012 Cooperative coevolution of elman recurrent neural networks for chaotic time series prediction. Neurocomputing 86: 116–123.
5. Chen, D. and W. Han, 2013 Prediction of multivariate chaotic time series via radial basis function neural network. Complexity 18: 55–66.
6. Chen, H.-C. and D.-Q. Wei, 2021 Chaotic time series prediction using echo state network based on selective opposition grey wolf optimizer. Nonlinear Dynamics 104: 3925–3935.
7. Cheng, W., Y. Wang, Z. Peng, X. Ren, Y. Shuai, et al., 2021 Highefficiency chaotic time series prediction based on time convolution neural network. Chaos Solitons & Fractals 152.
8. Dalia Pano-Azucena, A., E. Tlelo-Cuautle, S. X. D. Tan, B. Ovilla- Martinez, and L. Gerardo de la Fraga, 2018 Fpga-based implementation of a multilayer perceptron suitable for chaotic time series prediction. Technologies 6.

9. Dhanya, C. and D. Nagesh Kumar, 2010 Nonlinear ensemble prediction of chaotic daily rainfall. Advances in Water Resources 33: 327–347.
10. Dorigo, M., M. Birattari, and T. Stutzle, 2006 Ant colony optimization. IEEE computational intelligence magazine 1: 28–39. Drew, P. J. and J. R. Monson, 2000 Artificial neural networks. Surgery 127: 3–11.
11. Eberhart, R. and J. Kennedy, 1995 Particle swarm optimization. In Proceedings of the IEEE international conference on neural networks, volume 4, pp. 1942–1948, Citeseer.
12. Feng, S., W. Ren, M. Han, and Y. W. Chen, 2019a Robust manifold broad learning system for large-scale noisy chaotic time series prediction: A perturbation perspective. Neural Networks 117: 179–190.
13. Feng, T., S. Yang, and F. Han, 2019b Chaotic time series prediction using wavelet transform and multi-model hybrid method. Journal of Vibroengineering 21: 1983–1999.
14. Fu, Y.-Y., C.-J.Wu, J.-T. Jeng, and C.-N. Ko, 2010 Arfnns with svr for prediction of chaotic time series with outliers. Expert Systems with Applications 37: 4441–4451.
15. Fukushima, K., 1980 Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics 36: 193–202.
16. Ganjefar, S. and M. Tofighi, 2018 Optimization of quantuminspired neural network using memetic algorithm for function approximation and chaotic time series prediction. Neurocomputing 291: 175–186.
17. Gholizade-Narm, H. and M. R. Shafiee-Chafi, 2015 Using repetitive fuzzy method for chaotic time series prediction. Journal Of Intellıgent & Fuzzy Systems 28: 1937–1946.
18. Goudarzi, S., M. B. Khodabakhshi, and M. H. Moradi, 2016 Interactively recurrent fuzzy functions with multi objective learning and its application to chaotic time series prediction. Journal Of Intellıgent & Fuzzy Systems 30: 1157–1168.
19. Gromov, V. A. and E. A. Borisenko, 2015 Predictive clustering on non-successive observations for multi-step ahead chaotic time series prediction. Neural Computing & Applications 26: 1827– 1838.
20. Gromov, V. A. and A. N. Shulga, 2012 Chaotic time series prediction with employment of ant colony optimization. Expert Systems With Applıcations 39: 8474–8478.
21. Guo, F., L. Lin, and C. Wang, 2016a Novel continuous function prediction model using an improved takagi-sugeno fuzzy rule and its application based on chaotic time series. Engıneering Applications Of Artificial Intelligence 55: 155–164.
22. Guo, W., T. Xu, and Z. Lu, 2016b An integrated chaotic time series prediction model based on efficient extreme learning machine and differential evolution. Neural Computing & Applications 27: 883–898.
23. Guo, X., Y. Sun, and J. Ren, 2020 Low dimensional mid-term chaotic time series prediction by delay parameterized method. Information Sciences 516: 1–19.
24. Han, F., S. Yang, and S. Song, 2018 Local volterra multivariable chaotic time series multi-step prediction based on phase points clustering. Journal of Vibroengineering 20: 2486–2503.
25. Han, M., S. Feng, C. L. P. Chen, M. Xu, and T. Qiu, 2019a Structured manifold broad learning system: A manifold perspective for large-scale chaotic time series analysis and prediction. IEEE Transactıons On Knowledge And Data Engıneerıng 31: 1809– 1821.
26. Han, M., W. Li, S. Feng, T. Qiu, and C. L. P. Chen, 2021 Maximum information exploitation using broad learning system for large-scale chaotic time-series prediction. IEEE Transactions On Neural Networks And Learning Systems 32: 2320–2329.
27. Han, M., R. Zhang, and M. Xu, 2017 Multivariate chaotic time series prediction based on elm-plsr and hybrid variable selection algorithm. Neural Processing Letters 46: 705–717.
28. Han, M., S. Zhang, M. Xu, T. Qiu, and N. Wang, 2019b Multivariate chaotic time series online prediction based on improved kernel recursive least squares algorithm. IEEE Transactions On Cybernetics 49: 1160–1172.
29. Han, M., K. Zhong, T. Qiu, and B. Han, 2019c Interval type-2 fuzzy neural networks for chaotic time series prediction: A concise overview. IEEE Transactions on Cybernetics 49: 2720–2731.
30. Heydari, G., M. Vali, and A. A. Gharaveisi, 2016 Chaotic time series prediction via artificial neural square fuzzy inference system. Expert Systems with Applications 55: 461–468.
31. Hochreiter, S. and J. Schmidhuber, 1997 Long Short-Term Memory. Neural Computation 9: 1735–1780.
32. Hua, Q., M. Wen-Tao, Z. Ji-Hong, and C. Ba-Dong, 2013 Kernel least mean kurtosis based online chaotic time series prediction. Chinese Physics Letters 30.
33. Huang, W., Y. Li, and Y. Huang, 2020 Deep hybrid neural network and improved differential neuroevolution for chaotic time series prediction. IEEE Access 8: 159552–159565.
34. Jaeger, H., 2007 Echo state network. scholarpedia 2: 2330.
35. Jian-Ling, Q., W. Xiao-Fei, Q. Yu-Chuan, G. Feng, and D. Ya-Zhou, 2014 An improved local weighted linear prediction model for chaotic time series. Chinese Physics Letters 31.
36. Jianshan, L., W. Changming, Z. Aijun, and X. Xiaomin, 2012 Residual gm(1,1) model-based prediction method for chaotic time series. Journal of Grey System 24: 379–388.
37. Jingjing, L., Z. Qijin, Z. Yumei, W. Xiaojun, W. Xiaoming, et al., 2018 Hidden phase space reconstruction: A novel chaotic time series prediction method for speech signals. Chinese Journal of Electronics 27: 1221–1228.
38. Jokar, M., H. Salarieh, and A. Alasty, 2019 On the existence of proper stochastic markov models for statistical reconstruction and prediction of chaotic time series. Chaos Solitons & Fractals 123: 373–382.
39. Kurogi, S., M. Toidani, R. Shigematsu, and K. Matsuo, 2018 Performance improvement via bagging in probabilistic prediction of chaotic time series using similarity of attractors and loocv predictable horizon. Neural Computing & Applications 29: 341– 349.
40. Lau, K. W. and Q. H. Wu, 2008 Local prediction of non-linear time series using support vector regression. Pattern Recogn. 41: 1539–1547.
41. Li, D., M. Han, and J. Wang, 2012 Chaotic time series prediction based on a novel robust echo state network. IEEE Transactions On Neural Networks and Learning Systems 23: 787–799.
42. Li, Q. and R.-C. Lin, 2016 A new approach for chaotic time series prediction using recurrent neural network. Mathematical Problems in Engineering 2016.
43. Li, T.-Y. and J. A. Yorke, 1975 Period three implies chaos. The American Mathematical Monthly 82: 985–992.
44. Li, Y., X. Jiang, H. Zhu, X. He, S. Peeta, et al., 2016 Multiple measures-based chaotic time series for traffic flow prediction based on bayesian theory. Nonlinear Dynamics 85: 179–194.
45. Li-yun, S., 2010 Prediction of multivariate chaotic time series with local polynomial fitting. Computers & Mathematiıcs with Applications 59: 737–744.
46. Liu, Z., 2010 Chaotic time series analysis. Mathematical Problems in Engineering 2010: 720190.
47. Lorenz, E. N., 1963 Deterministic nonperiodic flow. Journal of Atmospheric Sciences 20: 130 – 141.
48. Lukoševiˇcius, M., 2012 A practical guide to applying echo state networks. In Neural Networks: Tricks of the Trade: Second Edition, edited by G. Montavon, G. B. Orr, and K.-R. Müller, pp. 659–686, Springer Berlin Heidelberg, Berlin, Heidelberg.
49. Lv, M., X. Zhang, H. Chen, C. Ling, and J. Li, 2020 An accurate online prediction model for kiln head temperature chaotic time series. IEEE Access 8: 44288–44299.
50. Ma,W., J. Duan,W. Man, H. Zhao, and B. Chen, 2017 Robust kernel adaptive filters based on mean p-power error for noisy chaotic time series prediction. Engıneerıng Applıcatıons Of Artıfıcıal Intellıgence 58: 101–110.
51. Mackey, M. C. and L. Glass, 1977 Oscillation and chaos in physiological control systems. Science 197: 287–289.
52. McCulloch, W. S. and W. Pitts, 1943 A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics 5: 115–133.
53. Miranian, A. and M. Abdollahzade, 2013 Developing a local leastsquares support vector machines-based neuro-fuzzy model for nonlinear and chaotic time series prediction. IEEE Transactions on Neural Networks and Learning Systems 24: 207–218.
54. Mirjalili, S. and A. Lewis, 2016 The whale optimization algorithm. Advances in engineering software 95: 51–67.
55. Mirjalili, S., S. M. Mirjalili, and A. Lewis, 2014 Grey wolf optimizer. Advances in engineering software 69: 46–61.
56. Na, X.,W. Ren, and X. Xu, 2021 Hierarchical delay-memory echo state network: A model designed for multi-step chaotic time series prediction. Engıneerıng Applıcatıons Of Artıfıcıal Intellıgence 102.
57. Nguyen, H. M., G. Kalra, T. Jun, and D. Kim, 2020 Chaotic time series prediction using a novel echo state network model with input reconstruction, bayesian ridge regression and independent component analysis. Internatıonal Journal Of Pattern Recognıtıon And Artıfıcıal Intellıgence 34.
58. Ong, P. and Z. Zainuddin, 2019 Optimizing wavelet neural networks using modified cuckoo search for multi-step ahead chaotic time series prediction. Applied Soft Computing 80: 374–386.
59. Pano-Azucena, A. D., E. Tlelo-Cuautle, B. Ovilla-Martinez, L. G. de la Fraga, and R. Li, 2021 Pipeline fpga-based implementations of anns for the prediction of up to 600-steps-ahead of chaotic time series. Journal Of Cırcuıts Systems And Computers 30.
60. Price, K., R. M. Storn, and J. A. Lampinen, 2006 Differential evolution: a practical approach to global optimization. Springer Science & Business Media.
61. Samanta, B., 2011 Prediction of chaotic time series using computational intelligence. Expert Systems With Applications 38: 11406–11411.
62. Shi, Y., X. Liu, T. Li, X. Peng,W. Chen, et al., 2010 Chaotic time series prediction using immune optimization theory. International Journal Of Computational Intelligence Systems 3: 43–60.
63. Shinozaki, A., T. Miyano, and Y. Horio, 2020 Chaotic time series prediction by noisy echo state network. IEICE Nonlinear Theory and its Applications 11: 466–479.
64. Shoaib, B., I. M. Qureshi, Shafqatullah, and Ihsanulhaq, 2014 Adaptive step-size modified fractional least mean square algorithm for chaotic time series prediction. Chinese Physics B 23.
65. Su, L. and C. Li, 2015a Local functional coefficient autoregressive model for multistep prediction of chaotic time series. Discrete Dynamics In Nature and Society 2015.
66. Su, L. and C. Li, 2015b Local prediction of chaotic time series based on polynomial coefficient autoregressive model. Mathematical Problems in Engineering 2015.
67. Su, L. and F. Yang, 2021 Prediction of chaotic time series based on ben-aga model. Complexity 2021.
68. Suykens, J. A. and J. Vandewalle, 1999 Least squares support vector machine classifiers. Neural processing letters 9: 293–300.
69. Swinburne, R., 2004 Bayes’ theorem. Revue Philosophique de la France Et de l 194.
70. Tang, L.-H., Y.-L. Bai, J. Yang, and Y.-N. Lu, 2020 A hybrid prediction method based on empirical mode decomposition and multiple model fusion for chaotic time series. Chaos Solitons & Fractals 141.
71. Wang, C., H. Zhang,W. Fan, and P. Ma, 2017 A new chaotic time series hybrid prediction method of wind power based on eemdse and full-parameters continued fraction. Energy 138: 977–990.
72. Wang, H. and J. Lian, 2011 Fuzzy predıctıon of chaotıc tıme serıes based on fuzzy clusterıng. Asian Journal Of Control 13: 576–581.
73. Wang, H., F. Sun, Y. Cai, and Z. Zhao, 2010 Online chaotic time series prediction using unbiased composite kernel machine via cholesky factorization. Soft Computing 14: 931–944.
74. Wang, R., C. Peng, J. Gao, Z. Gao, and H. Jiang, 2020a A dilated convolution network-based lstm model for multi-step prediction of chaotic time series. Computational & Applied Mathematics 39.
75. Wang, Y., Z. Man, and M. Lu, 2020b Prediction of energy-efficient production of coalbed methane based on chaotic time series and bayes-least squares-support vector machine. Internatıonal Journal Of Heat And Technology 38: 933–940.
76. Wolf, A., J. B. Swift, H. L. Swinney, and J. A. Vastano, 1985 Determining lyapunov exponents from a time series. Physica D: nonlinear phenomena 16: 285–317.
77. Wu, X. and Z. Song, 2013 Multi-step prediction of chaotic timeseries with intermittent failures based on the generalized nonlinear filtering methods. Applıed Mathematıcs And Computatıon 219: 8584–8594.
78. Xiao, Y., X. Xie, Q. Li, and T. Li, 2019 Nonlinear dynamics model for social popularity prediction based on multivariate chaotic time series. Physıca A-Statıstıcal Mechanıcs And Its Applıcatıons 525: 1259–1275.
79. Xin, B. andW. Peng, 2020 Prediction for chaotic time series-based ae-cnn and transfer learning. Complexity 2020.
80. Xing, B. and W.-J. Gao, 2014 Fruit fly optimization algorithm. In Innovative Computational Intelligence: A Rough Guide to 134 Clever Algorithms, pp. 167–170, Springer.
81. Xu, M., M. Han, T. Qiu, and H. Lin, 2019 Hybrid regularized echo state network for multivariate chaotic time series prediction. IEEE Transactions on Cybernetics 49: 2305–2315.
82. Yang, H., H. Ye, G.Wang, and T. Hu, 2005 Fuzzy neural very-shortterm load forecasting based on chaotic dynamics reconstruction. In Advances in Neural Networks – ISNN 2005, edited by J.Wang, X.-F. Liao, and Z. Yi, pp. 622–627, Berlin, Heidelberg, Springer Berlin Heidelberg.
83. Yang, L., J. Zhang, X.Wu, Y. Zhang, and J. Li, 2016 A chaotic time series prediction model for speech signal encoding based on genetic programming. Applied Soft Computing 38: 754–761.
84. Yang, X.-S. and S. Deb, 2010 Engineering optimisation by cuckoo search. International Journal of Mathematical Modelling and Numerical Optimisation 1: 330–343.
85. Yong, Z., 2013 New prediction of chaotic time series based on local lyapunov exponent. Chinese Physics B 22.
86. Yumei, Z., B. Shulin, L. Gang, and W. Xiaojun, 2019 Kernel estimation of truncated volterra filter model based on dfp technique and its application to chaotic time series prediction. Chınese Journal of Electronıcs 28: 127–135.
87. Zhang, D., 2019a Wavelet transform. In Fundamentals of Image Data Mining, pp. 35–44, Springer.
88. Zhang, D. and M. Jiang, 2020 Hetero-dimensional multitask neuroevolution for chaotic time series prediction. IEEE Access 8: 123135–123150.
89. Zhang, L., 2019b Evaluating the effects of size and precision of training data on ann training performance for the prediction of chaotic time series patterns. Internatıonal Journal Of Software Scıence And Computatıonal Intellıgence-IJSSCI 11: 16–30.
90. Zhang, L., F. Tian, S. Liu, L. Dang, X. Peng, et al., 2013 Chaotic time series prediction of e-nose sensor drift in embedded phase space. Sensors And Actuators B-Chemıcal 182: 71–79.
91. Zhang, M., B. Wang, Y. Zhou, and H. Sun, 2020 Woa-based echo state network for chaotic time series prediction. Journal of the Korean Physıcal Society 76: 384–391.
92. Zhao, C., W. Ren, and M. Han, 2021 Adaptive sparse quantization kernel least mean square algorithm for online prediction of chaotic time series. Cırcuıts Systems And Sıgnal Processıng 40: 4346–4369.
93. Zhao, J., Y. Li, X. Yu, and X. Zhang, 2014 Levenberg-marquardt algorithm for mackey-glass chaotic time series prediction. Discrete Dynamics In Nature and Society 2014.
94. Zheng, Y., S.Wang, J. Feng, and C. K. Tse, 2016 A modified quantized kernel least mean square algorithm for prediction of chaotic time series. Digital Signal Processing 48: 130–136.
95. Zhongda, T., L. Shujiang, W. Yanhong, and S. Yi, 2017 A prediction method based on wavelet transform and multiple models fusion for chaotic time series. Chaos Solitons & Fractals 98: 158–172.
96. Zhou, Y.-T., Y. Fan, Z.-Y. Chen, and J.-C. Sun, 2017 Multimodality prediction of chaotic time series with sparse hard-cut em learning of the gaussian process mixture model. Chınese Physıcs Letters 34.