Comparing and Combining MLP and NEAT for Time Series Forecasting

Abstract

Neural networks are one of the widely-used time series forecasting methods in time series applications. Among different neural network architectures and learning algorithms, the most popular choice is the feedforward Multilayer Perceptron (MLP). However, it suffers from some drawbacks such as getting trapped in local minima, human intervention during the stage of training, and limitations in architecture design. The aims of this study were twofold. The first was to employ NeuroEvolution of Augmenting Topologies (NEAT), which has many successful applications in numerous fields. In this paper, we applied it to time series forecasting for the first time and compared its performance with that of the MLP. The second aim was to analyse the performance resulting from the pairwise combination of these methods. In general, the results suggested that the forecasts from the NEAT algorithm were more accurate than those of the MLP. The results also showed that pairwise combined forecasts in general were better than single forecasts. The best forecasts of all were obtained by pairwise combination of MLP and NEAT.

Keywords

• Forecasting,Time Series,NEAT,MLP,Combined Forecast,Neural Networks

References

1. Angeline, P. J., Saunders, G. M., & Pollack, J. B. (1994). An evolutionary algorithm that constructs recurrent neural networks. IEEE transactions on Neural Networks, 5(1), 54-65.
2. Aras, S., & Kocakoç, İ. D. (2016). A new model selection strategy in time series forecasting with artificial neural networks: IHTS. Neurocomputing, 174, 974-987.
3. Armstrong, J. S. (1989). Combining forecasts: The end of the beginning or the beginning of the end? International Journal of Forecasting, 5(4), 585-588.
4. Armstrong, J. S. (Ed.). (2001). “Combining forecasts”, Chapter 13 in Principles of forecasting: a handbook for researchers and practitioners (Vol. 30). Springer Science & Business Media.
5. Armstrong, J. S., & Fildes, R. (1995). Correspondence on the selection of error measures for comparisons among forecasting methods. Journal of Forecasting, 14(1), 67-71.
6. Bates, J. M., & Granger, C. W. (1969). The combination of forecasts. Journal of the Operational Research Society, 20(4), 451-468.
7. Clemen, R. T. (1989). Combining forecasts: A review and annotated bibliography. International Journal of Forecasting, 5(4), 559-583.
8. Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4), 303-314.

9. De Gooijer, J. G., & Kumar, K. (1992). Some recent developments in non-linear time series modelling, testing, and forecasting. International Journal of Forecasting, 8(2), 135-156.
10. De Groot, C., & Würtz, D. (1991). Analysis of univariate time series with connectionist nets: A case study of two classical examples. Neurocomputing, 3(4), 177-192.
11. De Menezes, L. M., Bunn, D. W., & Taylor, J. W. (2000). Review of guidelines for the use of combined forecasts. European Journal of Operational Research, 120(1), 190-204.
12. Enders, W. (2008). Applied econometric time series. John Wiley & Sons.
13. Floreano, D., Dürr, P., & Mattiussi, C. (2008). Neuroevolution: from architectures to learning. Evolutionary Intelligence, 1(1), 47-62.
14. Ghiassi, M., & Saidane, H. (2005). A dynamic architecture for artificial neural networks. Neurocomputing, 63, 397-413.
15. Ginzburg, I., & Horn, D. (1994). Combined neural networks for time series analysis. Advances in Neural Information Processing Systems, 224-224.
16. Granger, C. W. (1989). Invited review combining forecasts—twenty years later. Journal of Forecasting, 8(3), 167-173.
17. Gruau, F., Whitley, D., & Pyeatt, L. (1996, July). A comparison between cellular encoding and direct encoding for genetic neural networks. In Proceedings of the 1st annual conference on genetic programming (pp. 81-89). MIT Press.
18. Hagan, M. T., Demuth, H. B., Beale, M. H., & De Jesús, O. (1996). Neural network design (Vol. 20). Boston: PWS publishing company. Haykin, S. S. (2001). Neural networks: a comprehensive foundation. Tsinghua University Press.
19. Hornik, K., Stinchcombe, M., & White, H. (1989). Multilayer feedforward networks are universal approximators. Neural Networks, 2(5), 359-366.
20. Hyndman, R. J. (2012). Time series data library. www.robjhyndman.com/TSDL/. Accessed on 12 June 2012.
21. Islam, M. M., Yao, X., & Murase, K. (2003). A constructive algorithm for training cooperative neural network ensembles. IEEE Transactions on neural networks, 14(4), 820-834.
22. Kaastra, I., & Boyd, M. (1996). Designing a neural network for forecasting financial and economic time series. Neurocomputing, 10(3), 215-236.
23. Khashei, M., & Bijari, M. (2010). An artificial neural network (p, d, q) model for timeseries forecasting. Expert Systems with applications, 37(1), 479-489.
24. Keppel, G. (1973). Design and Analysis: A Researcher’s Handbook. Prentice-Hall, New Jersey.
25. Kodogiannis, V., & Lolis, A. (2002). Forecasting financial time series using neural network and fuzzy system-based techniques. Neural Computing & Applications, 11(2), 90-102.
26. Lachtermacher, G., & Fuller, J. D. (1995). Back propagation in time-series forecasting. Journal of Forecasting, 14(4), 381-393.
27. Leung, F. H. F., Lam, H. K., Ling, S. H., & Tam, P. K. S. (2003). Tuning of the structure and parameters of a neural network using an improved genetic algorithm. IEEE Transactions on Neural Networks, 14(1), 79-88.
28. Luxhøj, J. T., Riis, J. O., & Stensballe, B. (1996). A hybrid econometric—neural network modeling approach for sales forecasting. International Journal of Production Economics, 43(2), 175-192.
29. Makridakis, S., Andersen, A., Carbone, R., Fildes, R., Hibon, M., Lewandowski, R., & Winkler, R. (1982). The accuracy of extrapolation (time series) methods: Results of a forecasting competition. Journal of Forecasting, 1(2), 111-153.
30. Maniezzo, V. (1994). Genetic evolution of the topology and weight distribution of neural networks. IEEE Transactions on Neural Networks, 5(1), 39-53.
31. Newbold, P., & Granger, C. W. (1974). Experience with forecasting univariate time series and the combination of forecasts. Journal of the Royal Statistical Society. Series A (General), 131-165.
32. Newbold, P., Carlson, W., & Thorne, B. (2009). Statistics for business and economics. Prentice Hall.
33. Nikolopoulos, K., Goodwin, P., Patelis, A., & Assimakopoulos, V. (2007). Forecasting with cue information: A comparison of multiple regression with alternative forecasting approaches. European Journal Of Operational Research, 180(1), 354-368.
34. Palm, F. C., & Zellner, A. (1992). To combine or not to combine? Issues of combining forecasts. Journal of Forecasting, 11(8), 687-701.
35. Palmes, P. P., Hayasaka, T., & Usui, S. (2005). Mutation-based genetic neural network. IEEE Transactions on Neural Networks, 16(3), 587-600.
36. García-Pedrajas, N., Hervás-Martínez, C., & Muñoz-Pérez, J. (2003). COVNET: a cooperative coevolutionary model for evolving artificial neural networks. IEEE Transactions on Neural Networks, 14(3), 575-596.
37. Perrone, M.P. & Cooper, L.N. (1993). When networks disagree: ensemble method for hybrid neural networks. in: R.J. Mammone (Ed.), Neural Networks for Speechand Image Processing, Chapman & Hall, London, pp.126-142.
38. Priestly, M.B. (1988). Non-linear and Non-Stationary Time Series Analysis. Academic Press.
39. Reid, D. J. (1968). Combining three estimates of gross domestic product. Economica, 35(140), 431-444.
40. Reisinger, J., Bahceci, E., Karpov, I., & Miikkulainen, R. (2007, April). Coevolving strategies for general game playing. In 2007 IEEE Symposium on Computational Intelligence and Games (pp. 320-327).
41. Satterthwaite, F. E. (1946). An approximate distribution of estimates of variance components. Biometrics Bulletin, 2(6), 110-114.
42. Stanley, K. O., & Miikkulainen, R. (2002). Evolving neural networks through augmenting topologies. Evolutionary Computation, 10(2), 99-127.
43. Stanley, K. O., & Miikkulainen, R. (2004). Competitive coevolution through evolutionary complexification. J Artif Intell Res (JAIR), 21, 63-100.
44. Stanley, K., Kohl, N., Sherony, R., & Miikkulainen, R. (2005, June). Neuroevolution of an automobile crash warning system. In Proceedings of the 7th annual conference on Genetic and evolutionary computation (pp. 1977-1984).
45. Stock, J.H. & Watson, M. (1999). A Comparison of Linear and Nonlinear Uni-variate Models for Forecasting Macroeconomic Time Series. in Robert F. Engle and Halbert White eds., Cointegration, Causality, and Forecasting: A Festschrift in Honor of Clive W. J. Granger, Oxford University Press, Oxford, U.K., pp.1-44.
46. Stone, L., & He, D. (2007). Chaotic oscillations and cycles in multi-trophic ecological systems. Journal of Theoretical Biology, 248(2), 382-390.
47. Tseng, F. M., Yu, H. C., & Tzeng, G. H. (2002). Combining neural network model with seasonal time series ARIMA model. Technological Forecasting and Social Change, 69(1), 71-87.
48. Wedding, D. K., & Cios, K. J. (1996). Time series forecasting by combining RBF networks, certainty factors, and the Box-Jenkins model. Neurocomputing, 10(2), 149-168.
49. Welch, B. L. (1947). The generalization ofstudent’s’ problem when several different population variances are involved. Biometrika, 34(1/2), 28-35.
50. Wong, C. S., & Li, W. K. (2000). On a mixture autoregressive model. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 62(1), 95-115.
51. Yao, X. (1999). Evolving artificial neural networks. Proceedings of the IEEE, 87(9), 1423-1447.
52. Yao, X., & Liu, Y. (1997). A new evolutionary system for evolving artificial neural networks. IEEE Transactions on Neural Networks, 8(3), 694-713.
53. Zhang, G., Patuwo, B. E., & Hu, M. Y. (1998). Forecasting with artificial neural networks:: The state of the art. InternationalJjournal of Forecasting, 14(1), 35-62.
54. Zhang, G. P. (2001). An investigation of neural networks for linear time-series forecasting. Computers & Operations Research, 28(12), 1183-1202.
55. Zhang, G. P. (2003). Time series forecasting using a hybrid ARIMA and neural network model. Neurocomputing, 50, 159-175.
56. Zhang, G. P., Patuwo, B. E., & Hu, M. Y. (2001). A simulation study of artificial neural networks for nonlinear time-series forecasting. Computers & Operations Research, 28(4), 381-396.
57. Zou, H. F., Xia, G. P., Yang, F. T., & Wang, H. Y. (2007). An investigation and comparison of artificial neural network and time series models for Chinese food grain price forecasting. Neurocomputing, 70(16), 2913-2923.