Comparison of Gradient Boosting Decision Tree Algorithms for CPU Performance

Year 2021, Volume: 37 Issue: 1, 157 - 168, 28.04.2021

Haithm Alshari , Abdulrazak Saleh , Alper Odabas

Abstract

Gradyan Artırıcı Karar Ağacı (GBDT) algoritmalarının regresyon ve sınıflandırma problemlerinin çüzümünde makine öğrenimindeki en iyi algoritmalar arasında olduğu kanıtlanmıştır. Kaggle gibi web sitelerinin düzenlediği birçok yarışmayı kazanması sebebiyle en popüler GBDT algoritması olan XGBoost son teknoloji performansa sahip tek GBDT algoritması değildir. LightGBM ve CatBoost gibi kimi zaman XGBoost'a göre daha fazla avantajları olan başka GBDT algoritmaları da vardır. Bu makale, en iyi üç gradyan artırıcı algoritmanın işlemci(CPU) performansını karşılaştırmayı amaçlamaktadır. Bunun için ilk olarak bu üç algoritmanın nasıl çalıştığını ve aralarındaki hiperparametre benzerliklerini açıklayacağız. Daha sonra performanslarını değerlendirmek için doğruluk, hız, güvenilirlik ve kullanım kolaylığı olarak dörde ayırdığımız performans kriterleri kullanacağız. Üç algoritmanın performansı beş sınıflandırma ve regresyon problemi ile test edilmiştir. Bulgularımız, LightGBM algoritmasının, dengeli bir doğruluk, hız, güvenilirlik ve kullanım kolaylığı kombinasyonuyla üçü arasında en iyi performansa sahip olduğunu, bunu histogram yöntemiyle XGBoost'un izlediğini ve CatBoost'un ise özellikle yavaş ve tutarsız performansla diğerlerinin gerisinde kaldığını göstermektedir.

Keywords

Decision Tree, Gradient Boosting, XGBoost, LightGBM, CatBoost

CPU Performansı için Gradyan Artırıcı Karar Ağacı Algoritmalarının Karşılaştırılması

Abstract

Gradient Boosting Decision Trees (GBDT) algorithms have been proven to be among the best algorithms in machine learning. XGBoost, the most popular GBDT algorithm, has won many competitions on websites like Kaggle. However, XGBoost is not the only GBDT algorithm with state-of-the-art performance. There are other GBDT algorithms that have more advantages than XGBoost and sometimes even more potent like LightGBM and CatBoost. This paper aims to compare the performance of CPU implementation of the top three gradient boosting algorithms. We start by explaining how the three algorithms work and the hyperparameters similarities between them. Then we use a variety of performance criteria to evaluate their performance. We divide the performance criteria into four: accuracy, speed, reliability, and ease of use. The performance of the three algorithms has been tested with five classification and regression problems. Our findings show that the LightGBM algorithm has the best performance of the three with a balanced combination of accuracy, speed, reliability, and ease of use, followed by XGBoost with the histogram method, and CatBoost came last with slow and inconsistent performance.

Keywords

Karar Ağacı, Gradyan Artırıcı, XGBoost, LightGBM, CatBoost

References

• [1] J. H. Friedman, “Stochastic gradient boosting”, Computational statistics & data analysis, vol 38, no. 4, 367-378, (2002).
• [2] J. H. Friedman, “Greedy function approximation: a gradient boosting machine”. Annals of statistics, 1189-1232, (2001).
• [3] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg et al., “Scikit-learn: Machine learning in python," Journal of machine learning research, vol. 12, no. Oct, 2825-2830, (2011).
• [4] A. Swalin. “Catboost vs. Lightgbm vs. Xgboost”. (2018), [Online]. Available: https://towardsdatascience.com/catboost-vs-light-gbm-vs-xgboost-5f93620723db. Access date: 19.03.2019.
• [5] X. Team. “Introduction to boosted trees”. (2019), [Online]. Available: https://xgboost. readthedocs.io/en/latest/tutorials/model.html. Access date: 19.03.2019.
• [6] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu, Lightgbm: A highly efficient gradient boosting decision tree," in Advances in Neural Information Processing Systems, , 3146-3154, (2017).
• [7] P. Langley and S. Sage, “Oblivious decision trees and abstract cases” Working notes of the AAAI-94 workshop on case-based reasoning. Seattle, WA, 113-117, (1994).
• [8] R. Kohavi and C.-H. Li, “Oblivious decision trees, graphs, and top-down pruning,” IJCAI. Citeseer, 1071-1079, (1995).
• [9] V. Ershov. “Catboost enables fast gradient boosting on decision trees using gpus”. (2018), [Online]. Available: https://devblogs.nvidia.com/catboost-fast-gradientboosting-decision-trees/. Access date: 13.04.2019.
• [10] C. Team. “How training is performed”. (2018), [Online]. Available: https://catboost.ai/docs/ concepts/algorithm-main-stages.html. Access date: 23.05.2019.
• [11] T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system”, Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. ACM, 785-794, (2016).
• [12] X. Team. “Xgboost parameters”. (2019) [Online]. Available: https://xgboost.readthedocs.io/ en/latest/parameter.html. Access date: 19.03.2019.
• [13] A. V. Dorogush, V. Ershov, and A. Gulin, “Catboost: gradient boosting with categorical features support”, arXiv preprint arXiv:1810.11363, 2018.
• [14] L. Team. “Advanced topics, missing value handle”. (2019), [Online]. Available: https://lightgbm. readthedocs.io/en/latest/Advanced-Topics.html#missing-value-handle . Access date: 11.04.2019.
• [15] A. V. Dorogush, “Catboost - the new generation of gradient boosting," PyData, (2018), [Online]. Available: https://www.youtube.com/watch?v=8o0e-r0B5xQ .Access date: 23.05.2019.
• [16] L. Team. “Advanced topics, categorical feature support” (2019) [Online]. Available: https://lightgbm. readthedocs.io/en/latest/Advanced-Topics.html#categorical-feature-support. Access date: 11.04.2019.
• [17] L. Prokhorenkova, G. Gusev, A. Vorobev, A. V. Dorogush, and A. Gulin, “Catboost: unbiased boosting with categorical features”, Advances in Neural Information Processing Systems, 2018, 6638-6648, (2018).
• [18] C. database. Adult data set. (1996) [Online]. Available: http://archive.ics.uci.edu/ml/datasets/Adult. Access date: 18.03.2019.
• [19] WHO. “Life expectancy” (2018) [Online]. Available: https://www.kaggle.com/kumarajarshi/life-expectancy-who. Access date: 18.03.2019.
• [20] L. The National Heart and B. I. (NHLBI). “Framingham heart study”, (2018). [Online]. Available: https://www.nhlbi.nih.gov/science/framingham-heartstudy-fhs. Access date: 18.03.2019.