HADOOP ÜZERİNDE ÖLÇEKLENEBİLİR BETİMLEYİCİ İSTATİSTİK UYGULAMALARI

Öz

Büyük Veri, İngilizce dilindeki karşılığı ile Big Data, çağımızın en güncel teknolojilerinden biri olarak karşımıza çıkmaktadır. Sosyal medya, sensör verileri, Nesnelerin İnternet’i gibi seri halde veri üreten teknolojilerin sayesinde veri hacmi gün geçtikçe artmaktadır. Dünyada veri miktarındaki büyük artış, büyük verinin depolanması, işlenmesi ve analiz edilmesi için farklı yaklaşımlar gerektirmektedir. Bir nicel veriseti birçok özelliğe sahiptir ve betimleyici istatistikler veri setindeki bu özellikleri her bir değeri listelemek zorunda kalmadan anlamlı ve yönetilebilir bir biçimde tanımlayabilir. Bununla birlikte, standart istatistiksel teknikler, verinin büyüklüğü, karmaşıklığı ve hızı nedeniyle büyük verilere uygun olmayabilir. Nicel verileri analiz etmek için kullanıma hazır çok sayıda istatistiksel araç olmasına rağmen, her zaman büyük veri dosya sistemleri ile çalışmak için uyumlu değildir. Bu yazıda, betimleyici istatistik algoritmalarının büyük veri setleri üzerindeki uygulamaları sergilenmektedir ve deneylerin 196 yivli küçük bir Hadoop kümesinde ölçeklenebilirliğini gösterilmektedir. Bu çalışma, büyük veri kümeleri için tanımlayıcı istatistiklerin bir Hadoop kümesinin dağıtılmış hesaplama özelliklerinden yararlanabileceğini göstermektedir. Çalışma TÜBİTAK TEYDEB desteği ile tamamlanmıştır.

Anahtar Kelimeler

• Büyük Veri,Betimleyici İstatistik,Hadoop,MapReduce

SCALABLE IMPLEMENTATIONS OF DESCRIPTIVE STATISTICS ON HADOOP

Öz

Big Data is one of the most trendy technologies of our time. The volume of data is increasing day by day, thanks to serial data generation technologies such as social media, sensor data, Internet of Things. The massive increase in the amount of data accumulated around the world requires different approaches to store, process and analyze the big data. A set of quantitative data has many features and the descriptive statistics can describe these features in a meaningful and manageable form without having to list every value in the dataset. However, the standard statistical techniques cannot suit big data due to the size, complexity and velocity of the data. Though there are many off-the-shelf statistical tools available to analyze quantitative data they are not always compatible with the big data file systems. In this paper, we describe our implementations of the descriptive statistics algorithms over big data and show the scalability of our experiments on a small Hadoop cluster with 196 threads. This study presents that descriptive statistics for large datasets can benefit from distributed computation features of a Hadoop cluster.

Anahtar Kelimeler

• Big Data,Descriptive Statistics,Hadoop,MapReduce

Kaynakça

1. Apache Software Foundation, Hadoop Releases, apache.org, Dec. 10, 2011. [Online]. http://en.wikipedia.org/wiki/Apache_Hadoop. [Accessed: Oct. 06, 2018]
2. Battiato, S., Cantone, D., Catalano, D., Cincotti, G., and Hofri, M. (2000), An efficient algorithm for the approximate median selection problem. Algorithms and complexity, 226-238.
3. Buragohain C., and Suri S. (2009), Encyclopedia of Database Systems, 2235-2240, Springer US.
4. Chen C. P., and Zhang C.Y. (2014), Data-intensive applications, challenges, techniques and technologies: A survey on Big Data, Information Sciences ,275, 314-347
5. Cheung P. (2012), Big Data, Official Statistics and Social Science Research: Emerging Data Challenges, Presentation at the World Bank.
6. Ciaccio A. Di, Coli M., Ibanez A., and Miguel J. (2012), Advanced Statistical Methods for the Analysis of Large Data-Sets.
7. Daas P., Tennekes M., Jonge E. De, Priem A., Buelens B., Pelt M. Van, and Hurk P. Van Den (2012), Data Science and the Future of Statistics Presentation at the first Data Science NL meetup, http://www.slideshare.net/pietdaas/data-science-and-the-future-of-statistics.
8. Douglas L. (2001), 3d data management: Controlling data volume, velocity and variety, https://blogs.gartner.com/doug-laney/files/2012/01/ad949-3D-Data-Management-Controlling-Data-Volume-Velocity-and-Variety.pdf.

9. Glasson M., Trepanier J., Patruno V., Daas P., and Skaliotis M., Khan A. (2013), What does “Big Data” mean for Official Statistics? https://statswiki.unece.org/pages/viewpage.action?pageId=77170614&preview=/77170614/80805923/Big%20Data%20HLG%20Final%20Published%20Version.docx.
10. Gobble M. (2013), Big Data: the next big thing in innovation, Research-Technology Management, 56, 64–66.
11. Hastie T., Tibshirani R., and Friedman J. (2002), The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Springer, Stanford.
12. HDFS Architecture, https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html.
13. HDFS Users Guide, http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html.
14. Hive, https://hive.apache.org.
15. Jain, R., and Chlamtac, I. (1985). The P2 algorithm for dynamic calculation of quantiles and histograms without storing observations. Communications of the ACM, 28(10), 1076-1085.
16. Kelley, I., and Blumenstock, J. (2014). Computational challenges in the analysis of large, sparse, spatiotemporal data. In Proceedings of the sixth international workshop on Data intensive distributed computing, 41-46. ACM.
17. Klemens B. (2008), Modeling with Data: Tools and Techniques for Statistical Computing, Princeton University Press.
18. Manku, G. S., Rajagopalan, S., and Lindsay, B. G. (1998), Approximate medians and other quantiles in one pass and with limited memory. In ACM SIGMOD Record, 27 (2), 426-435.
19. MapReduce, https://wiki.apache.org/hadoop/MapReduce.
20. Mardia, K. V., and Zemroch, P. J. (1975), Algorithm AS 84: Measures of multivariate skewness and kurtosis. Journal of the Royal Statistical Society. Series C (Applied Statistics), 24(2), 262-265.
21. Munro J.J., and Paterson M.S. (1980), Selection and sorting with limited storage, Theor. Comput. Sci., 12, 315–323.
22. NIST/SEMATECH, e-Handbook of Statistical Methods, http://www.itl.nist.gov/div898/handbook/, [Accessed: 19.01.2018].
23. NSF (2012), Core techniques and technologies for advancing big data science and engineering (BIGDATA), https://www.nsf.gov/pubs/2012/nsf12499/nsf12499.htm.
24. Philippe P., Thompson D., Bennett J., and Mascarenhas A. (2011), Design and performance of a scalable, parallel statistics toolkit, 2011 IEEE International Symposium on Parallel and Distributed Processing Workshops and Phd Forum (IPDPSW), 1475–1484.
25. Spark, https://spark.apache.org/documentation.html.
26. Sysoev O., Oleg B., and Grimvalla A. (2011), A segmentation-based algorithm for large-scale partially ordered monotonic regression, Comput. Stat.Data Anal, 55 (8), 2463–2476.
27. White T., (2012), Hadoop the Definitive Guide, 3rd Edition, O’Reilly Media.
28. Wilkinson L., (2008), The future of statistical computing, Technometrics, 50 (4), 418–435.
29. Yarn, https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html.