EMACrawler: Web Search Engine Database Freshness Optimization

Öz

In today's information and technology age, search engines have become an important part of our lives. Although search engines are the first to be used to access information, old and unnecessary information is included in the content offered to users. In terms of providing up-to-date data, today's search engines often cannot offer the desired success. In order to keep the data presented by web browsers up-to-date, the time of return visits must be accurately estimated. In this study, EMACrawler based on exponential moving average is proposed to determine the revisit times, which is the most important feature that affects the performance of search engines. The proposed method is tested using precision, total coverage and efficiency metrics. It has been seen that EMACrawler obtains the current data on the web pages in an accurate and quick manner. As a result of the experimental studies, it has been seen that EMACrawler is more successful than other methods in obtaining up-to-date data and maintaining the freshness of the browser database.

Anahtar Kelimeler

• Web crawler
• update module
• data collection
• data indexing

Proje Numarası

118C127

EMACrawler: Web Arama Motoru Veritabanı Tazeliği Optimizasyonu

Öz

Günümüz bilgi ve teknoloji çağında arama motorları hayatımızın önemli bir parçası haline gelmiştir. Her ne kadar bilgiye erişimde ilk başvurulan arama motorları olsa da kullanıcılara sunulan içerikte eski ve gereksiz bilgiler yer almaktadır. Güncel verileri sağlamak açısından günümüzdeki arama motorları çoğunlukla istenen başarıyı sunamamaktadır. Web tarayıcılarının sunduğu verilerin güncelliğini sağlamak için tekrar ziyaret zamanının doğru tahmin edilmesi gerekmektedir. Bu çalışmada arama motorlarının performanslarını etkileyen en önemli özellik olan tekrar ziyaret zamanlarının belirlenmesi için üstel hareketli ortalamaya dayanan EMACrawler önerilmiştir. Önerilen yöntem kesinlik, toplam kapsama alanı ve verimlilik metrikleri kullanılarak test edilmiştir. EMACrawler’ın web sayfalarındaki güncel veriyi doğru tahmin zamanında ve hızlı bir şekilde elde ettiği görülmüştür. Yapılan deneysel çalışmaların sonucunda EMACrawler’ın güncel verilerin elde edilmesi ve tarayıcı veri tabanının tazeliğinin korunmasında diğer yöntemlerden daha başarılı olduğu görülmüştür.

Anahtar Kelimeler

• Web tarayıcısı
• güncelleme modülü
• veri toplama
• veri indeksleme

Destekleyen Kurum

Türkiye Bilimsel ve Teknolojik Araştırma Kurumu (TÜBİTAK)

Proje Numarası

118C127

Teşekkür

Bu çalışma, TÜBİTAK tarafından BİDEB-2244 Sanayi Doktora Programı kapsamında 118C127 numara ile desteklenen "İnternette Heterojen Veri Kaynaklarından Veri Toplanması, Doğrulanması ve Sorgulanması" başlıklı projenin bir parçasıdır. Sağladığı destek için TÜBİTAK’a teşekkür ederiz.

Kaynakça

1. [1] Google,"How Google Search Works", www.google.com, [Erişim Tarihi: 10/08/2022).
2. [2] Sadiku M., Musa S., and Nelatury S. R., "Future Internet research," International Journal of Advances in Scientific Research and Engineering (IJASRE), Erie, PY 2(3):23-25, (2017).
3. [3] Jaiganesh S., Babu P., and Satheesh K. N., "Comparative study of various web search algorithms for the improvement of web crawler," Int. J. Eng. Res. Technol.(IJERT), 4(2): (2013).
4. [4] Li K., Fei J., and Fan C., "Optimization and application of web crawler architecture," SPIE, 12506: 151-155, (2022).
5. [5] Patil T. A. and Chobe S., "Web Crawler for searching Deep web sites," in 2017 International Conference on Computing, Communication, Control and Automation (ICCUBEA), Pune, India,1-5 (2017).
6. [6] Avrachenkov K., Borkar V., and Patil K., "Deep reinforcement learning for web crawling," in Seventh Indian Control Conference (ICC), Mumbai, India:201-206 (2021).
7. [7] Mallawaarachchi V., Meegahapola L., Madhushanka R., Heshan E., Meedeniya D., and Jayarathna S., "Change detection and notification of web pages: A survey," ACM Computing Surveys (CSUR), 1(53):1-35, (2020).
8. [8] Bullot H., Gupta S. K., and Mohania M. K., "A data-mining approach for optimizing performance of an incremental crawler," in Proceedings IEEE/WIC International Conference on Web Intelligence (WI 2003), 13(17):610-615, (2003)

9. [9] Kharazmi S., Nejad A. F., and Abolhassani H., "Freshness of Web search engines: Improving performance of Web search engines using data mining techniques," in 2009 International Conference for Internet Technology and Secured Transactions, (ICITST), London, UK,1-7, (2009).
10. [10] Jianchao H., Cercone N., and Xiaohua H., "A Weighted Freshness Metric for Maintaining Search Engine Local Repository," in IEEE/WIC/ACM International Conference on Web Intelligence (WI'04), Beijing, China, 677-680, (2004).
11. [11] Amudhan V. and Thirupathi D., "Traffic Adaptive Optimum Updating Scheme for Search Engines," in 2006 1st International Conference on Digital Information Management, 6(6):395-403, (2007)
12. [12] Zhu W., Li Y., Li S., Xu Y., and Cui X., "Optimal bandwidth allocation for web crawler systems with time constraints," Journal of Ambient Intelligence and Humanized Computing, 5(14):5279-5292, (2023)
13. [13] Souza C., Laber E., Valentim C., and Cardoso E., "A Polite Policy for Revisiting Web Pages," in 2007 Latin American Web Conference (LA-WEB 2007), Santiago, Chile,128-135, (2007).
14. [14] Bhatia S., Sharma M., and Bhatia K. K., "A Novel Approach for Crawling the Opinions from World Wide Web," (in English), International journal of information retrieval research, 2(6): 1-23, (2016).
15. [15] Tan Q. and Mitra P., "Clustering-based incremental web crawling," ACM Trans. Inf. Syst.,4(28):1-27, (2010)
16. [16] Radinsky K. and Bennett P. N., "Predicting content change on the web," presented at the Proceedings of the sixth ACM international conference on Web search and data mining, Rome,415-424 (2013).
17. [17] Li H., Guo M., Cai L., and Yang Y., "An incremental update strategy in Deep Web," in 2010 Sixth International Conference on Natural Computation, Yantai, China, 131-134, (2010).
18. [18] Mor J., Rai D., and Kumar N., "An XML based Web Crawler with Page Revisit Policy and Updation in Local Repository of Search Engine," International Journal of Engineering & Technology,7(3): 1119-1123, (2018).
19. [19] Kausar M. A., Nasar M., and Singh S. K., "Maintaining the repository of search engine freshness using mobile crawler," in 2013 Annual International Conference on Emerging Research Areas and 2013 International Conference on Microelectronics, Communications and Renewable Energy, Kanjirapally, India, 1-6,(2013).
20. [20] Badawi M., Mohamed A., Hussein A., and Gheith M., "Maintaining the search engine freshness using mobile agent," Egyptian Informatics Journal, 1(14):27-36, (2013)
21. [21] Gupta A., Dixit A., and Sharma A., "A Novel Web Page Change Detection Technique for Migrating Crawlers," In: Sensors and Image Processing: Proceedings of CSI. Springer, Singapore, 49-57 (2018).
22. [22] Sethi S., "An optimized crawling technique for maintaining fresh repositories," Multimedia Tools and Applications, 7(80):11049-11077, (2021).
23. [23] Santos A. S. R., Carvalho C. R., Almeida J. M., Moura E. S. de, Silva A. S. da, and Ziviani N., "A genetic programming framework to schedule webpage updates," Information Retrieval Journal, 1(18):73-94, (2015).
24. [24] Fasolin K. et al., "Efficient Execution of Conjunctive Complex Queries on Big Multimedia Databases," in 2013 IEEE International Symposium on Multimedia, Anaheim, CA, 536-543,(2013).
25. [25] Gani A., Siddiqa A., Shamshirband S., and Hanum F., "A survey on indexing techniques for big data: taxonomy and performance evaluation," Knowledge and Information Systems, 2(46): 241-284 (2016).
26. [26] Shah S. and Shaikh A., "Hash based optimization for faster access to inverted index," in 2016 International Conference on Inventive Computation Technologies (ICICT), Coimbatore, India,1-5,(2016).
27. [27] Petri M. and Moffat A., "Compact inverted index storage using general-purpose compression libraries," Software: Practice and Experience, 4(48):974-982,(2018).
28. [28] "World Wide Web Size", https://www.worldwidewebsize.com/ [Erişim Tarihi : 18/8/2023].
29. [29] Burkov A. and Chaib-draa B., "Effective learning in the presence of adaptive counterparts," Journal of Algorithms, 4(65):127-138, (2009).
30. [30] Hansun S., "A new approach of moving average method in time series analysis," in 2013 Conference on New Media Studies (CoNMedia), Tangerang, Indonesia, 1-4, (2013).
31. [31] Zuo X. L., Wang W. Wang B., Y., and Zuo W. L., "Research and Implementation of Improved Real-Time Crawler Modeling," in Applied Mechanics and Materials, vol. 312:791-795 (2013).
32. [32] Zerfos P., Cho J., and Ntoulas A., "Downloading textual hidden web content through keyword queries," in Proceedings of the 5th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL '05), Denver, CO:100-109, (2005).
33. [33] "Most Visited Websites in Turkey "https://www.semrush.com/website/top/turkey/all/ [Erişim Tarihi: 12/03/2023]