A Modular Efficiency Determination Formula for Information Retrieval Evaluations and Optimizations

Abstract

The notion of efficiency has typically been associated with the efficiency of systems rather than users in Information Retrieval (IR) literature. In the usability literature, on the other hand, this notion is defined from a user-based perspective, corresponding to how long a user accomplishes a task. Despite this, the common aim for both has to do with the efficient use of time. This study examines the efficiency notion in the IR literature from a user-based efficiency window in the usability literature. In the present study, a modular efficiency determination formula (MEDEF) to create different efficiency indicators by focusing on IR system evaluations and optimizations from the usability perspective is proposed. The MEDEF can be thought of as an efficiency indicator creator based on both effectiveness metrics and efficiency indicators already used in IR studies. In the scope of this study, eight MEDEF-based efficiency indicators were created and compared to several baseline efficiency indicators already used in IR studies. While the study’s first aim is to reveal how consistent the MEDEF-based indicators are and whether these indicators are more successful/reliable than the baselines, the second is to set an example of the usage of efficiency indicators in evaluations of IR systems from a usability perspective. General findings from interactive user behaviour for one month show that the MEDEF-based indicators outperform the baseline indicators and further strengthen the reflections in the baseline indicators. Several usage scenarios regarding the potential of the MEDEF are also shared and discussed in the scope of the study.

Keywords

• Human-Computer Interaction
• Efficiency Indicator
• Session Abandonment
• Interactive Information Retrieval

Bilgi Erişim Değerlendirmeleri ve Optimizasyonları İçin Modüler Bir Verimlilik Belirleme Formülü

Abstract

Verimlilik kavramı, Bilgi Erişim (BE) literatüründe temel olarak kullanıcılardan ziyade sistemlerin verimliliği ile ilişkilendirilmiştir. Kullanılabilirlik literatüründe ise bu kavram, bir kullanıcının bir görevi ne kadar sürede tamamladığına karşılık gelen kullanıcı tabanlı bir bakış açısıyla tanımlanır. Yine de, ortak amaç her iki literatür için de zamanı verimli kullanmaktır. Bu çalışma, BE literatüründeki etkinlik kavramını, kullanılabilirlik literatüründeki kullanıcı tabanlı etkinlik penceresinden incelemektedir. Bu çalışmada, kullanılabilirlik perspektifinden BE sistem değerlendirmelerine ve optimizasyonlarına odaklanarak farklı verimlilik göstergeleri oluşturmak için modüler bir verimlilik belirleme formülü (MEDEF) önerilmiştir. MEDEF, BE çalışmalarında hâlihazırda kullanılan etkililik metriklerine ve verimlilik göstergelerine dayalı bir verimlilik göstergesi üreticisi şeklinde düşünülebilir. Bu çalışma kapsamında, sekiz MEDEF tabanlı verimlilik göstergesi oluşturulmuş ve hâlihazırda BE çalışmalarında kullanılan birkaç temel verimlilik göstergesiyle karşılaştırılmıştır. Çalışmanın ilk amacı, MEDEF temelli göstergelerin ne kadar tutarlı olduğunu ve bu göstergelerin mevcut temel göstergelere göre daha başarılı/güvenilir olup olmadığını ortaya koymak iken, ikincisi, kullanılabilirlik açısından BE sistemlerinin değerlendirmelerinde verimlilik göstergelerinin kullanımına bir örnek oluşturmaktır. Bir aylık etkileşimli kullanıcı davranışlarından elde edilen genel bulgular, MEDEF tabanlı göstergelerin temel göstergelerden daha iyi performans gösterdiğini ve temel göstergelerdeki yansımaları daha da güçlendirdiğini göstermiştir. MEDEF‘in potansiyeline ilişkin çeşitli kullanım senaryoları da çalışma kapsamında paylaşılmakta ve tartışılmaktadır.

Keywords

• İnsan-Bilgisayar Etkileşimi
• Verimlilik Göstergesi
• Oturum Terk Etme
• Etkileşimli Bilgi Erişim

References

1. Agichtein, E., Brill, E., & Dumais, S. (2006). Improving web search ranking by incorporating user behavior information. Proceedings of the Twenty-Ninth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2006, 19-26. New York, New York, USA: ACM Press. https://doi.org/10.1145/1148170.1148177 google scholar
2. Agichtein, E., Brill, E., Dumais, S., & Ragno, R. (2006). Learning user interaction models for predicting web search result preferences. Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval - SIGIR ’06, 3. New York, New York, USA: ACM Press. https://doi.org/10.1145/1148170.1148175 google scholar
3. Alhabashneh, O., Iqbal, R., Doctor, F., & James, A. (2017). Fuzzy rule based profiling approach for enterprise information seeking and retrieval. Information Sciences, 394-395, 18-37. https://doi.org/10.1016/J.INS.2016.12.040 google scholar
4. Arguello, J. (2014). Predicting search task difficulty. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics): Vol. 8416 LNCS (pp. 88-99). https://doi.org/10.1007/978-3-319-06028-6_8 google scholar
5. Arkhipova, O., Grauer, L., Kuralenok, I., & Serdyukov, P. (2015). Search Engine Evaluation based on Search Engine Switching Prediction. Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, 723-726. New York, NY, USA: ACM. https://doi.org/10.1145/2766462.2767786 google scholar
6. Aula, A., Khan, R. M., & Guan, Z. (2010). How does search behavior change as search becomes more difficult? Proceedings of the 28th International Conference on Human Factors in Computing Systems - CHI ’10, 35. New York, New York, USA: ACM Press. https://doi.org/10.1145/1753326.1753333 google scholar
7. Balakrishnan, V., & Zhang, X. (2014). Implicit user behaviours to improve post-retrieval document relevancy. Computers in Human Behavior, 33, 104-112. https://doi.org/10.1016/J.CHB.2014.01.001 google scholar
8. Beierle, F., Aizawa, A., Collins, A., & Beel, J. (2020). Choice overload and recommendation effectiveness in related-article recommendations. International Journal on Digital Libraries, 21(3), 231-246. https://doi.org/10.1007/s00799-019-00270-7 google scholar

9. Belkin, N. J., Kelly, D. F., Kim, G., Kim, J. Y., Lee, H., Muresan, G., Tang, M. C., Yuan, X., Cool, C. (2003). Query Length in Interactive Information Retrieval. SIGIR Forum (ACM Special Interest Group on Information Retrieval), (SPEC. ISS.), 205-212. New York, New York, USA: ACM Press. https://doi.org/10.1145/860472.860474 google scholar
10. Borisov, A., Markov, I., de Rijke, M., & Serdyukov, P. (2016). A Context-aware Time Model for Web Search. Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, 205-214. New York, NY, USA: ACM. https://doi.org/10.1145/2911451.2911504 google scholar
11. Buckley, C., & Voorhees, E. M. (2004). Retrieval evaluation with incomplete information. Proceedings of the 27th Annual International Conference on Research and Development in Information Retrieval - SIGIR ’04, 25. New York, New York, USA: ACM Press. https://doi.org/10.1145/1008992.1009000 google scholar
12. Büttcher, S., Clarke, C. L. A., & Cormack, G. V. (2010). Information Retrieval: Implementing and Evaluating Search Engines. The MIT Press. google scholar
13. Cen, R., Liu, Y., Zhang, M., Ru, L., & Ma, S. (2009). Study on the click context of web search users for reliability analysis. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics): Vol. 5839 LNCS (pp. 397-408). https://doi.org/10.1007/978-3-642-04769-5_35 google scholar
14. Chapelle, O., Metlzer, D., Zhang, Y., & Grinspan, P. (2009). Expected reciprocal rank for graded relevance. Proceeding of the 18th ACM Conference on Information and Knowledge Management - CIKM ’09, 621. New York, New York, USA: ACM Press. https://doi.org/10.1145/1645953.1646033 google scholar
15. Claypool, M., Brown, D., Le, P., & Waseda, M. (2001). Inferring user interest. IEEE Internet Computing, 5(6), 32-39. https://doi.org/10.1109/4236.968829 google scholar
16. Colace, F., De Santo, M., Greco, L., & Napoletano, P. (2015). Improving relevance feedback-based query expansion by the use of a weighted word pairs approach. Journal of the Association for Information Science and Technology, 66(11), 2223-2234. https://doi.org/10.1002/asi.23331 google scholar
17. Croft, B., Metzler, D., & Strohman, T. (2009). Search Engines: Information Retrieval in Practice (1st ed.). Boston: Pearson. google scholar
18. Diriye, A., White, R., Buscher, G., & Dumais, S. (2012). Leaving so soon?: understanding and predicting web search abandonment rationales. Proceedings of the 21st ACM International Conference on Information and Knowledge Management - CIKM ’12, 1025. New York, New York, USA: ACM Press. https://doi.org/10.1145/2396761.2398399 google scholar
19. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E., and Duchesnay, E. (2011). Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12, 2825-2830. Retrieved from http://scikit-learn.sourceforge.net. google scholar
20. Fox, S., Karnawat, K., Mydland, M., Dumais, S., & White, T. (2005). Evaluating implicit measures to improve Web search. ACM Transactions on Information Systems, 23(2), 147-168. https://doi.org/10.1145/1059981.1059982 google scholar
21. Frokjrer, E., Hertzum, M., & Hornbrek, K. (2000). Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’00, 345-352. New York, New York, USA: ACM Press. https://doi. org/10.1145/332040.332455 google scholar
22. Hassan, A. (2012). A semi-supervised approach to modeling web search satisfaction. Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval - SIGIR ’12, 275. New York, New York, USA: ACM Press. https://doi.org/10.1145/2348283.2348323 google scholar
23. Hassan, A., Song, Y., & He, L. (2011). A task level metric for measuring web search satisfaction and its application on improving relevance estimation. Proceedings of the 20th ACM International Conference on Information and Knowledge Management - CIKM ’11, 125. New York, New York, USA: ACM Press. https://doi.org/10.1145/2063576.2063599 google scholar
24. Hassan, A., White, R. W., Dumais, S. T., & Wang, Y.-M. (2014). Struggling or exploring? Disambiguating Long Search Sessions. Proceedings of the 7th ACM International Conference on Web Search and Data Mining, 53-62. New York, NY, USA: ACM. https://doi.org/10.1145/2556195.2556221 google scholar
25. Hornbsk, K. (2006). Current practice in measuring usability: Challenges to usability studies and research. International Journal of Human-Computer Studies, 64(2), 79-102. https://doi.org/10.1016/J.IJHCS.2005.06.002 google scholar
26. Jansen, B. J., & Spink, A. (2005). An analysis of Web searching by European AlltheWeb.com users. Information Processing & Management, 41(2),361-381. https://doi.org/10.1016/S0306-4573(03)00067-0 google scholar
27. Jansen, B. J., Spink, A., Bateman, J., & Saracevic, T. (1998). Real life information retrieval: a study of user queries on the Web. ACM SIGIR Forum, 32(1), 5-17. https://doi.org/10.1145/281250.281253 google scholar
28. Jansen, B. J., Spink, A., & Saracevic, T. (2000). Real life, real users, and real needs: a study and analysis of user queries on the web. Information Processing & Management, 36(2), 207-227. https://doi.org/10.1016/S0306-4573(99)00056-4 google scholar
29. Jarvelin, K., & Kekalainen, J. (2000). IR evaluation methods for retrieving highly relevant documents. SIGIR Forum (ACM Special Interest Group on Information Retrieval), 41-48. New York, New York, USA: ACM Press. https://doi.org/10.1145/3130348.3130374 google scholar
30. Jarvelin, K., & Kekalainen, J. (2002). Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20(4), 422-446. https://doi.org/10.1145/582415.582418 google scholar
31. Jiang, D., Leung, K. W. T., Yang, L., & Ng, W. (2015). Query suggestion with diversification and personalization. Knowledge-Based Systems, 89, 553-568. https://doi.org/10.1016/J.KNOSYS.2015.09.003 google scholar
32. Joachims, T. (2002). Optimizing search engines using clickthrough data. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 133-142. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/775047.775067 google scholar
33. Joachims, T., Granka, L., Pan, B., Hembrooke, H., & Gay, G. (2005). Accurately interpreting clickthrough data as implicit feedback. SIGIR 2005 -Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 154-161. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/1076034.1076063 google scholar
34. Jung, S., Herlocker, J. L., & Webster, J. (2007). Click data as implicit relevance feedback in web search. Information Processing & Management, 43(3), 791-807. https://doi.org/10.1016/J.IPM.2006.07.021 google scholar
35. Kelly, D., & Fu, X. (2007). Eliciting better information need descriptions from users of information search systems. Information Processing & Management,43(1), 30-46. https://doi.org/10.1016/J.IPM.2006.03.006 google scholar
36. Kim, Y., Hassan, A., White, R. W., & Zitouni, I. (2014). Modeling dwell time to predict click-level satisfaction. Proceedings ofthe 7th ACM International Conference on Web Search and Data Mining, 193-202. New York, NY, USA: ACM. https://doi.org/10.1145/2556195.2556220 google scholar
37. Kuhar, M., & Mercun, T. (2022). Exploring user experience in digital libraries through questionnaire and eye-tracking data. Library & Information Science Research, 44(3), 101175. https://doi.org/10.1016/J.LISR.2022.101175 google scholar
38. Lee, C.-J., Teevan, J., & de la Chica, S. (2014). Characterizing multi-click search behavior and the risks and opportunities of changing results during use. Proceedings ofthe 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, 515-524. New York, NY, USA: ACM. https://doi.org/10.1145/2600428.2609588 google scholar
39. Liu, Y., Chen, Y., Tang, J., Sun, J., Zhang, M., Ma, S., & Zhu, X. (2015). Different Users, Different Opinions. Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, 493-502. New York, NY, USA: ACM. https://doi.org/10.1145/2766462.2767721 google scholar
40. Liu, C., Gwizdka, J., & Liu, J. (2010). Helping identify when users find useful documents: examination of query reformulation intervals. Proceeding of the Third Symposium on Information Interaction in Context - IIiX ’10, 215. New York, New York, USA: ACM Press. https://doi.org/10.1145/1840784.1840816 google scholar
41. Liu, C., Liu, J., & Belkin, N. J. (2014). Predicting Search Task Difficulty at Different Search Stages. Proceedings of the 23rd ACM International Conference on Conference on Information andKnowledgeManagement, 569-578. New York, NY, USA: ACM. https://doi.org/10.1145/2661829.2661939 google scholar
42. Makkar, A., & Kumar, N. (2018). User behavior analysis-based smart energy management for webpage ranking: Learning automata-based solution. google scholar
43. Sustainable Computing: Informatics and Systems, 20, 174-191. https://doi.org/10.1016/J.SUSCOM.2018.02.003 google scholar
44. Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to information retrieval. New York: Cambridge University Press. google scholar
45. Moffat, A., & Zobel, J. (2008). Rank-biased precision for measurement of retrieval effectiveness. ACM Transactions on Information Systems, 27(1), 1-27. https://doi.org/10.1145/1416950.1416952 google scholar
46. Nasir, J. A., Varlamis, I., & Ishfaq, S. (2019). A knowledge-based semantic framework for query expansion. Information Processing & Management, 56(5), 1605-1617. https://doi.org/10.1016/j.ipm.2019.04.007 google scholar
47. Nie, L., Jiang, H., Ren, Z., Sun, Z., & Li, X. (2016). Query Expansion Based on Crowd Knowledge for Code Search. IEEE Transactions on Services Computing, 9(5), 771-783. https://doi.org/10.1109/TSC.2016.2560165 google scholar
48. Nielsen, J. (1993). Usability Engineering (1st ed.). Morgan Kaufmann. google scholar
49. Radlinski, F., Kurup, M., & Joachims, T. (2008). How does clickthrough data reflect retrieval quality? Proceeding of the 17th ACM Conference on Information and Knowledge Mining - CIKM ’08, 43. New York, New York, USA: ACM Press. https://doi.org/10.1145/1458082.1458092 google scholar
50. Rosenzweig, E. (2015). Successful User Experience: Strategies and Roadmaps (1st ed.). Morgan Kaufmann. google scholar
51. Rubin, J., & Chisnell, D. (2008). Handbook Of Usability Testing How To Plan Design And Conduct Effective Tests. Wiley Pub. google scholar
52. Sarkar, S., Mitsui, M., Liu, J., & Shah, C. (2020). Implicit information need as explicit problems, help, and behavioral signals. Information Processing & Management, 57(2), 102069. https://doi.org/10.1016/J.IPM.2019.102069 google scholar
53. Singh, J., & Sharan, A. (2017). A new fuzzy logic-based query expansion model for efficient information retrieval using relevance feedback approach. Neural Computing and Applications, 28(9), 2557-2580. https://doi.org/10.1007/s00521-016-2207-x google scholar
54. Singla, A., & White, R. W. (2010). Sampling high-quality clicks from noisy click data. Proceedings of the 19th International Conference on World Wide Web - WWW ’10, 1187. New York, New York, USA: ACM Press. https://doi.org/10.1145/1772690.1772867 google scholar
55. Song, Y., Shi, X., White, R., & Awadallah, A. H. (2014). Context-aware web search abandonment prediction. Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, 93-102. New York, NY, USA: ACM. https://doi.org/10.1145/2600428.2609604 google scholar
56. Stamou, S., & Efthimiadis, E. N. (2010). Interpreting user inactivity on search results. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics): Vol. 5993 LNCS (pp. 100-113). https://doi. org/10.1007/978-3-642-12275-0_12 google scholar
57. Vidinli, I. B., & Ozcan, R. (2016). New query suggestion framework and algorithms: A case study for an educational search engine. Information Processing and Management, 52(5), 733-752. https://doi.org/10.1016/j.ipm.2016.02.001 google scholar
58. Voorhees, E. M. (1999). The TREC-8 Question Answering Track Report. Text REtrieval Conference, 77-82. google scholar
59. Zhai, C., & Massung, S. (2016). Text Data Management and Analysis: A Practical Introduction to Information Retrieval and Text Mining. ACM. https:// doi.org/10.1145/2915031 google scholar