Advances in Transformer-Based Semantic Search: Techniques, Benchmarks, and Future Directions

Abstract

Semantic search has developed quickly as the need for accurate information retrieval has increased in a variety of fields, from expert knowledge systems to web search engines. Conventional search methods that rely solely on keywords frequently fail to understand user intent and contextual hints. This survey focuses on recent advances in Transformer-based models, such as BERT, RoBERTa, T5, and GPT, which leverage self-attention mechanisms and contextual embeddings to deliver heightened precision and recall across diverse domains. Key architectural elements underlying these models are discussed, including dual-encoder and cross-encoder frameworks, and how Dense Passage Retrieval extends their capabilities to large-scale applications is examined. Practical considerations, such as domain adaptation and fine-tuning strategies, are reviewed to highlight their impact on real-world deployment. Benchmark evaluations (e.g., MS MARCO, TREC, and BEIR) are also presented to illustrate performance gains over traditional Information Retrieval methods and explore ongoing challenges involving interpretability, bias, and resource-intensive training. Lastly, emerging trends—multimodal semantic search, personalized retrieval, and continual learning—that promise to shape the future of AI-driven information retrieval are identified for more efficient and interpretable semantic search.

Keywords

• semantic search
• transformer
• information retrieval
• natural language processing

References

1. Abdi, H., Singular value decomposition (SVD) and generalized singular value decomposition (GSVD), In: Encyclopedia of Measurement and Statistics, Sage Publications, 2007.
2. Aliyu, M.B., Efficiency of Boolean search strings for Information retrieval, American Journal of Engineering Research, 6(11)(2017), 216–222.
3. Aumuller, M., Bernhardsson, E., Faithfull, A., ANN-benchmarks: A benchmarking tool for approximate nearest neighbor algorithms, Information Systems, 87(2020), 101374.
4. Bringmann, K., Chaudhuri, K., Dao, T., et al., Domain adaptation in the presence of distribution shift, Artificial Intelligence, 321(2024), 103946.
5. Chen, H., Tian, X., Liu, B., Overview of the MS MARCO 2024 passage ranking challenge, Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2024.
6. Church, K.W., Word2vec, Natural Language Engineering, 23(1)(2017), 155–162.
7. Cong, Y., Chai, Z., Zeng, Y., et al., Self-supervised weight prediction for continual learning, IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10)(2023), 11939–11952.
8. Delobelle, P., Winters, T., Berendt, B., RobBERT: A Dutch RoBERTa-based language model, arXiv preprint arXiv:2001.06286, 2020.

9. Deng, J., Berg, A.C., Li, K., Fei-Fei, L., Hierarchical semantic indexing for large scale image retrieval, CVPR 2011, 785–792, 2011.
10. Dakhel, A.M., Majdinasab, V., Nikanjam, A., Khomh, F., Gu´eh´eneuc, Y.G., An empirical study of bugs in GitHub Copilot generated code, Information and Software Technology, 156(2023), 107155.
11. Ekanayake, J.B., Godaliyadda, G.M.R.I., Ekanayake, M.P.B., Dinalankara, D.M.S.K., Amaratunga, P.G.C., A novel SHAP based interpretable machine learning approach for short term load forecasting, IEEE Access, 10(2022), 22997–23010.
12. Erk, K., Vector space models of word meaning and phrase meaning: A survey, Language and Linguistics Compass, 6(10)(2012), 635–653.
13. Esteva, A., Kale, A., Paulus, R., Hashimoto, K., Yin, W., et al., CO-Search: COVID-19 information retrieval with semantic search, question answering, and abstractive summarization, arXiv preprint arXiv:2006.09595, 2021.
14. Floridi, L., Chiriatti, M., GPT-3: Its nature, scope, limits, and consequences, Minds and Machines, 30(4)(2020), 681–694.
15. Gillioz, A., Casas, J., Mugellini, E., Khaled, O.A., Overview of the transformer-based models for NLP tasks, 2020 15th Conference on Computer Science and Information Systems (FedCSIS), IEEE, 2020 179–183.
16. Goyal, N., Du, J., Neubig, G., Berg-Kirkpatrick, T., Carbonell, J., Larger-scale transformers for multilingual masked language modeling, arXiv preprint arXiv:2105.00572, 2021.
17. Harman, D.K., Overview of the first TREC conference, Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, 1993, 36–47.
18. Jawahar, G., Sagot, B., Seddah, D., What does BERT learn about the structure of language?, ACL 2019-57th Annual Meeting of the Association for Computational Linguistics, 2019.
19. Johnson, J., Douze, M., J´egou, H., Billion-scale similarity search with GPUs, IEEE Transactions on Big Data, 7(3)(2019), 535–547.
20. Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., et al., Dense passage retrieval for open-domain question answering, arXiv preprint arXiv:2004.04906, 2020.
21. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., et al., Natural questions: A benchmark for question answering research, Transactions of the Association for Computational Linguistics, 7(2019), 453–466.
22. Lamport, L., The part-time parliament, ACM Transactions on Computer Systems, 16(2)(1990), 133–169.
23. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., et al., ALBERT: A lite BERT for self-supervised learning of language representations, arXiv preprint arXiv:1909.11942, 2019.
24. Landauer, T.K., Foltz, P.W., Laham, D., An introduction to latent semantic analysis, Discourse processes, 25(2-3)(1998), 259–284.
25. Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., et al., BioBERT: A pre-trained biomedical language representation model for biomedical text mining, Bioinformatics, 36(4)(2020), 1234–1240.
26. Lin, J.J., Han, S.C., Luo, X., et al., Using CLIP for image classification: An evaluation, IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5)(2023), 5779–5793.
27. Lin, Y., Han, S., Mao, H., Wang, Y., Dally, W.J., Deep gradient compression: Reducing the communication bandwidth for distributed training, arXiv preprint arXiv:1712.01887, 2020.
28. Liu, P., Li, W., Zou, L., A survey of contextual embedding models for pre-trained embeddings based natural language understanding, Neurocomputing, 421(2020), 146–158.
29. Lupu, M., Piroi, F., Huang, X., Wade, J., Tait, J., Overview of the TREC 2014 legal track, TREC, 2014.
30. Montavon, G., Binder, A., Lapuschkin, S., Samek, W., M¨uller, K.R., Layer-wise relevance propagation: An overview, Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, 2019 193–209.
31. Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., et al., MTEB: Massive text embedding benchmark, arXiv preprint arXiv:2210.07316, 2022.
32. Naqvi, M.A., Bader, Y., Shashanka, C., Unlocking AI potential: A deep dive into generative AI, ethical implications, and the future of technology, Journal of Applied Business, Social Sciences and Technology, 1(2)(2024), 193–202.
33. Ni, J., A´ brego, G.H., Constant, N., Ma, J., Hall, K.B., et al., Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models, arXiv preprint arXiv:2108.08877, 2021.
34. Patel, P., Patel, P., Deval, D., Shah, M., Tinysearch: A lightweight search engine for scholarly papers, arXiv preprint arXiv:1912.08878, 2019.
35. Pennington, J., Socher, R., Manning, C.D., Glove: Global vectors for word representation, Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014 1532–1543.
36. Poleksic, A., Tingay, M., The effects of fine-tuning and vocabulary overriding in SciBERT, Frontiers in Artificial Intelligence, 6(2023), 1138183.
37. Preethi, G., Krishna, P.V., Obaidat, M.S., Saritha, V., Yenduri, S., Application of deep learning to sentiment analysis for recommender system on cloud, 2017 International conference on computer, information and telecommunication systems (CITS), IEEE, 93–97, 2017.
38. Priyadarshini, I., Mohanty, P., Kumar, R., Sharma, R., Puri, V., et al., A novel autoencoder and neural network-based selective ensemble learning scheme for effective detection of fake news, IEEE Access, 9(2021), 45498–45513.
39. Rahali, I., Ben-Abacha, A., Zhang, Y., et al., End-to-end biomedical entity linking with span-level sequence tagging, Journal of Biomedical Informatics, 138(2023), 104298.
40. Ramos, J., Using tf-idf to determine word relevance in document queries, Proceedings of the first instructional conference on machine learning, 242(1)(2003), 29–48.
41. Rietzler, A., Stabinger, S., Opitz, P., Engl, S., Adapt or get left behind: Domain adaptation through BERT language model finetuning for aspect-target sentiment classification, arXiv preprint arXiv:1908.11860, 2019.
42. Robertson, S., Understanding inverse document frequency: On theoretical arguments for IDF, Journal of Documentation, 60(5)(2004), 503–520.
43. Rothman, D., Transformers for natural language processing: Build innovative deep neural network architectures for NLP with Python, PyTorch, TensorFlow, BERT, RoBERTa, and more, Packt Publishing Ltd, 2021.
44. Rothman, D., Transformers for natural language processing: build, train, and fine-tune deep neural network architectures for NLP with Python, TensorFlow 2.0, and the Hugging Face Transformers library, Packt Publishing Ltd, 2022.
45. Roy, D., Paul, D., Mitra, M., Garain, U., Using word embeddings for automatic query expansion, arXiv preprint arXiv:1606.07608, 2016.
46. Sakketou, F., Ampazis, N., A constrained optimization algorithm for learning GloVe embeddings with semantic lexicons, Knowledge-Based Systems, 195(2020), 105628.
47. Salton, G., Buckley, C., Term-weighting approaches in automatic text retrieval, Information processing & management, 24(5)(1988), 513–523.
48. Sharma, M., Selvi, V., Chauhan, R., Khan, S.A., Siddiqua, A., et al., The Future of Business with Generative AI Models and Insights, 2025 3rd International Conference on Intelligent Systems, Advanced Computing and Communication (ISACC), 2025, 386–391.
49. Seo, M.J., Lee, J., Jeong, T., Kwiatkowski, T., Bhagavatula, C., et al., Real-time open-domain question answering with retrieval-augmented language models, arXiv preprint arXiv:2207.13353, 2022.
50. Smith, J.S., Valkov, L., Halbe, S., Gutta, V., Feris, R., et al., Adaptive memory replay for continual learning, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, 3605–3615.
51. Stucke, M.E., Ezrachi, A., How digital assistants can harm our economy, privacy, and democracy, Berkeley Technology Law Journal, 32(3)(2017), 1239–1300.
52. Tabani, H., Arnau, J.M., Tubella, J., Gonz´alez, A., Improving the efficiency of transformer-based language models: Memory bandwidth optimization through compact weight reconstruction, arXiv preprint arXiv:2103.12621, 2021.
53. Tan, H., Bansal, M., LXMERT: Learning cross-modality encoder representations from transformers, arXiv preprint arXiv:1908.07490, 2019.
54. Thakur, N., Reimers, N., R¨uckl´e, A., Srivastava, A., Gurevych, I., BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models, arXiv preprint arXiv:2104.08663, 2021.
55. Tsatsaronis, G., Balikas, G., Malakasiotis, P., et al., An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition, BMC bioinformatics, 16(1)(2015), 1–28.
56. Voorhees, E., Rajput, S., Soboroff, I., Overview of the TREC 2021 deep learning track, arXiv preprint arXiv:2203.09870, 2021.
57. Wang, P., Xu, B., Xu, J., Tian, G., Liu, C.L., et al., Semantic expansion using word embedding clustering and convolutional neural network for improving short text classification, Neurocomputing, 174(2017), 806–814.
58. Wang, H., Yu, L., Xia, S., Chen, H., Feng, H., A distilled dual-encoder model for vision-language understanding, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021 16483–16492.
59. Wermelinger, M., Talbot, P., Using Codex for automated assessment of student software design, Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, 2023, 1209–1215.
60. Wu, F., Fast text searching: allowing errors, Communications of the ACM, 35(10)(1992), 83–91.
61. Wu, S., Dredze, M., Are all languages created equal in multilingual BERT?, arXiv preprint arXiv:2005.09093, 2020.
62. Xu, F., Uszkoreit, H., Du, Y., Fan, W., Zhao, D., Zhu, J., Explainable AI: A brief survey on history, research areas, approaches and challenges, CCF international conference on natural language processing and Chinese computing, Springer, 2019.
63. Ye, Z., Jin, Y., Han, Y., Ding, X., Feng, Y., et al., A comprehensive survey on generative pre-trained transformer (gpt) language models, arXiv preprint arXiv:2305.12693, 2023.
64. Yu, Y., Si, X., Hu, C., Zhang, J., A review of recurrent neural networks: LSTM cells and network architectures, Neural Computation, 31(7)(2019), 1235–1270.
65. Zha, H., Yang, G., Li, S., Huang, X., Hu, X., The role of position information in transformer language models, Computational Linguistics, 49(2)(2023), 359–383.