A Text Scam Detection Model Using Text-To-Text Transfer Transformer Architecture

Abstract

The increase in the complexity of text message fraud in online communication endangers privacy, financial stability and user confidence. The traditional approaches to detection, like rule-based systems, keyword matching, and statistical classifiers, have difficulties in adapting to new scam techniques: they tend to end up with high false positive and false negative rates. The proposed work is the development of an improved scam detector using the Text to Text Transfer Transformer (T5) architecture with the Sentence -T5 encoding for semantic data compression. The proposed model was optimized using data comprised of the Kaggle benchmark dataset and scam messages that were previously unseen (collected and directly fed to the model using crowd sourcing and web scraping). Experiments were conducted, and the results were compared with Na¨ıve Bayes, SVM, Decision Tree, Random Forest, LSTM, Logistic Regression and K Nearest Neighbor. The comparative experiments have shown that the proposed model achieved 98% in all aspects of accuracy, precision, recall, and F1 Score, performing better than all the other baseline models. The proposed model displayed good generalization across unseen scam variations and, therefore, can be applied in secure messaging infrastructures. This work presents a robust and scalable, context-aware method to address performance and adaptability problems associated with fraud detection.

Keywords

• Cyber security
• NLP
• Sentence-T5 encoding
• Transformer architecture

Ethical Statement

We declare that this work was conducted in accordance with proven scientific and ethical standards. The work applies a publicly available method and does not involve direct interaction with human participants or animals. Subsequently, all referenced works have been properly cited in the bibliography section, and adherence to ethical guidelines has been maintained for this work.

References

1. P. Laskov and R. Lippmann, “Machine learning in adversarial environments,” Mach. Learn., vol. 81, no. 2, pp. 115–119, 2010, doi : 10.1007/s10994-010-5207-6.
2. G. Park and J. M. Taylor, “Towards text-based phishing detection,” arXiv preprint arXiv:2111.01676, 2021. [Online]. Available: https://arxiv.org/abs/2111.01676
3. S. Rojas-Galeano, “Using BERT encoding to tackle the mad-lib attack in SMS spam detection,” arXiv preprint arXiv:2107.06400, 2021. [Online]. Available: https://arxiv.org/abs/2107.06400
4. O. Agboola, “Spam detection using machine learning and deep learning,” Ph.D. dissertation, Louisiana State Univ. and Agric. & Mech. College, Baton Rouge, LA, USA, 2022, doi : 10.31390/gradschool dissertations.6011.
5. S. Gadde, A. Lakshmanarao, and S. Satyanarayana, “SMS spam detection using machine learning and deep learning techniques,” in Proc. 7th Int. Conf. Adv. Comput. Commun. Syst. (ICACCS), Coimbatore, India, Mar. 2021, pp. 358–362, doi : 10.1109/ICACCS51430.2021.9441783.
6. Y. Lee, J. Saxe, and R. Harang, “CATBERT: Context-aware tiny BERT for detecting social engineering emails,” arXiv preprint arXiv:2010.03484, 2020. [Online]. Available: https://arxiv.org/abs/2010.03484
7. T. Brown et al., “Language models are few-shot learners,” in Adv. Neural Inf. Process. Syst., vol. 33, Vancouver, Canada, pp. 1877–1901, 2020.
8. A. Radford, J. Wu, R. Child, D. Luan, and D. Amodei, “Language models are unsupervised multitask learners,” OpenAI, Tech. Rep., 2019. [Online]. Available: https://cdn.openai.com/better-language-models/language models are unsupervised multitask learners.pdf

9. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. NAACL-HLT, Minneapolis, MN, USA, 2019, pp. 4171–4186.
10. M. Salman, M. Ikram, and M. A. Kaafar, “An empirical analysis of SMS scam detection systems,” arXiv preprint arXiv:2210.10451, 2022. [Online]. Available: https://arxiv.org/abs/2210.10451
11. D. Mamakas, P. Tsotsi, I. Androutsopoulos, and I. Chalkidis, “Processing long legal documents with pre-trained transformers: Modding LegalBERT and Longformer,” in Proc. Nat. Legal Lang. Process. Workshop, Abu Dhabi, United Arab Emirates, 2022, pp. 130–142.
12. C. Raffel et al., “Exploring the limits of transfer learning with a unified text-to-text transformer,” J. Mach. Learn. Res., vol. 21, no. 140, pp. 1–67, 2020.
13. Y. Tay, M. Dehghani, D. Bahri, and D. Metzler, “Efficient transformers: A survey,” ACM Comput. Surv., vol. 55, no. 6, pp. 1–28, 2022, doi : 10.1145/3530811.
14. J. Ni et al., “Sentence-T5: Scalable sentence encoders from pre-trained text-to-text models,” in Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, 2022, pp. 1864–1874.
15. T. Gao, X. Yao, and D. Chen, “SimCSE: Simple contrastive learning of sentence embeddings,” in Proc. EMNLP, Punta Cana, Dominican Republic, 2021, pp. 6894–6910.
16. Y. Liu et al., “RoBERTa: A robustly optimized BERT pretraining approach,” arXiv preprint arXiv:1907.11692, 2019. [Online]. Available: https://arxiv.org/abs/1907.11692