LoCEBERT: Transformer-Based Line-of-Code Embeddings for Smart Contract Code Understanding and Vulnerability Detection

Abstract

Smart contracts play a vital role in blockchain technology, but are vulnerable to security risks due to inadequate semantic representations of their source code. Current transformer-based approaches such as BERT, CodeBERT and CodeT5 excel in lexical and contextual understanding. However, they operate mainly at the token level. This limitation prevents the capture of relationships across multiple source code lines or functions. For instance, BERT and CodeBERT are restricted to 512 tokens, thereby causing truncation of longer sequences, loss of long-range dependencies and reduced vulnerability detection accuracy in complex contracts. To address these, we propose a Line-of-Code (LoC)-based embedding approach, treating each line as a separate entity to preserve program structure. Each line is encoded using CodeBERT, and embeddings are aggregated using summation pooling to retain global context. Experiments on the SmartBugs Solidity dataset validate the effectiveness of the approach. LoC embeddings improve clustering performance, increasing Silhouette Score to 0.0888 from 0.0252 and reducing Davies-Bouldin Index to 2.9521 from 5.2315. Vulnerability detection experiments achieved classification accuracy of 96.2%, outperforming token-level approaches at 87.3%. The proposed approach improves interpretability, scalability and vulnerability detection. Future research should explore larger datasets and additional blockchain codebases. These directions will strengthen robustness, performance and generalization capabilities.

Keywords

• CodeBert
• Long range dependencies
• Blockchain security
• Large language model

Ethical Statement

The authors declare that all scientific and ethical principles were strictly followed during the preparation of this study. All sources consulted and utilized have been appropriately acknowledged and cited in the bibliography.

References

1. N. A. Gupta, M. Bansal, S. Sharma, D. Mehrotra, and M. Kakkar, “Detection of vulnerabilities in blockchain smart contracts using deep learning,” Wireless Networks, vol. 31, no. 1, pp. 201–217, 2025, doi: 10.1007/s11276-024-03755-9.
2. J. Yang, S. Liu, S. Dai, Y. Fang, K. Xie, and Y. Lu, “ByteEye: A smart contract vulnerability detection framework at bytecode level with graph neural networks,” Automated Software Engineering, vol. 33, no. 1, pp. 1–38, 2026.
3. R. Agarwal, T. Thapliyal, and S. K. Shukla, “Vulnerability and transaction behavior based detection of malicious smart contracts,” in International Symposium on Cyberspace Safety and Security. Springer, 2021, pp. 79–96.
4. T. Alladi, V. Chamola, R. M. Parizi, and K.-K. R. Choo, “Blockchain applications for industry 4.0 and industrial IoT: A review,” IEEE Access, vol. 7, pp. 176 935–176 951, 2019, doi: 10.1109/ACCESS.2019.2956748.
5. J. F. Ferreira, P. Cruz, T. Durieux, and R. Abreu, “SmartBugs: A framework to analyze Solidity smart contracts,” in Proceedings of the 35th IEEE/ACM international conference on automated software engineering, 2020, pp. 1349–1352, doi: 10.1145/3324884.3415298.
6. M. Bresil, P. Prasad, M. S. Sayeed, and U. A. Bukar, “Deep learning-based vulnerability detection solutions in smart contracts: A comparative and meta-analysis of existing approaches,” IEEE Access, vol. 13, pp. 28 894–28 919, 2025, doi: 10.1109/ACCESS.2025.3532326.
7. J. Scharfman, “Decentralized autonomous organization (DAO) fraud, hacks, and controversies,” in The Cryptocurrency and Digital Asset Fraud Casebook, Volume II: DeFi, NFTs, DAOs, Meme Coins, and Other Digital Asset Hacks. Springer, 2024, pp. 65–106, doi: 10.1007/978-3-031-60836-0 3.
8. K. Polak, “Hack solidity: Reentrancy attack,” 2022. [Online]. Available: https://hackernoon.com/hack-solidity-reentrancy-attack

9. P. T. Duy et al., “VulnSense: Efficient vulnerability detection in Ethereum smart contracts by multimodal learning with graph neural network and language model,” International Journal of Information Security, vol. 24, no. 1, p. 48, 2025.
10. H. Almeida, P. Pinto, and A. F. Vilas, “A review on cryptocurrency transaction methods for money laundering,” arXiv preprint arXiv:2311.17203, 2023, doi: 10.48550/arXiv.2311.17203.
11. J. Bu, W. Li, Z. Li, Z. Zhang, and X. Li, “SmartBugBERT: BERT-enhanced vulnerability detection for smart contract bytecode,” arXiv preprint arXiv:2504.05002, 2025.
12. X. Cao, J. Zhang, X. Wu, and B. Liu, “A survey on security in consensus and smart contracts,” Peer-to-Peer Networking and Applications, vol. 15, no. 2, pp. 1008–1028, 2022, doi: 10.1007/s12083-021-01268-2.
13. P. Wei, C. Fan, X. Yang, J. Zhang, X. Deng, J. Gan, and Z. Li, “A hybrid local-global feature attention network for thin section rock image classification,” Scientific Reports, vol. 16, p. 6446, 2026, doi: 10.1038/s41598-026-36669-x.
14. S¸. Kara, F. ˙Ilkbahar, and M. Z. G¨und¨uz, “AI-powered vulnerability detection and adaptive defense strategies in cybersecurity,” Sakarya University Journal of Computer and Information Sciences, vol. 8, no. 3, pp. 536–552, 2025.
15. X. Sun, L. Tu, J. Zhang, J. Cai, B. Li, and Y. Wang, “ASSBert: Active and semi-supervised bert for smart contract vulnerability detection,” Journal of Information Security and Applications, vol. 73, p. 103423, 2023, doi: 10.1016/j.jisa.2023.103423.
16. X. Tang, Y. Du, A. Lai, Z. Zhang, and L. Shi, “Deep learning-based solution for smart contract vulnerabilities detection,” Scientific Reports, vol. 13, no. 1, p. 20106, 2023, doi: 10.1038/s41598-023-47219-0.
17. Y. Wang, W. Wang, S. Joty, and S. C. Hoi, “CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” in Proceedings of the 2021 conference on empirical methods in natural language processing, 2021, pp. 8696–8708.
18. I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,” arXiv preprint arXiv:2004.05150, 2020.
19. M. Zaheer et al., “BigBird: Transformers for longer sequences,” Advances in neural information processing systems, vol. 33, pp. 17 283–17 297, 2020.
20. M. J. Islam, S. Islam, M. Hossain, S. Noor, and S. R. Islam, “Securing blockchain systems: A layer-oriented survey of threats, vulnerability taxonomy, and detection methods,” Future Internet, vol. 17, no. 5, p. 205, 2025.
21. Z. Feng et al., “CodeBERT: A pre-trained model for programming and natural languages,” in Findings of the association for computational linguistics: EMNLP 2020, 2020, pp. 1536–1547, doi: 10.18653/v1/2020.findings-emnlp.139.
22. T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” arXiv preprint arXiv:1301.3781, 2013.
23. J. Pennington, R. Socher, and C. D. Manning, “GloVe: Global vectors for word representation,” in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp. 1532–1543, doi: 10.3115/v1/D14-1162.
24. D. Guo et al., “GraphCodeBERT: Pre-training code representations with data flow,” arXiv preprint arXiv:2009.08366, 2020, doi: 10.48550/arXiv.2009.08366.
25. H. Li, G. Xiong, C. Hou, G. Gou, Z. Chen, and Z. Li, “Smart contract vulnerability detection based on AST-augmented heterogeneous graphs,” in 2024 IEEE International Performance, Computing, and Communications Conference (IPCCC). IEEE, 2024, pp. 1–10.
26. R. Ma, Z. Jian, G. Chen, K. Ma, and Y. Chen, “Rejection: An AST-based reentrancy vulnerability detection method,” in Chinese Conference on Trusted Computing and Information Security. Springer, 2019, pp. 58–71.
27. K. S. Tai, R. Socher, and C. D. Manning, “Improved semantic representations from tree-structured long short-term memory (LSTM) networks,” in Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2015, pp. 1556–1566, doi: 10.48550/arXiv.1503.00075.
28. U. Alon, M. Zilberstein, O. Levy, and E. Yahav, “code2vec: Learning distributed representations of code,” Proceedings of the ACM on Programming Languages, vol. 3, no. POPL, pp. 40:1–40:29, 2019, doi: 10.1145/3290353.
29. B. Yang, Z. Zou, and J. Ye, “GNN-based anchor embedding for efficient exact subgraph matching,” arXiv preprint arXiv:2502.00031, 2025.
30. M. Allamanis, E. T. Barr, P. Devanbu, and C. Sutton, “A survey of machine learning for big code and naturalness,” ACM Computing Surveys (CSUR), vol. 51, no. 4, pp. 1–37, 2018.
31. V. J. Hellendoorn, Machine Learning and the Science of Software Engineering. University of California, Davis, 2020.
32. L. Luu, D.-H. Chu, H. Olickel, P. Saxena, and A. Hobor, “Making smart contracts smarter,” in Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, 2016, pp. 254–269.
33. B. Jiang, Y. Liu, and W. K. Chan, “ContractFuzzer: Fuzzing smart contracts for vulnerability detection,” in Proceedings of the 33rd ACM/IEEE international conference on automated software engineering, 2018, pp. 259–269.
34. Y. Zhuang, Z. Liu, P. Qian, Q. Liu, X. Wang, and Q. He, “Smart contract vulnerability detection using graph neural networks,” in Proceedings of the twenty-ninth international conference on international joint conferences on artificial intelligence, 2021, pp. 3283–3290.
35. U. B. Usman, K. Umar, and A. I. Agaie, “CodeELECTRA: An ELECTRA-based approach for improved vulnerability detection in blockchain smart contracts,” Dutse Journal of Pure and Applied Sciences, vol. 10, no. 3b, pp. 95–105, 2024.
36. C. Pornprasit and C. K. Tantithamthavorn, “DeepLineDP: Towards a deep learning approach for line-level defect prediction,” IEEE Transactions on Software Engineering, vol. 49, no. 1, pp. 84–98, 2022.
37. P. Mahbub and M. M. Rahman, “Predicting line-level defects by capturing code contexts with hierarchical transformers,” in Proceedings of the 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), 2024, pp. 308–319, doi: 10.1109/SANER60148.2024.00038.