SmartLLMSentry: A Comprehensive LLM Based Smart Contract Vulnerability Detection Framework

Abstract

Smart contracts are essential for managing digital assets in blockchain networks, highlighting the need for effective security measures. This paper introduces SmartLLMSentry, a novel framework that leverages large language models (LLMs), specifically ChatGPT with in-context training, to advance smart contract vulnerability detection. Traditional rule-based frameworks have limitations in integrating new detection rules efficiently. In contrast, SmartLLMSentry utilizes LLMs to streamline this process. We created a specialized dataset of five randomly selected vulnerabilities for model training and evaluation. Our results show an exact match accuracy of 91.1% with sufficient data, although GPT-4 demonstrated reduced performance compared to GPT-3 in rule generation. This study illustrates that SmartLLMSentry significantly enhances the speed and accuracy of vulnerability detection through LLM-driven rule integration, offering a new approach to improving Blockchain security and addressing previously underexplored vulnerabilities in smart contracts.

Keywords

• Smart contract
• Vulnerability
• Software security
• Blockchain
• Large Language Models

References

1. Shabani Baghani, A., Rahimpour, S., & Khabbazian, M. (2022). The DAO Induction Attack: Analysis and Countermeasure. IEEE Internet of Things Journal, 9(7), 4875–4887. IEEE Internet of Things Journal. https://doi.org/10.1109/JIOT.2021.3108154
2. Fatima Samreen, N., & Alalfi, M. H. (2020). Reentrancy Vulnerability Identification in Ethereum Smart Contracts. 2020 IEEE International Workshop on Blockchain Oriented Software Engineering (IWBOSE), 22–29. https://doi.org/10.1109/IWBOSE50093.2020.9050260
3. Zaazaa, O., & Bakkali, H. E. (n.d.). Unveiling the Landscape of Smart Contract Vulnerabilities: A Detailed Examination and Codification of Vulnerabilities in Prominent Blockchains.
4. Matulevicius, N., & Cordeiro, L. C. (2021). Verifying Security Vulnerabilities for Blockchain-based Smart Contracts. 2021 XI Brazilian Symposium on Computing Systems Engineering (SBESC), 1–8. https://doi.org/10.1109/SBESC53686.2021.9628229
5. etherscan.io. (n.d.). Ethereum Daily Deployed Contracts Chart | Etherscan. Ethereum (ETH) Blockchain Explorer. Retrieved July 22, 2024, from https://etherscan.io/chart/deployed-contracts
6. Singh, N., Meherhomji, V., & Chandavarkar, B. R. (2020). Automated versus Manual Approach of Web Application Penetration Testing. 2020 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT), 1–6.
7. OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., … Zoph, B. (2023). GPT-4 Technical Report (arXiv:2303.08774). arXiv.
8. Cao, J., Li, M., Wen, M., & Cheung, S. (2023). A study on Prompt Design, Advantages and Limitations of ChatGPT for Deep Learning Program Repair. Association for Computing Machinery, 1(1).

9. Sobania, D., Briesch, M., Hanna, C., & Petke, J. (2023). An Analysis of the Automatic Bug Fixing Performance of ChatGPT. 2023 IEEE/ACM International Workshop on Automated Program Repair (APR), 23–30. https://doi.org/10.1109/APR59189.2023.00012
10. OpenAI Platform. (n.d.). Retrieved June 11, 2024, from https://platform.openai.com
11. Austin, A., Holmgreen, C., & Williams, L. (2013). A comparison of the efficiency and effectiveness of vulnerability discovery techniques. Information and Software Technology, 55(7), 1279–1288. https://doi.org/10.1016/j.infsof.2012.11.007
12. Schneidewind, C., Grishchenko, I., Scherer, M., & Maffei, M. (2020). eThor: Practical and Provably Sound Static Analysis of Ethereum Smart Contracts. Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, 621–640. https://doi.org/10.1145/3372297.3417250
13. Nguyen, T. D., Pham, L. H., & Sun, J. (2021). SGUARD: Towards Fixing Vulnerable Smart Contracts Automatically. 2021 IEEE Symposium on Security and Privacy (SP), 1215–1229. https://doi.org/10.1109/SP40001.2021.00057
14. Wang, D., Jiang, B., & Chan, W. K. (2020). WANA: Symbolic Execution of Wasm Bytecode for Cross-Platform Smart Contract Vulnerability Detection*#. CoRR, abs/2007.15510, 12. https://doi.org/10.48550/arXiv.2007.15510
15. Hao, X., Ren, W., Zheng, W., & Zhu, T. (2020). SCScan: A SVM-Based Scanning System for Vulnerabilities in Blockchain Smart Contracts. 2020 IEEE 19th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom), 1598–1605. https://doi.org/10.1109/TrustCom50675.2020.00221
16. Ye, J., Ma, M., Lin, Y., Sui, Y., & Xue, Y. (2020). Clairvoyance: Cross-contract static analysis for detecting practical reentrancy vulnerabilities in smart contracts. Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Companion Proceedings, 274–275. https://doi.org/10.1145/3377812.3390908
17. Feist, J., Grieco, G., & Groce, A. (2019). Slither: A Static Analysis Framework for Smart Contracts. 2019 IEEE/ACM 2nd International Workshop on Emerging Trends in Software Engineering for Blockchain (WETSEB), 8–15. https://doi.org/10.1109/WETSEB.2019.00008
18. Tang, Y., Li, Z., & Bai, Y. (2021). Rethinking of Reentrancy on the Ethereum. 2021 IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech), 68–75. https://doi.org/10.1109/DASC-PICom-CBDCom-CyberSciTech52372.2021.00025
19. Grech, N., Kong, M., Jurisevic, A., Brent, L., Scholz, B., & Smaragdakis, Y. (2018). MadMax: Surviving out-of-gas conditions in Ethereum smart contracts. Proceedings of the ACM on Programming Languages, 2(OOPSLA), 1–27. https://doi.org/10.1145/3276486
20. Nguyen, T. D., Pham, L. H., Sun, J., Lin, Y., & Minh, Q. T. (2020). sFuzz: An efficient adaptive fuzzer for solidity smart contracts. Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, 778–788. https://doi.org/10.1145/3377811.3380334
21. Ren, M., Ma, F., Yin, Z., Li, H., Fu, Y., Chen, T., & Jiang, Y. (2021). SCStudio: A secure and efficient integrated development environment for smart contracts. Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, 666–669. https://doi.org/10.1145/3460319.3469078
22. Gao, Z., Jayasundara, V., Jiang, L., Xia, X., Lo, D., & Grundy, J. (2019). SmartEmbed: A Tool for Clone and Bug Detection in Smart Contracts through Structural Code Embedding. 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME), 394–397. https://doi.org/10.1109/ICSME.2019.00067
23. Yu, X., Zhao, H., Hou, B., Ying, Z., & Wu, B. (2021). DeeSCVHunter: A Deep Learning-Based Framework for Smart Contract Vulnerability Detection. 2021 International Joint Conference on Neural Networks (IJCNN), 1–8. https://doi.org/10.1109/IJCNN52387.2021.9534324
24. Wu, H., Zhang, Z., Wang, S., Lei, Y., Lin, B., Qin, Y., Zhang, H., & Mao, X. (2021). Peculiar: Smart Contract Vulnerability Detection Based on Crucial Data Flow Graph and Pre-training Techniques. 2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE), 378–389. https://doi.org/10.1109/ISSRE52982.2021.00047
25. Ashizawa, N., Yanai, N., Cruz, J. P., & Okamura, S. (2021). Eth2Vec: Learning Contract-Wide Code Representations for Vulnerability Detection on Ethereum Smart Contracts. Proceedings of the 3rd ACM International Symposium on Blockchain and Secure Critical Infrastructure, 47–59. https://doi.org/10.1145/3457337.3457841
26. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N., Chen, A., Creel, K., Davis, J. Q., Demszky, D., … Liang, P. (2021). On the Opportunities and Risks of Foundation Models. CoRR, abs/2108.07258.
27. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., … Amodei, D. (2020). Language Models are Few-Shot Learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems (p. 25).
28. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., Kluska, A., Lewkowycz, A., Agarwal, A., Power, A., Ray, A., Warstadt, A., Kocurek, A. W., Safaya, A., Tazarv, A., … Wu, Z. (2022). Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. In ArXiv preprint arXiv:2206.04615. http://arxiv.org/abs/2206.04615
29. Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., Liu, Z., … Wen, J.-R. (2023). A Survey of Large Language Models. ArXiv Preprint ArXiv:2303.18223.
30. GPT-4 | OpenAI. (n.d.). Retrieved June 12, 2024, from https://openai.com/index/gpt-4/
31. Gemini Team, Reid, M., Savinov, N., Teplyashin, D., Dmitry, Lepikhin, Lillicrap, T., Alayrac, J., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., Antonoglou, I., Anil, R., Borgeaud, S., Dai, A., Millican, K., Dyer, E., Glaese, M., … Vinyals, O. (2024). Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. ArXiv Preprint ArXiv:2403.05530.
32. Kevian, D., Syed, U., Guo, X., Havens, A., Dullerud, G., Seiler, P., Qin, L., & Hu, B. (2024). Capabilities of Large Language Models in Control Engineering: A Benchmark Study on GPT-4, Claude 3 Opus, and Gemini 1.0 Ultra (arXiv:2404.03647). arXiv.
33. Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., & Jiang, D. (2023). WizardCoder: Empowering Code Large Language Models with Evol-Instruct. ArXiv Preprint ArXiv:2306.08568.
34. Hu, S., Huang, T., İlhan, F., Tekin, S. F., & Liu, L. (2023). Large Language Model-Powered Smart Contract Vulnerability Detection: New Perspectives. 2023 5th IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA), 297–306. https://doi.org/10.1109/TPS-ISA58951.2023.00044
35. Sun, Y., Wu, D., Xue, Y., Liu, H., Ma, W., Zhang, L., Shi, M., & Liu, Y. (2024). LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs’ Vulnerability Reasoning. ArXiv.
36. Zaazaa, O., & El Bakkali, H. (2022). Automatic Static Vulnerability Detection Approaches and Tools: State of the Art (pp. 449–459). https://doi.org/10.1007/978-3-030-91738-8_41
37. Zhang, Y., Feng, S., & Tan, C. (2022). Active Example Selection for In-Context Learning. ArXiv Preprint ArXiv:2211.04486. https://doi.org/10.48550/arXiv.2211.04486
38. Pan, J., Gao, T., Chen, H., & Chen, D. (2023). What In-Context Learning “Learns” In-Context: Disentangling Task Recognition and Task Learning. Princeton University.
39. Wang, Y.-X., Ramanan, D., & Hebert, M. (2017). Growing a Brain: Fine-Tuning by Increasing Model Capacity. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3029–3038.
40. Church, K. W., Chen, Z., & Ma, Y. (2021). Emerging trends: A gentle introduction to fine-tuning. Natural Language Engineering, 27(6), 763–778. https://doi.org/10.1017/S1351324921000322
41. Chen, J., Liu, Z., Huang, X., Wu, C., Liu, Q., Jiang, G., Pu, Y., Lei, Y., Chen, X., Wang, X., Zheng, K., Lian, D., & Chen, E. (2024). When large language models meet personalization: Perspectives of challenges and opportunities. World Wide Web, 27(4), 42. https://doi.org/10.1007/s11280-024-01276-1
42. Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023). Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. ACM Computing Surveys, 55(9), 1–35. https://doi.org/10.1145/3560815
43. Pitis, S., Zhang, M. R., Wang, A., & Ba, J. (2023). Boosted Prompt Ensembles for Large Language Models. ArXiv Preprint ArXiv:2304.05970.
44. Zhao, Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021). Calibrate Before Use: Improving Few-shot Performance of Language Models. Proceedings of the 38th International Conference on Machine Learning, 12697–12706. https://proceedings.mlr.press/v139/zhao21c.html
45. Gao, T., Fisch, A., & Chen, D. (2021). Making Pre-trained Language Models Better Few-shot Learners. In C. Zong, F. Xia, W. Li, & R. Navigli (Eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (pp. 3816–3830). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.acl-long.295
46. Jiang, Z., Xu, F. F., Araki, J., & Neubig, G. (2020). How Can We Know What Language Models Know? Transactions of the Association for Computational Linguistics, 8, 423–438. https://doi.org/10.1162/tacl_a_00324
47. Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., & Chen, W. (2022). What Makes Good In-Context Examples for GPT-3? Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, 100–114. https://doi.org/10.18653/v1/2022.deelio-1.10
48. Gu, Y., Han, X., Liu, Z., & Huang, M. (2022). PPT: Pre-trained Prompt Tuning for Few-shot Learning. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 8410--8423. https://doi.org/10.18653/v1/2022.acl-long.576
49. Lester, B., Al-Rfou, R., & Constant, N. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 3045--3059. https://doi.org/10.18653/v1/2021.emnlp-main.243
50. Li, X. L., & Liang, P. (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation. In C. Zong, F. Xia, W. Li, & R. Navigli (Eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (pp. 4582–4597). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.acl-long.353
51. Liu, V., & Chilton, L. B. (2022). Design Guidelines for Prompt Engineering Text-to-Image Generative Models. CHI Conference on Human Factors in Computing Systems, 1–23. https://doi.org/10.1145/3491102.3501825
52. Maddigan, P., & Susnjak, T. (2023). Chat2VIS: Generating Data Visualizations via Natural Language Using ChatGPT, Codex and GPT-3 Large Language Models. IEEE Access, 11, 45181–45193. IEEE Access. https://doi.org/10.1109/ACCESS.2023.3274199
53. Liu, C., Bao, X., Zhang, H., Zhang, N., Hu, H., Zhang, X., & Yan, M. (2023). Improving ChatGPT Prompt for Code Generation. ArXiv Preprint ArXiv:2305.08360.
54. White, J., Hays, S., Fu, Q., Spencer-Smith, J., & Schmidt, D. C. (2023). ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design. Generative AI for Effective Software Development, 71--108. https://doi.org/10.1007/978-3-031-55642-5_4
55. Demir, M., Alalfi, M., Turetken, O., & Ferworn, A. (2019). Security Smells in Smart Contracts. 2019 IEEE 19th International Conference on Software Quality, Reliability and Security Companion (QRS-C), 442–449. https://doi.org/10.1109/QRS-C.2019.00086
56. EIP-1884: Repricing for trie-size-dependent opcodes. (n.d.). Ethereum Improvement Proposals. Retrieved August 12, 2024, from https://eips.ethereum.org/EIPS/eip-1884
57. How Ethereum’s Istanbul Network Upgrade Affects DeFi. (2021, July 14). Defi Pulse Blog. https://defipulse.com/blog/how-ethereums-istanbul-network-upgrade-affects-defi/
58. Staderini, M., Palli, C., & Bondavalli, A. (2020). Classification of Ethereum Vulnerabilities and their Propagations. 2020 Second International Conference on Blockchain Computing and Applications (BCCA), 44–51. https://doi.org/10.1109/BCCA50787.2020.9274458
59. ERC 721—OpenZeppelin Docs. (2024, August 12). https://docs.openzeppelin.com/contracts/2.x/api/token/ERC721
60. Publications/reviews/2023-07-arcade-securityreview.pdf at master · trailofbits/publications. (n.d.). GitHub. Retrieved August 12, 2024, from https://github.com/trailofbits/publications/blob/master/reviews/2023-07-arcade-securityreview.pdf
61. Sending Ether (transfer, send, call) | Solidity by Example | 0.8.24. (n.d.). Retrieved August 12, 2024, from https://solidity-by-example.org/sending-ether/
62. PublicReports/Solidity Smart Contract Audits/Persistence_StkBNB_Smart_Contract_Security_Audit_Report_Halborn_Final.pdf at master · HalbornSecurity/PublicReports. (n.d.). GitHub. Retrieved August 12, 2024, from https://github.com/HalbornSecurity/PublicReports/blob/master/Solidity%20Smart%20Contract%20Audits/Persistence_StkBNB_Smart_Contract_Security_Audit_Report_Halborn_Final.pdf
63. PublicReports/Solidity Smart Contract Audits/Cere_Bridge_Smart_Contract_Security_Audit_Solidity_Report_Halborn_Final.pdf at master · HalbornSecurity/PublicReports. (n.d.). GitHub. Retrieved August 12, 2024, from https://github.com/HalbornSecurity/PublicReports/blob/master/Solidity%20Smart%20Contract%20Audits/Cere_Bridge_Smart_Contract_Security_Audit_Solidity_Report_Halborn_Final.pdf
64. Hou, W., & Ji, Z. (2024). A systematic evaluation of large language models for generating programming code. ArXiv Preprint ArXiv:2403.00894.