Comparing AI-Assisted Code with Human-Written Code: A Python Case Study

Abstract

The rise of generative artificial intelligence (AI)-based code generation tools is transforming code writing processes in software engineering. This study aims to systematically compare Python programs generated by the AI-powered “Bolt.New” tool with equivalent human-written programs in terms of quality, complexity, and readability. Thirty human-written programs selected from GitHub prior to 2019, along with AI-generated programs performing the same functions, were categorized into simple, medium, and complex based on line count and analyzed. Metrics such as quality score, error count, cyclomatic complexity, style violations, code duplication rate, and documentation score, measured using the Python-specific static analysis tool Pylint, were statistically compared between the two groups. The findings reveal that AI offers high quality and low error rates in simple tasks, while in complex tasks, it provides lower complexity and better documentation but shows limitations in contextual appropriateness. This study evaluates the compliance of AI-generated code with Python-specific standards and its potential for error reduction using objective metrics, offering a guiding framework for balancing automation and human oversight in software development processes.

Keywords

• Artificial Intelligence
• Code Quality
• Software Engineering
• Code Readability
• Code Complexity
• Python

Yapay Zeka Destekli Üretilen Kodlar ile İnsan Yazımı Kodların Karşılaştırılması: Python Örneği

Abstract

Üretken yapay zeka (YZ) temelli kod üretim araçlarının yükselişi, yazılım mühendisliğinde kod yazım süreçlerini dönüştürmektedir. Bu çalışma, YZ destekli “Bolt.New” aracıyla üretilen Python programlarını, insan eliyle yazılmış eşdeğer programlarla kalite, karmaşıklık ve okunabilirlik açısından sistematik olarak karşılaştırmayı amaçlamaktadır. 2019 öncesi GitHub’dan seçilen 30 adet insan yapımı program ile aynı işlevleri gerçekleştiren YZ üretimi programlar, satır sayılarına göre basit, orta ve karmaşık kategorilere ayrılarak analiz edilmiştir. Python’a özgü statik analiz aracı Pylint ile ölçülen kalite puanı, hata sayısı, siklomatik karmaşıklık, stil ihlali, kod tekrar oranı ve dokümantasyon puanı metrikleri, iki grup arasında istatistiksel olarak karşılaştırılmıştır. Bulgular, YZ’nin basit görevlerde yüksek kalite ve düşük hata oranı sunduğunu, karmaşık görevlerde ise daha düşük karmaşıklık ve daha iyi dokümantasyon sağladığını, ancak bağlamsal uygunlukta sınırlamalar gösterebildiğini ortaya koymaktadır. Bu çalışma, YZ destekli kod üretiminin Python’a özgü standartlara uyumunu ve hata azaltma potansiyelini nesnel metriklerle değerlendirerek, yazılım geliştirme süreçlerinde otomasyon ve insan denetimi arasında denge kurulmasına yönelik rehber bir çerçeve sunmaktadır.

Keywords

• Yapay Zeka
• Kod Kalitesi
• Yazılım Mühendisliği
• Kod Okunabilirliği
• Kod Karmaşıklığı
• Python

References

1. Allamanis, M., Barr, E. T., Devanbu, P., Sutton, C. A survey of machine learning for big code and naturalness. ACM Computing Surveys, 51(4), 1–37, 2018. https://doi.org/10.1145/3212695
2. Beller, M., Bholanath, R., McIntosh, S., Zaidman, A. Analyzing the state of static analysis: A large-scale evaluation in open source software. In 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), pp. 470–481, 2016. https://doi.org/10.1109/SANER.2016.13
3. Bird, J. J., Ekárt, A., Faria, D. R. Ethical considerations of large language models in code generation: A systematic literature review. Ethics and Information Technology, 25(3), 1–15, 2023. https://doi.org/10.1007/s10676-023-09684-7
4. Boehm, B. W., Brown, J. R., Kaspar, H., Lipow, M., MacLeod, G. J., Merritt, M. J. Characteristics of software quality. North-Holland Publishing, Amsterdam, Netherlands, 1978.
5. Bolt.New Team. Bolt.New: AI-powered code generation for Python. https://bolt.new/docs, erişim tarihi: 15 Mayıs 2025.
6. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Amodei, D. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. https://doi.org/10.48550/arXiv.2005.14165
7. Buse, R. P., Weimer, W. R. Learning a metric for code readability. IEEE Transactions on Software Engineering, 36(4), 546–558, 2010. https://doi.org/10.1109/TSE.2009.71
8. Carlini, N., Wagner, D. Adversarial examples are not bugs, they are features. Advances in Neural Information Processing Systems, 32, 184–194, 2019.

9. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. de O., Kaplan, J., Zaremba, W. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. https://arxiv.org/abs/2107.03374
10. Dakhel, A. M., Majd, M., Nikanjam, A., Khomh, F., Desmarais, M. C., Jiang, Z. M. J. GitHub Copilot AI pair programmer: Asset or liability? Journal of Systems and Software, 203, 111734, 2023. https://doi.org/10.1016/j.jss.2023.111734
11. Dijkstra, E. W. The structure of the “THE”-multiprogramming system. Communications of the ACM, 11(5), 341–346, 1968. https://doi.org/10.1145/363095.363143
12. Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., Prather, J. The robots are coming: Exploring the implications of OpenAI Codex on introductory programming. In Proceedings of the 24th Australasian Computing Education Conference, pp. 10–19, 2022. https://doi.org/10.1145/3511861.3511863
13. Goodfellow, I., Bengio, Y., Courville, A. Deep learning. MIT Press, 2016.
14. Halstead, M. H. Elements of software science. Elsevier, Amsterdam, Netherlands, 1977.
15. Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Steinhardt, J. Measuring coding challenge competence with APPS. Advances in Neural Information Processing Systems, 34, 22214–22226, 2021.
16. Imai, S. Is GitHub Copilot a substitute for human pair-programming? An empirical study. In 2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), pp. 319–321, 2022. https://doi.org/10.1109/ICSE-Companion55297.2022.9793819
17. Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Vinyals, O. Competition-level code generation with AlphaCode. Science, 378(6624), 1092–1097, 2023. https://doi.org/10.1126/science.abq1158
18. McCabe, T. J. A complexity measure. IEEE Transactions on Software Engineering, SE-2(4), 308–320, 1976. https://doi.org/10.1109/TSE.1976.233837
19. Mozannar, H., Bansal, G., Sontag, D. Evaluating the robustness of neural code generation models. arXiv preprint arXiv:2210.12468, 2022. https://doi.org/10.48550/arXiv.2210.12468
20. Nguyen, T., Li, Y., Le-Cong, T., Nguyen, T. N. Exploring the impact of code style in Python: An empirical study. arXiv preprint arXiv:2208.09051, 2022. https://arxiv.org/abs/2208.09051
21. Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., Karri, R. Asleep at the keyboard? Assessing the security of GitHub Copilot’s code contributions. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 754–768, 2022. https://doi.org/10.1109/SP46214.2022.9833651
22. Posnett, D., Hindle, A., Devanbu, P. A simpler model of software readability. In Proceedings of the 8th Working Conference on Mining Software Repositories, pp. 73–82, 2011. https://doi.org/10.1145/1985441.1985454
23. Pylint Development Team. Pylint: Static code analysis for Python. https://pylint.readthedocs.io/en/latest/, erişim tarihi: 15 Mayıs 2025.
24. Python Software Foundation. PEP 8 – Style guide for Python code. https://peps.python.org/pep-0008/, 2023.
25. Scalabrino, S., Linares-Vásquez, M., Oliveto, R., Bavota, G. A comprehensive model for code readability. Journal of Software: Evolution and Process, 30(6), e1928, 2018. https://doi.org/10.1002/smr.1928
26. Sommerville, I. Software engineering (10th ed.). Pearson, Boston, MA, 2015.
27. Vaithilingam, P., Zhang, T., Glassman, E. L. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1–14, 2022. https://doi.org/10.1145/3491102.3517665
28. Weisz, J. D., Muller, M., Houde, S., Richards, J., Ross, S. I., Cabrero, F. Perfection not required? Human-AI partnerships in code generation. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 402–412, 2021. https://doi.org/10.1145/3461702.3462537
29. Vaswani, A., Shazeer, N., Parmar, N., Uszoreit, J., Jones, L., Gomez, A. N., Polosukhin, I. Attention is all you need. Advances in Neural Information Processing Systems, 30, pp. 5998–6008, 2017. https://arxiv.org/abs/1706.03762
30. Xu, F. F., Vaswani, A., Parmar, N. Fine-tuning large language models for domain-specific code generation. arXiv preprint arXiv:2302.01687, 2023. https://doi.org/10.48550/arXiv.2302.01687
31. Ziegler, A., Kalliamvakou, E., Simister, A., Sittampalam, G., Li, A., Rice, A., Rifkin, R. Productivity assessment of neural code completion. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pp. 21–29, 2022. https://doi.org/10.1145/3520312.3534864