Performance Comparison of Artificial Intelligence Applications in Python Code Generation

Abstract

The employment of artificial intelligence techniques in the realm of software production has emerged as a pivotal instrument within the software development process. A multitude of studies have examined the potential applications of artificial intelligence at various stages of the software development lifecycle. The field of software engineering incorporates artificial intelligence in areas such as project management, code generation, software testing, defect prediction, and vulnerability detection. In this study, an objective comparison was made of AI applications in terms of code quality, productivity, creativity and flexibility, fault tolerance, and documentation. The AI applications evaluated include OpenAI (ChatGPT, Codex), Google Gemini (Bard), Copilot (GitHub), DeepCode (Snyk Code), and Microsoft Copilot. For specific Python tasks, the codes generated by the AI application were compared by creating scenarios, and the solution accuracy, fault tolerance, code annotations, and practical performance of each application were evaluated. Consequently, OpenAI Codex was identified as a highly effective solution for complex projects, GitHub Copilot emerged as an ideal choice for daily development, Google Gemini proved to be adequate for basic solutions, Microsoft Copilot demonstrated its capacity for enterprise integration and security-compliant solutions, and DeepCode was distinguished for its emphasis on code security and quality control.

Keywords

• Artificial intelligence
• coding
• fault tolerance
• code quality
• Python code development

Yapay Zeka Uygulamalarının Python Kod Üretimindeki Performans Karşılaştırması

Abstract

Yapay zeka tekniklerinin yazılım üretimi alanında kullanılması, yazılım geliştirme sürecinde çok önemli bir araç olarak ortaya çıkmıştır. Çok sayıda çalışma, yazılım geliştirme yaşam döngüsünün çeşitli aşamalarında yapay zekanın potansiyel uygulamalarını incelemiştir. Yazılım mühendisliği alanı, proje yönetimi, kod üretimi, yazılım testi, hata tahmini ve güvenlik açığı tespiti gibi alanlarda yapay zekayı kullanmaktadır. Bu çalışmada, yapay zekâ uygulamalarının kod kalitesi, üretkenlik, yaratıcılık ve esneklik, hata toleransı ve dokümantasyon açısından objektif bir karşılaştırması yapılmıştır. Değerlendirilen YZ uygulamaları arasında OpenAI (ChatGPT, Codex), Google Gemini (Bard), Copilot (GitHub), DeepCode (Snyk Code) ve Microsoft Copilot bulunmaktadır. Belirli Python görevleri için, yapay zekâ uygulaması tarafından üretilen kodlar senaryolar oluşturularak karşılaştırılmış ve her uygulamanın çözüm doğruluğu, hata toleransı, kod ek açıklamaları ve pratik performansı değerlendirilmiştir. Sonuç olarak, OpenAI Codex karmaşık projeler için oldukça etkili bir çözüm olarak belirlenmiş, GitHub Copilot günlük geliştirme için ideal bir seçim olarak ortaya çıkmış, Google Gemini temel çözümler için yeterli olduğunu kanıtlamış, Microsoft Copilot kurumsal entegrasyon ve güvenlik uyumlu çözümler için kapasitesini göstermiş ve DeepCode kod güvenliği ve kalite kontrolüne verdiği önemle öne çıkmıştır.

Keywords

• Yapay zeka
• kodlama
• hata toleransı
• kod kalitesi
• Python kod geliştirme

References

1. Crawford, T., Duong, S., Fueston, R., Lawani, A., Owoade, S. J., Uzoka, A., Parizi, R. M., & Yazdinejad, A. (2023). AI in Software Engineering: A Survey on Project Management Applications. In arXiv (Cornell University). https://doi.org/10.48550/arxiv.2307.15224
2. Lyu, M. R. (2018). AI Techniques in Software Engineering Paradigm (p. 2). ICPE '18: Proceedings of the 2018 ACM/SPEC International Conference on Performance Engineering. https://doi.org/10.1145/3184407.3184440
3. Kalech, M., Abreu, R., & Last, M. (2021). Artificial Intelligence Methods for Software Engineering. In WORLD SCIENTIFIC eBooks. World Scientific. https://doi.org/10.1142/12360
4. Nagulapati, V., Rapelli, S. R., & Fiaidhi, J. (2020). Automating Software Development using Artificial Intelligence. Techrxiv. https://doi.org/10.36227/techrxiv.12089139.v1
5. Perez, L., Ottens, L., & Viswanathan, S. (2021). Automatic code generation using pre-trained language models. ArXiv. https://doi.org/10.48550/arXiv.2102.10535
6. Soliman, A. S., Hadhoud, M. M., & Shaheen, S. I. (2022). MarianCG: a code generation transformer model inspired by machine translation. Journal of Engineering and Applied Science, 69(104), 1-23. https://doi.org/10.1186/s44147-022-00159-4
7. Liu, H., Tsai, C., & Day, M. (2024). A Pilot Study on AI-Assisted Code Generation with Large Language Models for Software Engineering. Technologies and Applications of Artificial Intelligence. TAAI 2023. Communications in Computer and Information Science (s. 162-175). https://doi.org/10.1007/978-981-97-1711-8_12
8. Zhang, H., & Shao, H. (2023, December 15). Exploring the Latest Applications of OpenAI and ChatGPT: An In-Depth Survey. Computer Modeling in Engineering & Sciences 2024, 138(3), 2061-2102. https://doi.org/10.32604/cmes.2023.030649

9. Hunt, A. &. (1999). The Pragmatic Programmer: Your Journey to Mastery. Addison-Wesley.
10. Rossum, G. V. (2001, April 15). Python Reference Manual. Berkeley: https://lab.demog.berkeley.edu/Docs/Refs/Python2.1/ref.pdf
11. Natarajan, R., Madlambayan, M. A., & Patalay, M. M. (2022). Python Programming For Beginners. Xoffencer Publication. https://doi.org/10.5281/zenodo.7580581
12. Javed, A., Monika, Z., Uddin, M. M., & Nusrat, T. (2019). An Analysis on Python Programming Language Demand and Its Recent Trend in Bangladesh. ICCPR '19: Proceedings of the 2019 8th International Conference on Computing and Pattern Recognition, 458-465. https://doi.org/10.1145/3373509.3373540
13. Hanke, M. (2009, February 1). PyMVPA: a unifying approach to the analysis of neuroscientific data. In Frontiers in Neuroinformatics, 3, 1-13. https://doi.org/10.3389/neuro.11.003.2009
14. Peps. (2001, Julay 05). Toggle light / dark / auto colour theme. Retrieved January 01, 2025, from Python Enhancement Proposals: https://peps.python.org/pep-0008/
15. Zaczyński , B. (2025, January 01). Profiling in Python: How to Find Performance Bottlenecks. Retrieved January 01, 2025, from Realpython: https://realpython.com/python-profiling/
16. Cormen, T. H., Leiserson, C. E., Rivest, R. L., & Stein, C. (2009). Introduction to Algorithms. MIT Press.
17. Martin, R. C. (2008). Clean Code: A Handbook of Agile Software Craftsmanship. Prentice Hall.
18. Brett, S. (2020). Effective Python: 90 Specific Ways to Write Better Python (Effective Software Development Series). Addison Wesley.
19. OpenAI. (2025). Some section of this work were drafted with the help of the AI model GPT4. https://chat.openai.com/
20. Google Gemini. (2025, Fabruary 1). Gemini: https://gemini.google.com/
21. Github Copilot. (2025, Fabruary 1). Github Copilot: https://github.com/features/copilot
22. Snyk. (2025, January 1). Snyk. Snyk: https://snyk.io/product/
23. Microsoft. (2025, Fabruary 1). Microsoft: https://www.microsoft.com /en-us/microsoft-365/blog/2025/01/15/copilot-for-all-introducing-microsoft-365-copilot-chat/
24. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam,P., Sastry, G., Askell, Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., & Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877–1901. https://arxiv.org/abs/2005.14165
25. OpenAI. (2023). GPT-4 technical report. https://openai.com/research/gpt-4
26. Du, X., Liu, M., Wang, K., Wang, H., Liu, J., Chen, Y., Feng, J., Sha, C., Peng, X., & Lou, Y. (2024, April). Evaluating large language models in class-level code generation. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (pp. 1-13). https://doi.org/10.1145/3597503.36392
27. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint, arXiv:1810.04805. https://arxiv.org/abs/1810.04805
28. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140), 1–67. https://arxiv.org/abs/1910.10683
29. Google DeepMind. (2024). Gemini 1.5 Technical Report. https://deepmind.google /technologies/gemini
30. Microsoft. (2024). Introducing Microsoft Copilot: Your everyday AI companion. https://blogs.microsoft.com/blog/2024/03/21/introducing-microsoft-copilot-your-everyday-ai-companion/
31. Zhang, B., Liang, P., Zhou, X., & Ahmad, A. (2023). Practices and Challenges of Using GitHub Copilot: An Empirical Study. The 35th International Conference on Software Engineering & Knowledge Engineering. https://doi.org/10.48550/arXiv.2303.08733
32. Meta AI. (2024). Introducing Code Llama: A Code-Specialized LLM. https://ai.meta.com/blog/code-llama-large-language-model-coding/
33. DeepSeek. (2024). DeepSeek Coder Technical Report. https://deepseek.com/research/deepseek-coder
34. Amazon Web Services. (2025). Amazon CodeWhisperer Documentation. https://docs.aws.amazon.com/ codewhisperer/
35. Replit. (2025). Replit Gho. Build apps and sites with AI. https://replit.com.
36. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, E., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., Zaremba, W., & Zaremba, W. (2021). Evaluating large language models trained on code. arXiv preprint, arXiv:2107.03374. https://arxiv.org/abs/2107.03374
37. Nijkamp, E., Tu, Z., Caldwell, A., Lin, X. V., Zhang, Y., Liu, Y., & Savarese, S. (2023). CodeGen2: Lessons for Training LLMs on Programming and Natural Languages. arXiv preprint arXiv:2303.17568.
38. ISO. (2011). ISO/IEC 25010:2011 - Systems and software engineering . Systems and software Quality Requirements and Evaluation (SQuaRE) — System and software quality models. ISO.
39. Christensen, C. M. (1997). The Innovator's Dilemma: When New Technologies Cause Great Firms to Fail. Harvard Business Review Press.
40. Boswell, D., & Foucher, T. (2011). The Art of Readable Code:Simple and Practical Techniques for Writing Better Code . O'Reilly Media.
41. Gamma, E., Helm, R. J., & Vlissides, J. (1994). Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley.
42. Pressman, R. S. (2014). Software Engineering: A Practitioner's Approach. McGraw-Hill.
43. Beck, K. (2003). Test-Driven Development: By Example. Addison-Wesley.
44. Aho, A. V., Hopcroft, J. E., & Ullman, J. D. (1974). The Design and Analysis of Computer Algorithms. Addison-Wesley.
45. McConnell, S. (2004). Code Complete: A Practical Handbook of Software Construction. Microsoft Press.
46. Boden, M. A. (2004). The Creative Mind: Myths and Mechanisms. Routledge. https://fulldisclosure.info/Files/Books/The%20Creative%20Mind%20%20Myths%20and%20M echanisms.pdf
47. Goodfellow, I. Y. (2016). Deep Learning. MIT Press. Retrieved January 01, 2025, from https://www.deeplearningbook.org/
48. Fowler, M. (2010). Domain-Specific Languages. Addison-Wesley. Addison-Wesley.
49. Spinellis, D. (2003). Code Reading: The Open Source Perspective. Addison-Wesley.
50. Brooks, F. P. (1995). The Mythical Man-Month: Essays on Software Engineering. Addison-Wesley.
51. Chartexpo. (2025, January 01). Radar Chart. Chartexpo: https://chartexpo.com/charts/radar-chart