Araştırma Makalesi

PyLinterBench: A Support-Aware Benchmark for Python Linter Diagnostic Conformance in Controlled and Real-Code Contexts

Cilt: 7 12 Ekim 2026
PDF İndir
EN TR

PyLinterBench: A Support-Aware Benchmark for Python Linter Diagnostic Conformance in Controlled and Real-Code Contexts

Öz

Python linters differ in rule coverage, configuration, and diagnostic behavior, so warning counts alone can mislead comparisons. PyLinterBench is a support-aware benchmark that separates documented rule support from oracle-defined target detection, localization, matched-clean control behavior, and record-to-target cardinality. It links a controlled corpus with deterministic injections into files from fixed commits of ten public Python repositories; the injected targets are not naturally occurring violations. We evaluate one pinned version each of Ruff, Pylint, and core Flake8 on shared documented diagnostic families. Each tool detected and localized 80/80 controlled and 60/60 injected targets. Binary detection is expected to approach a ceiling for these simple, isolated constructs. No mapped target alarms occurred in the matched-clean controls; in the real-code layer, this is a control-integrity check conditional on host screening. Flake8 emitted two records per duplicate-key target, affecting 8/80 controlled and 10/60 real-code positives, whereas Ruff and Pylint emitted one. Thus, saturated detection coexists with differences in support and record-to-target cardinality. PyLinterBench provides a reproducible conformance baseline, with cross-version stability still to be evaluated.

Anahtar Kelimeler

Python linters, Static code analysis, Software benchmarking, Conformance testing, Diagnostic local- ization

Kaynakça

  1. V. Lenarduzzi, F. Pecorelli, N. Saarimäki, S. Lujan, and F. Palomba, 2023. “A critical comparison on six static analysis tools: Detection, agreement, and precision,” Journal of Systems and Software, 198, 111575, https://doi.org/10.1016/j.jss.2022.111575.
  2. Astral Software, “Rules,” Ruff documentation. Available: https://docs.astral.sh/ruff/rules/ (accessed: September 15, 2026).
  3. Pylint Developers, “Messages,” Pylint documentation. Available: https://pylint.readthedocs.io/en/latest/user_guide/messages/ (accessed: September 15, 2026).
  4. Python Code Quality Authority, “Error / Violation Codes,” Flake8 7.3.0 documentation. Available: https://flake8.pycqa.org/en/7.3.0/user/error-codes.html (accessed: September 15, 2026).
  5. Astral Software, “FAQ,” Ruff documentation. Available: https://docs.astral.sh/ruff/faq/ (accessed: September 15, 2026).
  6. H. Gulabovska and Z. Porkoláb, “Survey on Static Analysis Tools of Python Programs,” Proceedings of the Eighth Workshop on Software Quality Analysis, Monitoring, Improvement, and Applications, Ohrid, North Macedonia, 2019, CEUR Workshop Proceedings, 2508, 3:1–3:10, https://ceur-ws.org/Vol-2508/paper-gul.pdf.
  7. H. B. Hassan, Q. I. Sarhan, and Á. Beszédes, 2024. “Evaluating Python static code analysis tools using FAIR principles,” IEEE Access, 12, 173647–173659, https://doi.org/10.1109/ACCESS.2024.3503493.
  8. B. A. A. Ali, 2025. “Static analysis of code quality in open-source Python projects,” Journal of Undergraduate Research International, 1(2), 75–81, https://doi.org/10.64589/juri/215034.
  9. A. Eghbali, F. Burk, and M. Pradel, 2025. “DyLin: A dynamic linter for Python,” Proceedings of the ACM on Software Engineering, 2(FSE), 2828–2849, https://doi.org/10.1145/3729395.
  10. C. Klinger, M. Christakis, and V. Wüstholz, “Differentially testing soundness and precision of program analyzers,” Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, Beijing, China, 2019, 239–250, https://doi.org/10.1145/3293882.3330553.

Kaynak Göster

APA
Yılmaz, E. C., Oktaş, R., & Subaş, İ. (2026). PyLinterBench: A Support-Aware Benchmark for Python Linter Diagnostic Conformance in Controlled and Real-Code Contexts. OMÜ Mühendislik Bilimleri ve Teknolojisi Dergisi, 7, 105-125. https://izlik.org/JA48FE38SD
AMA
1.Yılmaz EC, Oktaş R, Subaş İ. PyLinterBench: A Support-Aware Benchmark for Python Linter Diagnostic Conformance in Controlled and Real-Code Contexts. OMUJEST. 2026;7:105-125. https://izlik.org/JA48FE38SD
Chicago
Yılmaz, Emre Can, Recai Oktaş, ve İrfan Subaş. 2026. “PyLinterBench: A Support-Aware Benchmark for Python Linter Diagnostic Conformance in Controlled and Real-Code Contexts”. OMÜ Mühendislik Bilimleri ve Teknolojisi Dergisi 7 (Ekim): 105-25. https://izlik.org/JA48FE38SD.
EndNote
Yılmaz EC, Oktaş R, Subaş İ (01 Ekim 2026) PyLinterBench: A Support-Aware Benchmark for Python Linter Diagnostic Conformance in Controlled and Real-Code Contexts. OMÜ Mühendislik Bilimleri ve Teknolojisi Dergisi 7 105–125.
IEEE
[1]E. C. Yılmaz, R. Oktaş, ve İ. Subaş, “PyLinterBench: A Support-Aware Benchmark for Python Linter Diagnostic Conformance in Controlled and Real-Code Contexts”, OMUJEST, c. 7, ss. 105–125, Eki. 2026, [çevrimiçi]. Erişim adresi: https://izlik.org/JA48FE38SD
ISNAD
Yılmaz, Emre Can - Oktaş, Recai - Subaş, İrfan. “PyLinterBench: A Support-Aware Benchmark for Python Linter Diagnostic Conformance in Controlled and Real-Code Contexts”. OMÜ Mühendislik Bilimleri ve Teknolojisi Dergisi 7 (01 Ekim 2026): 105-125. https://izlik.org/JA48FE38SD.
JAMA
1.Yılmaz EC, Oktaş R, Subaş İ. PyLinterBench: A Support-Aware Benchmark for Python Linter Diagnostic Conformance in Controlled and Real-Code Contexts. OMUJEST. 2026;7:105–125.
MLA
Yılmaz, Emre Can, vd. “PyLinterBench: A Support-Aware Benchmark for Python Linter Diagnostic Conformance in Controlled and Real-Code Contexts”. OMÜ Mühendislik Bilimleri ve Teknolojisi Dergisi, c. 7, Ekim 2026, ss. 105-2, https://izlik.org/JA48FE38SD.
Vancouver
1.Emre Can Yılmaz, Recai Oktaş, İrfan Subaş. PyLinterBench: A Support-Aware Benchmark for Python Linter Diagnostic Conformance in Controlled and Real-Code Contexts. OMUJEST [Internet]. 01 Ekim 2026;7:105-2. Erişim adresi: https://izlik.org/JA48FE38SD