Bitlis Eren Üniversitesi Fen Bilimleri Dergisi

2147-3129 2147-3188

Bitlis Eren University

10.17798/bitlisfen.1796956

Natural Language Processing

Doğal Dil İşleme

AI vs. Human Text Detection: A High-Accuracy Ensemble Approach Using Machine Learning

https://orcid.org/0000-0002-9864-2866

Kökver

Yunus

ANKARA UNIVERSITY, ELMADAĞ VOCATIONAL SCHOOL, DEPARTMENT OF COMPUTER TECHNOLOGIES

03 24 2026

15 1 245 258 10 04 2025 02 09 2026

2012

Bitlis Eren Üniversitesi Fen Bilimleri Dergisi

This study aims to develop and evaluate a machine learning (ML)-based classification model for distinguishing between texts generated by artificial intelligence (AI) and those written by humans. Utilizing a comprehensive dataset comprising 487235 text samples, various ML algorithms—including Multilayer Perceptron (MLP), Random Forest (RF), Gradient Boosting (GB), Logistic Regression (LR), Support Vector Machines (SVM), Decision Trees (DT), and an Ensemble Model—were trained and evaluated to classify AI-generated and human-generated texts. Ensemble Model, which combines the best-performing algorithms, achieved an accuracy rate of 99.90%, outperforming individual models. Additionally, the study presents a user-friendly interface that enables real-time classification of texts using the weights of the ensemble model. This interface holds potential as a practical tool for researchers and professionals in fields such as education, academia, and media. The model's generalization capability was also tested on a user-generated dataset through the user interface, and it was found to be consistent with the primary dataset, achieving an "Almost Perfect" level according to the Kappa statistic. This study highlights the necessity of robust tools to mitigate ethical and security risks associated with AI-generated content. Moreover, ensemble models show great promise in handling complex classification tasks.

Natural Language Processing Artificial Intelligence and Ethics Machine Learning Ensemble Models Text Classification

Y. Kökver, H. M. Pektaş, and H. Çelik, “Artificial intelligence applications in education: Natural language processing in detecting misconceptions,” Educ. Inf. Technol., pp. 1–32, Aug. 2024, doi: 10.1007/S10639-024-12919-1/FIGURES/2.

T. B. Brown et al., “Language Models are Few-Shot Learners,” Adv. Neural Inf. Process. Syst., vol. 2020-December, May 2020, Accessed: Oct. 17, 2024. [Online]. Available: https://arxiv.org/abs/2005.14165v4

J. Devlin, M.-W. Chang, K. Lee, K. T. Google, and A. I. Language, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” Proc. 2019 Conf. North, pp. 4171–4186, 2019, doi: 10.18653/V1/N19-1423.

C. D. Manning, “Introduction to information retrieval,” 2008, Cambridge university press.

H. Alamleh, A. A. S. Alqahtani, and A. Elsaid, “Distinguishing Human-Written and ChatGPT-Generated Text Using Machine Learning,” 2023 Syst. Inf. Eng. Des. Symp. SIEDS 2023, pp. 154–158, 2023, doi: 10.1109/SIEDS58326.2023.10137767.

M. Nour, B. Arabacı, H. Öcal, and K. Polat, “New approaches to epileptic seizure prediction based on EEG signals using hybrid CNNs,” Int. J. Intell. Eng. Informatics, vol. 12, no. 1, pp. 85–102, 2024, doi: 10.1504/IJIEI.2024.137706.

M. T. Zamir, M. A. Ayub, A. Gul, N. Ahmad, and K. Ahmad, “Stylometry Analysis of Multi-authored Documents for Authorship and Author Style Change Detection,” Jan. 2024, Accessed: Oct. 15, 2024. [Online]. Available: https://arxiv.org/abs/2401.06752v1

A. de Pablo, O. Araque, and C. A. Iglesias, “Radical Text Detection based on Stylometry,” in International Conference on Information Systems Security and Privacy, Science and Technology Publications, Lda, 2020, pp. 524–531. doi: 10.5220/0008971205240531.

A. Pascucci, R. Manna, C. Caterino, V. Masucci, and J. Monti, “Is this hotel review truthful or deceptive? A platform for disinformation detection through computational stylometry,” in Proceedings for the First International Workshop on Social Threats in Online Conversations: Understanding and Management, 2020, pp. 35–40.

H. El-Fiqi, E. Petraki, and H. A. Abbass, “A computational linguistic approach for the identification of translator stylometry using Arabic-English text,” IEEE Int. Conf. Fuzzy Syst., pp. 2039–2045, 2011, doi: 10.1109/FUZZY.2011.6007535.

T. Kumarage, J. Garland, A. Bhattacharjee, K. Trapeznikov, S. Ruston, and H. Liu, “Stylometric detection of ai-generated text in twitter timelines,” arXiv Prepr. arXiv2303.03697, 2023.

G. K. Mikros, A. Koursaris, D. Bilianos, and G. Markopoulos, “AI-Writing Detection Using an Ensemble of Transformers and Stylometric Features.,” in IberLEF@ SEPLN, 2023.

H. Wang, J. Li, and Z. Li, “AI-generated text detection and classification based on BERT deep learning algorithm,” Theor. Nat. Sci., vol. 39, no. 1, pp. 312–317, Jul. 2024, doi: 10.54254/2753-8818/39/20240625.

D. Trandabat and D. Gifu, “Discriminating AI-generated Fake News,” Procedia Comput. Sci., vol. 225, pp. 3822–3831, Jan. 2023, doi: 10.1016/J.PROCS.2023.10.378.

R. Kumar and M. Mindzak, “Who wrote this? Detecting artificial intelligence–generated text from human-written text,” Can. Perspect. Acad. Integr., vol. 7, no. 1, 2024.

A. Akram, “An empirical study of ai generated text detection tools,” arXiv Prepr. arXiv2310.01423, 2023, doi: https://doi.org/10.48550/arXiv.2310.01423.

R. Gaggar, A. Bhagchandani, and H. Oza, “Machine-generated text detection using deep learning,” arXiv Prepr. arXiv2311.15425, 2023.

Y. Mo, H. Qin, Y. Dong, Z. Zhu, and Z. Li, “Large Language Model (LLM) AI text generation detection based on transformer deep learning algorithm,” Apr. 2024, doi: 10.48550/arxiv.2405.06652.

Y. Zhang et al., “Enhancing Text Authenticity: A Novel Hybrid Approach for AI-Generated Text Detection,” Jun. 2024, Accessed: Oct. 15, 2024. [Online]. Available: https://arxiv.org/abs/2406.06558v1

A. Najee-Ullah, L. Landeros, Y. Balytskyi, and S. Y. Chang, “Towards Detection of AI-Generated Texts and Misinformation,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 13176 LNCS, pp. 194–205, 2022, doi: 10.1007/978-3-031-10183-0_10/FIGURES/4.

T. T. Nguyen, A. Hatua, and A. H. Sung, “How to Detect AI-Generated Texts?,” 2023 IEEE 14th Annu. Ubiquitous Comput. Electron. Mob. Commun. Conf. UEMCON 2023, pp. 464–471, 2023, doi: 10.1109/UEMCON59035.2023.10316132.

P. C. Theocharopoulos, P. Anagnostou, A. Tsoukala, S. V. Georgakopoulos, S. K. Tasoulis, and V. P. Plagianakos, “Detection of Fake Generated Scientific Abstracts,” Proc. - IEEE 9th Int. Conf. Big Data Comput. Serv. Appl. BigDataService 2023, pp. 33–39, 2023, doi: 10.1109/BIGDATASERVICE58306.2023.00011.

M. A. Quidwai, C. Li, and P. Dube, “Beyond black box ai-generated plagiarism detection: From sentence to document level,” arXiv Prepr. arXiv2306.08122, 2023.

Y. Zhang, T. Zhou, H. Qiao, and T. Li, “Ethical Issues in AI-Generated Texts: A Systematic Review and Analysis,” Int. J. Human–Computer Interact., pp. 1–28, 2025.

Z. Ji et al., “Survey of hallucination in natural language generation,” ACM Comput. Surv., vol. 55, no. 12, pp. 1–38, 2023.

R. M. Branch, “Instructional design: The ADDIE approach,” Instr. Des. ADDIE Approach, pp. 1–203, 2010, doi: 10.1007/978-0-387-09506-6/COVER.

O. Karamustafaoğlu and H. M. Pektaş, “Developing students’ creative problem solving skills with inquiry-based STEM activity in an out-of-school learning environment,” Educ. Inf. Technol., vol. 28, no. 6, pp. 7651–7669, Jun. 2023, doi: 10.1007/S10639-022-11496-5/TABLES/4.

T. Trust and E. Pektas, “Using the ADDIE Model and Universal Design for Learning Principles to Develop an Open Online Course for Teacher Professional Development,” J. Digit. Learn. Teach. Educ., vol. 34, no. 4, pp. 219–233, Oct. 2018, doi: 10.1080/21532974.2018.1494521.

U. Shafique and H. Qaiser, “A comparative study of data mining process models (KDD, CRISP-DM and SEMMA),” Int. J. Innov. Sci. Res., vol. 12, no. 1, pp. 217–222, 2014.

S. Gerami, “AI Vs Human Text.” Accessed: Oct. 17, 2024. [Online]. Available: https://www.kaggle.com/datasets/shanegerami/ai-vs-human-text

“Better language models and their implications | OpenAI.” Accessed: Dec. 22, 2024. [Online]. Available: https://openai.com/index/better-language-models/

F. Martinelli, F. Mercaldo, L. Petrillo, and A. Santone, “A Method for AI-generated sentence detection through Large Language Models,” Procedia Comput. Sci., vol. 246, no. C, pp. 4853–4862, Jan. 2024, doi: 10.1016/J.PROCS.2024.09.351.

“andythetechnerd03/AI-human-text · Datasets at Hugging Face.” Accessed: Jan. 03, 2025. [Online]. Available: https://huggingface.co/datasets/andythetechnerd03/AI-human-text

W. Y. Wang, “‘Liar, Liar Pants on Fire’: A New Benchmark Dataset for Fake News Detection,” ACL 2017 - 55th Annu. Meet. Assoc. Comput. Linguist. Proc. Conf. (Long Pap., vol. 2, pp. 422–426, 2017, doi: 10.18653/V1/P17-2067.

“Data Science, Machine Learning, AI & Analytics - KDnuggets.” Accessed: Jan. 03, 2025. [Online]. Available: https://www.kdnuggets.com/