Comprehensive AI assessment framework: Enhancing educational evaluation with ethical AI integration

Year 2024, Volume: 7 Issue: 4 - ICETOL 2024 Special Issue, 521 - 540, 31.12.2024

Selçuk Kılınç

https://doi.org/10.31681/jetol.1492695

Abstract

The integration of generative artificial intelligence (GenAI) tools into education has been a game-changer for teaching and assessment practices, bringing new opportunities, but also novel challenges which need to be dealt with. This paper presents the Comprehensive AI Assessment Framework (CAIAF), an evolved version of the AI Assessment Scale (AIAS) by Perkins, Furze, Roe, and MacVaugh, targeted toward the ethical integration of AI into educational assessments. This is where the CAIAF differs, as it incorporates stringent ethical guidelines, with clear distinctions based on educational levels, and advanced AI capabilities of real-time interactions and personalized assistance. The framework developed herein has a very intuitive use, mainly through the use of a color gradient that enhances the user-friendliness of the framework. Methodologically, the framework has been developed through the huge support of a thorough literature review and practical insight into the topic, becoming a dynamic tool to be used in different educational settings. The framework will ensure better learning outcomes, uphold academic integrity, and promote responsible use of AI, hence the need for this framework in modern educational practice.

Keywords

Comprehensive AI Framework, AI Assessment, AI Integration, AI Ethics, AI in Education

References

• Adıgüzel, T., Kaya, M., & Cansu, F. (2023). Revolutionizing education with AI: Exploring the transformative potential of ChatGPT. Contemporary Educational Technology, 15(3), ep429. https://doi.org/10.30935/cedtech/13152
• Badeni, B., & Saparahayuningsih, S. (2019). Who is responsible for the child’s moral character education? Education Quarterly Reviews, 2(1), 23-32. https://doi.org/10.31014/aior.1993.02.01.35
• Borenstein, J., & Howard, A. (2021). Emerging challenges in AI and the need for AI ethics education. AI Ethics, 1, 61-65. https://doi.org/10.1007/s43681-020-00002-7
• Boscardin, C. (2024). ChatGPT and generative artificial intelligence for medical education: Potential impact and opportunity. Academic Medicine, 99(1), 22-27. https://doi.org/10.1097/acm.0000000000005439
• Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., … & Zhang, Y. (2023). Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv preprinarXiv:2303.12712.
• Chakraborty, S., Bedi, A. S., Zhu, S., An, B., Manocha, D., & Huang, F. (2023). On the possibilities of ai-generated text detection. arXiv preprint arXiv:2304.04736. https://doi.org/10.48550/arXiv.2304.04736
• Chan, C. K. Y., & Tsi, L. H. (2023). The AI revolution in education: Will AI replace or assist teachers in higher education? arXiv preprint arXiv:2305.01185.
• Chang, D. H., Lin, M. P. C., Hajian, S., & Wang, Q. Q. (2023). Educational design principles of using AI chatbot that supports self-regulated learning in education: Goal setting, feedback, and personalization. Sustainability, 15(17), 12921. https://doi.org/10.3390/su151712921
• De Laat, P. (2021). Companies committed to responsible AI: From principles towards implementation and regulation? Philosophy & Technology, 34(4), 1135-1193. https://doi.org/10.1007/s13347-021-00474-3
• Dickey, E., & Bejarano, A. (2023). A model for integrating generative AI into course content development. arXiv preprint arXiv:2308.12276.
• Elliot, A. J., & Maier, M. A. (2014). Color psychology: Effects of perceiving color on psychological functioning in humans. Annual Review of Psychology, 65, 95-120. https://doi.org/10.1146/annurev-psych-010213-115035
• Foray, D., & Raffo, J. (2012). Business-driven innovation: Is it making a difference in education? An analysis of educational patents. http://dx.doi.org/10.1787/5k91dl7pc835-en
• Frankel, F., & DePace, A. H. (2012). Visual strategies: A practical guide to graphics for scientists & engineers. Yale University Press.
• Gampala, S., Vankeshwaram, V., & Gadula, S. S. P. (2020). Is artificial intelligence the new friend for radiologists? A review article. Cureus, 12(10). https://doi.org/10.7759/cureus.11137
• Gao, X. (2024). Language education in a brave new world: A dialectical imagination. Modern Language Journal, 108(2), 556-562. https://doi.org/10.1111/modl.12930
• García-Martínez, I., Fernández-Batanero, J. M., Fernández-Cerero, J., & León, S. P. (2023). Analysing the impact of artificial intelligence and computational sciences on student performance: Systematic review and meta-analysis. Journal of New Approaches in Educational Research, 12(1), 171-197. https://doi.org/10.7821/naer.2023.1.1240
• Garg, S., & Sharma, S. (2020). Impact of artificial intelligence in special need education to promote inclusive pedagogy. International Journal of Information and Education Technology, 10(7), 523-527. https://doi.org/10.18178/ijiet.2020.10.7.1418
• Gillani, N., Eynon, R., Chiabaut, C., & Finkel, K. (2023). Unpacking the “Black Box” of AI in education. Educational Technology & Society, 26(1), 99-111. https://doi.org/10.30191/ETS.202301_26(1).0008
• Green, C., Mynhier, L., Banfill, J., Edwards, P., Kim, J., & Desjardins, R. (2020). Preparing education for the crises of tomorrow: A framework for adaptability. International Review of Education, 66, 857-879. https://doi.org/10.1007/s11159-020-09878-3
• Hamiti, M., Reka, B., & Baloghová, A. (2014). Ethical use of information technology in high education. Procedia-Social and Behavioral Sciences, 116, 4411-4415. https://doi.org/10.1016/j.sbspro.2014.01.957
• Holmes, W., Porayska-Pomsta, K., Holstein, K., Sutherland, E., Baker, T., Shum, S. B., … & Koedinger, K. R. (2022). Ethics of AI in education: Towards a community-wide framework. International Journal of Artificial Intelligence in Education, 1-23. https://doi.org/10.1007/s40593-021-00239-1
• Hong, Y., Nguyen, A., Dang, B., & Nguyen, B. P. T. (2022, July). Data ethics framework for artificial intelligence in education (AIED). In 2022 International Conference on Advanced Learning Technologies (ICALT) (pp. 297-301). IEEE. https://doi.org/10.1109/ICALT55010.2022.00095
• Jones, S. (2016). Doing the right thing: Computer ethics pedagogy revisited. Journal of Information, Communication and Ethics in Society, 14(1), 33-48. https://doi.org/10.1108/JICES-07-2014-0033
• Kamalov, F., & Gurrib, I. (2023). A new era of artificial intelligence in education: A multifaceted revolution. arXiv preprint arXiv:2305.18303.
• Kitsios, F., & Kamariotou, M. (2021). Artificial intelligence and business strategy towards digital transformation: A research agenda. Sustainability, 13(4), 2025. https://doi.org/10.3390/su13042025
• Kılınç, S. (2023). Embracing the future of distance science education: Opportunities and challenges of ChatGPT integration. Asian Journal of Distance Education, 18(1), 205-237. https://doi.org/10.5281/zenodo.7857396
• Klimova, B., Pikhart, M., & Kacetl, J. (2023). Ethical issues of the use of AI-driven mobile apps for education. Frontiers in Public Health. https://doi.org/10.3389/fpubh.2022.1118116
• Kulik, J. A., & Fletcher, J. D. (2016). Effectiveness of intelligent tutoring systems: a meta-analytic review. Review of educational research, 86(1), 42-78. https://doi.org/10.3102/0034654315581420
• Lam, T., Cheung, M., Munro, Y., Lim, K., Shung, D., & Sung, J. (2022). Randomized controlled trials of artificial intelligence in clinical practice: Systematic review. Journal of Medical Internet Research, 24(8), e37188. https://doi.org/10.2196/37188
• Lane, S. H., Haley, T., & Brackney, D. E. (2024). Tool or tyrant: Guiding and guarding generative artificial intelligence use in nursing education. Creative Nursing, 30(2), 125-132. https://doi.org/10.1177/10784535241247094
• Leimanis, A. (2020, February). Self-imposed ethical guidelines for AI in education. In International Scientific Conference “SOCIETY, INTEGRATION, EDUCATION-SIE2020”.
• Luckin, R., & Holmes, W. (2016). Intelligence unleashed: An argument for AI in education.
• Ma, X., & Jiang, C. (2023). On the ethical risks of artificial intelligence applications in education and its avoidance strategies. Journal of Education, Humanities and Social Sciences, 14, 354-359. https://doi.org/10.54097/ehss.v14i.8868
• Maghsudi, S., Lan, A., Xu, J., & van Der Schaar, M. (2021). Personalized education in the artificial intelligence era: What to expect next. IEEE Signal Processing Magazine, 38(3), 37-50. https://doi.org/10.1109/MSP.2021.3055032
• Mahligawati, F., Allanas, E., Butarbutar, M. H., & Nordin, N. A. N. (2023, September). Artificial intelligence in physics education: A comprehensive literature review. In Journal of Physics: Conference Series (Vol. 2596, No. 1, p. 012080). IOP Publishing. https://doi.org/10.1088/1742-6596/2596/1/012080
• Mâță, L. (2022). Ethical rules of online communication between university teachers and students. In: Mâță, L. (Ed.) Ethical use of information technology in higher education. EAI/Springer Innovations in Communication and Computing. Springer, Singapore. https://doi.org/10.1007/978-981-16-1951-9_7
• Mayer, J. (2009). The growing interdependence between financial and commodity markets (No. 195). United Nations Conference on Trade and Development.
• Mello, R. F., Freitas, E., Pereira, F. D., Cabral, L., Tedesco, P., & Ramalho, G. (2023). Education in the age of generative AI: Context and recent developments. arXiv preprint arXiv:2309.12332.
• Michaeli, T., Romeike, R., & Seegerer, S. (2022, August). What students can learn about artificial intelligence–recommendations for K-12 computing education. In IFIP World Conference on Computers in Education (pp. 196-208). Cham: Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-43393-1_19
• Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers college record, 108(6), 1017-1054. https://doi.org/10.1111/j.1467-9620.2006.00684.x
• Mollick, E. R., & Mollick, L. (2023a). Using AI to implement effective teaching strategies in classrooms: Five strategies, including prompts. Including Prompts (March 17, 2023). http://dx.doi.org/10.2139/ssrn.4391243
• Mollick, E., & Mollick, L. (2023b). Assigning AI: Seven approaches for students, with prompts. arXiv preprint arXiv:2306.10052.
• Morley, J., Elhalal, A., Garcia, F., Kinsey, L., Mökander, J., & Floridi, L. (2021). Ethics as a service: A pragmatic operationalisation of AI ethics. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.3784238
• Network, P. O. D. (2001). Ethical guidelines for educational developers. To Improve the Academy, 19.
• Ng, P. T. (2009). Innovation in education: Some observations and questions. International Journal of Innovation in Education, 1(1), 8-11. https://doi.org/10.1504/IJIIE.2009.0301
• Ng, D. T. K., Leung, J. K. L., Su, J., Ng, R. C. W., & Chu, S. K. W. (2023). Teachers’ AI digital competencies and twenty-first century skills in the post-pandemic world. Educational technology research and development, 71(1), 137-161. https://doi.org/10.1007/s11423-023-10203-6
• Nguyen, A., Ngo, H. N., Hong, Y., Dang, B., & Nguyen, B. P. T. (2023). Ethical principles for artificial intelligence in education. Education and Information Technologies, 28(4), 4221-4241. https://doi.org/10.1007/s10639-022-11316-w
• Ogunleye, B., Zakariyyah, K. I., Ajao, O., Olayinka, O., & Sharma, H. (2024). Higher education assessment practice in the era of generative AI tools. arXiv preprint arXiv:2404.01036.
• Olga, A., Saini, A., Zapata, G., Searsmith, D., Cope, B., Kalantzis, M., … & Kastania, N. P. (2023). Generative AI: Implications and applications for education. arXiv preprint arXiv:2305.07605.
• Pack, A., & Maloney, J. (2024). Using artificial intelligence in TESOL: Some ethical and pedagogical considerations. TESOL Quarterly, 58(2), 1007-1018. https://doi.org/10.1002/tesq.3320
• Park, W., & Kwon, H. (2024). Implementing artificial intelligence education for middle school technology education in Republic of Korea. International Journal of Technology and Design Education, 34(1), 109-135. https://doi.org/10.1007/s10798-023-09812-2
• Pasricha, S. (2023, June). Ethics in computing education: Challenges and experience with embedded ethics. In Proceedings of the Great Lakes Symposium on VLSI 2023 (pp. 653-658). https://doi.org/10.1145/3583781.3590240
• Perkins, M., Furze, L., Roe, J., & MacVaugh, J. (2024). The artificial intelligence assessment scale (AIAS): A framework for ethical integration of generative AI in educational assessment. Journal of University Teaching and Learning Practice, 21(06). https://doi.org/10.53761/q3azde36
• Perkins, M., Roe, J., Vu, B. H., Postma, D., Hickerson, D., McGaughran, J., & Khuat, H. Q. (2024). GenAI detection tools, adversarial techniques and implications for inclusivity in higher education. arXiv preprint arXiv:2403.19148.
• Poza-Lujan, J. L., Calafate, C. T., Posadas-Yagüe, J. L., & Cano, J. C. (2015). Assessing the impact of continuous evaluation strategies: Tradeoff between student performance and instructor effort. IEEE Transactions on Education, 59(1), 17-23. https://doi.org/10.1109/TE.2015.2418740
• Pushpa, S. (2012). Ethical leadership: Need for business ethics education. International Journal of Advances in Management and Economics, 1(1), 14-19.
• Puyo, J. G. B. (2021). A value and character educational model: Repercussions for students, teachers, and families. Journal of Culture and Values in Education, 4(1), 100-115. https://doi.org/10.46303/jcve.2020.7
• Rafikov, I., Akhmetova, E., & Yapar, O. E. (2021). Prospects of morality-based education in the 21st century. Journal of Islamic Thought and Civilization, 11(1), 1-21. https://doi.org/10.32350/jitc.111.01
• Sajja, R., Sermet, Y., Cikmaz, M., Cwiertny, D., & Demir, I. (2023). Artificial intelligence-enabled intelligent assistant for personalized and adaptive learning in higher education. arXiv preprint arXiv:2309.10892.
• Selwyn, N. (2013). Distrusting educational technology: Critical questions for changing times. Routledge.
• Shih, P. K., Lin, C. H., Wu, L. Y., & Yu, C. C. (2021). Learning ethics in AI—teaching non-engineering undergraduates through situated learning. Sustainability, 13(7), 3718. https://doi.org/10.3390/su13073718
• Simms, R. C. (2024). Work with ChatGPT, not against: 3 teaching strategies that harness the power of artificial intelligence. Nurse Educator, 49(3), 158-161. https://doi.org/10.1097/nne.0000000000001634
• Singh, S., & Riedel, S. (2016). Creating interactive and visual educational resources for AI. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1). https://doi.org/10.1609/aaai.v30i1.9851
• Singh, S. V., & Hiran, K. K. (2022). The impact of AI on teaching and learning in higher education technology. Journal of Higher Education Theory and Practice, 22(13).
• Stobart, G. (2004). Developing and improving assessment instruments.
• Sullivan, M., Kelly, A., & McLaughlan, P. (2023). ChatGPT in higher education: Considerations for academic integrity and student learning. https://doi.org/10.37074/jalt.2023.6.1.17
• Swiecki, Z., Khosravi, H., Chen, G., Martinez-Maldonado, R., Lodge, J. M., Milligan, S., … & Gašević, D. (2022). Assessment in the age of artificial intelligence. Computers and Education: Artificial Intelligence, 3, 100075. https://doi.org/10.1016/j.caeai.2022.100075
• Tong, R., Li, H., Liang, J., & Wen, Q. (2024). Developing and Deploying Industry Standards for Artificial Intelligence in Education (AIED): Challenges, Strategies, and Future Directions. arXiv preprint arXiv:2403.14689.
• Volante, L., DeLuca, C., & Klinger, D. A. (2023). Leveraging AI to enhance learning. Phi Delta Kappan, 105(1), 40-45. https://doi.org/10.1177/00317217231197475
• Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes (Vol. 86). Harvard university press.
• Wang, Z., Chen, A., Tao, K., Han, Y., & Li, J. (2024). MatGPT: A vane of materials informatics from past, present, to future. Advanced Materials, 36(6). https://doi.org/10.1002/adma.202306733
• Wang, Z., & Zhai, J. (2019, October). Ethical challenges faced by students in the educational environment of artificial intelligence. In 2019 International Conference on Advanced Education Research and Modern Teaching (AERMT 2019) (pp. 1-3). Atlantis Press. https://doi.org/10.2991/aermt-19.2019.1
• Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T., Guerrero-Dib, J., Popoola, O., … & Waddington, L. (2023). Testing of detection tools for AI-generated text. International Journal for Educational Integrity, 19(1), 26. https://doi.org/10.1007/s40979-023-00146-z
• Xu, X., Zhang, J., Zhu, Q., & Xia, T. (2023). The influences of gradient color on the weight perception and stability perception: A preliminary study. i-Perception, 14(4), 20416695231197797. https://doi.org/10.1177/20416695231197797
• Yen, W. M., Lall, V. F., & Monfils, L. (2012). Evaluating academic progress without a vertical scale. ETS Research Report Series, 2012(1), i-55. https://doi.org/10.1002/j.2333-8504.2012.tb02289.x
• Yeo, K. (2023). Artificial intelligence in cardiology: Did it take off? Russian Journal for Personalized Medicine, 2(6), 16-22. https://doi.org/10.18705/2782-3806-2022-2-6-16-22
• Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education–where are the educators?. International Journal of Educational Technology in Higher Education, 16(1), 1-27. https://doi.org/10.1186/s41239-019-0171-0
• Zeileis, A., Hornik, K., & Murrell, P. (2009). Escaping RGBland: Selecting colors for statistical graphics. Computational Statistics & Data Analysis, 53(9), 3259-3270. https://doi.org/10.1016/j.csda.2008.11.033
• Zhang, C., Zhang, C., Li, C., Qiao, Y., Zheng, S., Dam, S. K., … & Hong, C. S. (2023). One small step for generative AI, one giant leap for AGI: A complete survey on ChatGPT in AIGC era. arXiv preprint arXiv:2304.06488.
• Zhang, K., & Aslan, A. B. (2021). AI technologies for education: Recent research & future directions. Computers and Education: Artificial Intelligence, 2. https://doi.org/10.1016/j.caeai.2021.100025
• Zhou, X., Van Brummelen, J., & Lin, P. (2020). Designing AI learning experiences for K-12: Emerging works, future opportunities and a design framework. arXiv preprint arXiv:2009.10228.