E-Sınavlardan Etkin Geri Bildirim Üretmek için Öneriler

Yıl 2020, Cilt: 35 Sayı: Special Issue, 60 - 72, 30.09.2020

Okan Bulut Maria Cutumısu Deepak Sıngh Alexandra M. Aquılına

Öz

Günümüzün eğitim sistemleri öğrencilerin öğrenme sürecini desteklemek amacıyla birçok yeni teknolojiyi uyarlamaya ve kullanmaya devam etmektedir. Bu teknolojilerden birisi de e-sınavlardır. Bu tarz sınavlar öğrencilerin sınav sorularını bilgisayar ya da tablet gibi dijital araçlar ile cevaplamasına imkân vermektedir. E-sınavların önemli faydalarından biri de etkili, hızlı ve bireye özgü geri bildirim üretmeye imkân sağlamasıdır. Bununla beraber e-sınavlardan etkin olarak geri bildirim üretilmesine dair önerilerin yer aldığı çalışmalara literatürde rastlanılmamıştır. Literatürdeki bu önemli boşluğu doldurmak adına e-sınavlarda geri bildirim üretimine dair sistematik bir derleme çalışması yapılmıştır. Bu çalışma üç aşamada gerçekleştirilmiştir. İlk aşamada geri bildirim üretimine dair uygulamaları içeren çalışmalar derlenmiş ve önemli noktalar özetlenmiştir. İkinci aşamada ise e-sınavlardan etkin geribildirim üretmek için araştırmacı ve uygulayıcılara yönelik öneriler sunulmuştur. Son aşamada ise e-sınavlardan hızlı ve bireye özgü geri bildirim üretiminde kullanılabilecek altı adımdan oluşan bir yönerge ortaya konmuştur. Bu yönergede görsel ve yazılı öğelerden oluşan geri bildirimler aracılığıyla öğrencilerin sınav sonuçlarını nasıl daha etkin bir şekilde yorumlayabileceği gösterilmiştir.

Anahtar Kelimeler

E-sınav, geri bildirim, sınav raporu, sınav geliştirme, bilgisayar destekli sınav

Guidelines for Generating Effective Feedback from E-Assessments

Öz

Today’s education systems continue to adopt new technologies to support student learning. One of these technologies is e-assessment, a form of assessment that enables students to answer items using digital devices, such as computers and tablets. One of the benefits of e-assessments is the ability to generate interactive, timely, and customized feedback for students. Yet, despite vast literature on the generation and delivery of feedback, there is no systematic review of the guidelines on how e-assessments can be used for generating effective feedback. The objectives of this study are threefold. First, we synthesize the literature on the current practices in feedback generation. Second, we provide researchers and practitioners with a synthesis of guidelines for best practices in generating effective feedback with e-assessments. Third, we introduce a new framework in which we demonstrate the six steps of creating an e-assessment that can help produce immediate, customized, and specific feedback for students. This framework combines multiple forms of feedback (e.g., graphs, tables, and text) to improve the understanding of feedback and engage students in the interpretation of their score reports. Implications for practice and future research are discussed.

Anahtar Kelimeler

E-assessment, feedback, score reporting, test development, computerized assessment

Kaynakça

• Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruikshank, K. A., Mayer, R. E., Pintrich, P. R., & Wittrock, M. C. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. New York, NY: Pearson: Allyn & Bacon.
• Bailey, R., & Garner, M. (2010). Is the feedback in higher education assessment worth the paper it is written on? Teachers’ reflections on their practices. Teaching in Higher Education, 15(2), 187–198.
• Beaumont, C., O’Doherty, M., & Shannon, L. (2011). Reconceptualising assessment feedback: a key to improving student learning? Studies in Higher Education, 36(6), 671–687.
• Black, P. & Wiliam, D. (1998) Assessment and classroom learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7–74.
• Blair, A., Curtis, S., Goodwin, M., & Shields, S. (2013). What feedback do students want? Politics, 33(1), 66–79.
• Bloxham, S. (2015). Assessing assessment: New developments in assessment design, feedback practices and marking in higher education. In H. Fry, S. Ketteridge, & S. Marshall (Eds.), A handbook for teaching and learning in higher education (4th ed., pp. 107–122). Abingdon: Routledge.
• Budge, K. (2011). A desire for the personal: student perceptions of electronic feedback. International Journal of Teaching and Learning in Higher Education, 23(3), 342–349.
• Bulut, O., Cutumisu, M., Aquilina, A. M., & Singh, D. (2019). Effects of digital score reporting and feedback on students’ learning in higher education. Frontiers in Education, 4(65), 1–16. doi: 10.3389/feduc.2019.00065
• Bulut, O., Cutumisu, M., Singh, D., & Aquilina, A. (2018, July). Guidelines for generating effective feedback from computer-based assessments. Paper presented at the International Testing Commission Conference, Montreal, QC, Canada.
• Bulut, O., Davison, M. L., & Rodriguez, M. C. (2017). Estimating between-person and within-person subscore reliability with profile analysis. Multivariate Behavioral Research, 52(1), 86–104. doi: 10.1080/00273171.2016.1253452
• Butler, A. C., Karpicke, J. D., & Roediger III, H. L. (2007). The effect of type and timing of feedback on learning from multiple-choice tests. Journal of Experimental Psychology: Applied, 13(4), 273–281.
• Carless, D., Salter, D., Yang, M., & Lam, J. (2011). Developing sustainable feedback practices. Studies in Higher Education, 36(4), 395–407.
• Clariana, R. B., Wagner, D., & Murphy, L. C. R. (2000). Applying a connectionist description of feedback timing. Educational Technology Research and Development, 48, 5–22.
• Daniels, L. M., & Bulut, O. (2019). Students’ perceived usefulness of computerized percentage-only vs. descriptive score reports: Associations with motivation and grades. Journal of Computer Assisted Learning, 36(2), 199–208. doi: 10.1111/jcal.12398
• Desrochers, C. G., & Zell, D. (2012). Provided timely and frequent feedback on tests, reports, projects, etc., to help students improve. IDEA Paper No. 17. Manhattan, KS: The IDEA Center.
• Evans, C. (2013). Making sense of assessment feedback in higher education. Review of Educational Research, 83(1), 70–120.
• Epstein, M. L., Lazarus A. D., Calvano, T. B., Matthews, K. A., Hendel, R. A., Epstein, B. B., & Brosvic, G. M. (2002). Immediate feedback assessment technique promotes learning and corrects inaccurate first responses. The Psychological Record, 52, 187–201.
• Fluckiger, J., Vigil, Y. T. y., Pasco, R., & Danielson, K. (2010). Formative feedback: Involving students as partners in assessment to enhance learning. College Teaching, 58(4), 136–140.
• Fui, C. S., & Lian, L. H. (2018). The effect of computerized feedback on students’ misconceptions in algebraic expression. Pertanika Journal of Social Sciences and Humanities, 26(3), 1387–1403.
• Goodman, D. P., & Hambleton, R. K. (2004). Student test score reports and interpretive guides: Review of current practices and suggestions for future research. Applied Measurement in Education, 17(2), 145–220.
• Gotch, C. M., & Roduta Roberts, M. (2018). Score reporting. In B. B. Frey (Ed.), The SAGE encyclopedia of educational research, measurement, and evaluation. Thousand Oaks, CA: Sage.
• Gutierrez, F., & Atkinson, J. (2011). Adaptive feedback selection for intelligent tutoring systems. Expert Systems with Applications, 38(5), 6146–6152.
• Haladyna, T. M., & Rodriguez, M. C. (2013). Developing and validating test items. New York, NY: Taylor & Francis.
• Hambleton, R. K., & Zenisky, A. L. (2013). Reporting test scores in more meaningful ways: A research-based approach to score report design. In K. F. Geisinger, B. A. Bracken, J. F. Carlson, J. I. C. Hansen, N. R. Kuncel, S. P. Reise, & M. C. Rodriguez (Eds.), APA handbook of testing and assessment in psychology, Vol. 3: Testing and assessment in school psychology and education (pp. 479–494). Washington, DC: American Psychological Association.
• Harks, B., Rakoczy, K., Hattie, J., Besser, M., & Klieme, E. (2014). The effects of feedback on achievement, interest and self-evaluation: the role of feedback’s perceived usefulness. Educational Psychology, 34(3), 269–290.
• Hattie, J. (2009, April). Visibly learning from reports: The validity of score reports. Paper presented at the meeting of the National Council on Measurement in Education, San Diego, CA.
• Hattie, J., & Gan, M. (2011). Instruction based on feedback. In P. Alexander & R. E. Mayer (Eds.), Handbook of research on learning and instruction (pp. 249–271). New York, NY: Routledge.
• Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112.
• Hatziapostolou, T., & Paraskakis, I. (2010). Enhancing the impact of formative feedback on student learning through an online feedback system. Electronic Journal of E-Learning, 8(2), 111–122.
• Hepplestone, S., Holden, G., Irwin, B., Parkin, H. J., & Thorpe, L. (2011). Using technology to encourage student engagement with feedback: a literature review. ALT‐J: Research in Learning Technology,19(2), 117–127.
• Jonsson, A. (2012). Facilitating productive use of feedback in higher education. Active Learning in Higher Education, 14(1), 63–76.
• Jug, R., Jiang, X., & Bean, S. M. (2019). Giving and receiving effective feedback: A review article and how-to guide. Archives of Pathology & Laboratory Medicine, 143(2), 244–250.
• Kulkarni, C. E., Bernstein, M. S., & Klemmer, S. R. (2015). PeerStudio: Rapid peer feedback emphasizes revision and improves performance. Vancouver, Canada: ACM.
• Kyllonen, P. (2009). New constructs, methods, and directions for computer-based assessment. In F. Sheuermann & J. Björnsson (Eds.), The transition to computer-based assessment. New approaches to skills assessment and implications for large-scale testing (pp. 151-156). Luxemburg: Office for Official Publications of the European Communities.
• Lea, M. R., & Street, B. V. (1998). Student writing in higher education: An academic literacies approach. Studies in Higher Education, 23(2), 157–172.
• Lepper, M. R., Woolverton, M., Mumme, D. L., & Gurtner, J. (1993). Motivational techniques of expert human tutors: Lessons for the design of computer-based tutors. In S. P. Lajoie & S. J. Derry (Eds.), Computers as cognitive tools (pp. 75–106). Hillsdale, NJ: Lawrence Erlbaum Associates.
• Lilley, M. & Barker, T. (2007). Students’ perceived usefulness of formative feedback for a computer-adapted test. The Electronic Journal of e-Learning, 5(1), 31–38.
• Lipnevich, A., & Smith, J. (2008). Response to assessment feedback: The effect of grades, praise, and source of information (ETS RR-08-30). Princeton, NJ: ETS.
• Lopez, L. (2009). Effects of delayed and immediate feedback in the computer-based testing environment [Electronic version]. Doctoral dissertation, Department of Curriculum, Instructional, and Media Technology, Indiana State University.
• Lowry, R. (2005). Computer aided self assessment – an effective tool. Chemistry Education Research and Practice, 6(4), 198–203.
• Maier, U., Wolf, N., & Randler, C. (2016). Effects of a computer-assisted formative assessment intervention based on multiple-tier diagnostic items and different feedback types. Computers & Education, 95, 85–98.
• McCarthy, J. (2017). Enhancing feedback in higher education: Students’ attitudes towards online and in-class formative assessment feedback models. Active Learning in Higher Education, 18(2), 127–141.
• Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). New York: American Council on Education.
• Moridis, C. N., & Economides, A. A. (2012). Applause as an achievement-based reward during a computerized self-assessment test. British Journal of Educational Technology, 43(3), 489-504.
• Moss, P. A. (1995). Themes and variations in validity theory. Educational Measurement: Issues and Practice, 14, 5–13.
• Narciss, S. (2008). Feedback strategies for interactive learning tasks. In J. M. Spector, M. D. Merrill, J. van Merrienboer & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (pp. 125–143). New York, NY: Lawrence Erlbaum Associates.
• Nicol, D. (2007). E‐assessment by design: Using multiple‐choice tests to good effect. Journal of Further and Higher Education, 31(1), 53–64.
• Nicol, D., & Macfarlane‐Dick, D. (2006). Formative assessment and self‐regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199–216.
• Pendleton, D., Schofield, T., Tate, P., & Havelock, P. (2003). The new consultation: Developing doctor–patient communication (2nd ed). Oxford, UK: Oxford University Press.
• Perie, M., & Huff, K. (2015). Determining content and cognitive demand for achievement tests. In S. Lane, M. R. Raymond, & T. Haladyna (Eds.), Handbook of test development (2nd ed., pp. 119–143), Routledge, UK: London.
• Prince, M., Handley, K., Millar, J., & O’Donovan, B. (2010). Feedback: all that effort, but what is the effect? Assessment & Evaluation in Higher Education, 35(3), 277–289.
• Quinton, S., & Smallbone, T. (2010). Feeding forward: Using feedback to promote student reflection and learning: A teaching model. Innovations in Education and Teaching International, 47, 125–135.
• Race, P. (2001). Using feedback to help students to learn. The Higher Education Academy. Retrieved from https://phil-race.co.uk/wp-content/uploads/Using_feedback.pdf
• Race, P. (2006) The lecturer’s toolkit: A practical guide to assessment, learning and teaching, (3rd ed.). London: Routledge.
• Schartel, S. A. (2012). Giving feedback–An integral part of education. Best Practice & Research Clinical Anaesthesiology, 26(1), 77–87.
• Schwartz, D. L., Tsang, J. M., & Blair, K. P. (2016). The ABCs of how we learn: 26 scientifically proven approaches, how they work, and when to use them. New York, NY: W.W. Norton & Company.
• Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189.
• Singh, U. G., & Villiers, M. R. (2017). An evaluation framework and instrument for evaluating e-assessment tools. International Review of Research in Open and Distributed Learning, 18(6), 165–185.
• Sinharay, S. (2010). How often do subscores have added value? Results from operational and simulated data. Journal of Educational Measurement, 47(2), 150–174.
• Slater, S., Livingston, S. A., & Silver, M. (2019). Score reports for large-scale testing programs. Managing the design process. In D. Zapata-Rivera (Ed.), Score reporting research and applications (pp. 91–106). New York, NY: Routledge.
• Suen, H. (2014). Peer assessment for massive open online courses (MOOCs). The International Review of Research in Open and Distributed Learning, 15(3), 312–327.
• Thurlings, M., Vermeulen, M., Bastiaens, T., & Stijnen, S. (2013). Understanding feedback: A learning theory perspective. Educational Research Review, 9, 1–15.
• Timmers, C. F. (2013). Computer-based formative assessment: Variables influencing feedback behaviour (Unpublished doctoral dissertation). University of Twente, Enschede, the Netherlands.
• Timmers, C. F., & Veldkamp, B. P. (2011). Attention paid to feedback provided by a computer-based assessment for learning on information literacy. Computers & Education, 56, 923–930.
• Van der Kleij, F. M., Eggen, T. J. H. M., Timmers, C. F., & Veldkamp, B. P. (2012). Effects of feedback in a computer-based assessment for learning. Computers & Education, 58, 263–272.
• Van der Kleij, F. M., Feskens, R. C., & Eggen, T. J. (2015). Effects of feedback in a computer-based learning environment on students’ learning outcomes a meta-analysis. Review of Educational Research, 85(4), 475–511.
• Walker, D. J., Topping, K., & Rodrigues, S. (2008). Student reflections on formative e-assessment: Expectations and perceptions. Learning, Media and Technology, 33(3), 221–234.
• Weedon, E. (2000). Do you read this the way I read this? British Journal of Educational Technology, 31, 185–197.
• Wiggins, G. (1998). Educative assessment: Designing assessments to inform and improve student performance. San Francisco, CA: Jossey-Bass.
• Wu, H. M., Kuo, B. C., & Wang, S. C. (2017). Computerized dynamic adaptive tests with immediately individualized feedback for primary school mathematics learning. Educational Technology & Society, 20(1), 61–72.
• Wuensch, K. L., Aziz, S., Ozan, E., Kishore, M., & Tabrizi, M. H. N. (2008). Pedagogical characteristics of online and face-to-face classes. International Journal on E-Learning, 7(3), 523–532.
• Yuan, J., & Kim, C. (2015). Effective feedback design using free technologies. Journal of Educational Computing Research, 52(3), 408–434.
• Zapata-Rivera, D., & Katz, I. R. (2014). Keeping your audience in mind: Applying audience analysis to the design of interactive score reports. Assessment in Education: Principles, Policy & Practice, 21, 442–463.
• Zenisky, A. L., & Hambleton, R. K. (2012). Developing test score reports that work: The process and best practices for effective communication. Educational Measurement: Issues and Practice, 31, 21–26.
• Zenisky, A. L., & Hambleton, R. K. (2016). A model and good practices for score reporting. In S. Lane, M. R. Raymond, & T. M. Haladyna (Eds.) Handbook of test development (2nd ed., pp. 585–602). New York, NY: Routledge.