Beyond self-reports: Addressing bias and improving data quality in educational research

Abstract

The use of self-report in educational research has facilitated data collection by providing easy access to diverse populations in a short period of time. However, research has shown that these opportunities come with challenges. From inadequate response efforts to culturally influenced responses, there are numerous failures that can call into question the validity of our findings. For example, discrepancies between self-reports and objective data often reveal underlying biases. Poor data quality, exacerbated by social desirability in sensitive constructs, individual and environmental factors, and changes in scale structure, has highlighted that our methods may have some limitations that reduce generalisability and trigger the replicability crisis. However, these limitations can also lead to improvements in both survey design and data interpretation. Our experiences point to the need to integrate multiple data sources, to improve survey development and adaptation methods, and to use true experimental studies more frequently. By reflecting on these challenges, we suggest new directions for survey implementation in educational research studies to improve the reliability and replicability of our findings and deepen our understanding of complex human dynamics.

Keywords

• Method effects
• survey experiments
• careless responding
• cultural sensitivity
• experimental design

References

1. Akbulut, Y. (2015). Predictors of inconsistent responding in web surveys. Internet Research, 25(1), 131-147. https://doi.org/10.1108/IntR-01-2014-0017
2. Akbulut, Y. (2024). Impact of different practice testing methods on learning outcomes. European Journal of Education, 59 (2), e12626. https://doi.org/10.1111/ejed.12626
3. Akbulut, Y., & Dönmez, O. (2018). Predictors of digital piracy among Turkish undergraduate students. Telematics and Informatics, 35(5), 1324-1334. https://doi.org/10.1016/j.tele.2018.03.004
4. Akbulut, Y., Saykılı, A., Öztürk, A., & Bozkurt, A. (2023). What if it's all an illusion? To what extent can we rely on self-reported data in open, online, and distance education systems? International Review of Research in Open and Distributed Learning, 24 (3), 1-17. https://doi.org/10.19173/irrodl.v24i3.7321
5. Akbulut, Y., Uysal, Ö., Odabasi, H. F., & Kuzu, A. (2008). Influence of gender, program of study and PC experience on unethical computer using behaviors of Turkish undergraduate students. Computers & Education, 51(2), 485-492. https://doi.org/10.1016/j.compedu.2007.06.004
6. Alemdag, E., & Cagiltay, K. (2018). A systematic review of eye tracking research on multimedia learning. Computers & Education, 125, 413-428. https://doi.org/10.1016/j.compedu.2018.06.023
7. Andrade, C. (2021). HARKing, cherry-picking, p-hacking, fishing expeditions, and data dredging and mining as questionable research practices. The Journal of Clinical Psychiatry, 82(1), 25941. https://doi.org/10.4088/JCP.20f13804
8. Andrejevic, M., & Selwyn, N. (2020). Facial recognition technology in schools: Critical questions and concerns. Learning, Media and Technology, 45(2), 115-128. https://doi.org/10.1080/17439884.2020.1686014

9. Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: a primer. Frontiers in Public Health, 6, 149. https://doi.org/10.3389/fpubh.2018.00149
10. Booth, C., Wong, E., Brown, M., & Fitzsimons, E. (2024). Evaluating the effect of monetary incentives on web survey response rates in the UK Millennium Cohort Study. Survey Research Methods, 18(1), 47-58. https://doi.org/10.18148/srm/2024.v18i1.8210
11. Chang, M., Büchel, D., Reinecke, K., Lehmann, T., & Baumeister, J. (2022). Ecological validity in exercise neuroscience research: A systematic investigation. European Journal of Neuroscience, 55(2), 487-509. https://doi.org/10.1111/ejn.15595
12. Dalton, D., & Ortegren, M. (2011). Gender differences in ethics research: The importance of controlling for the social desirability response bias. Journal of Business Ethics, 103, 73-93. https://doi.org/10.1007/s10551-011-0843-8
13. Danziger, S., Levav, J., & Avnaim-Pesso, L. (2011). Extraneous factors in judicial decisions. Proceedings of the National Academy of Sciences, 108(17), 6889-6892. https://doi.org/10.1073/pnas.1018033108
14. Demir, E. K. (2023). Investigating the effect of item order on the psychometric properties of a self-efficacy perception scale. E-International Journal of Educational Research, 14(5). https://doi.org/10.19160/e-ijer.1362442
15. Demirkol, S., & Kelecioglu, H. (2022). Investigating the effect of item position on person and item parameters: PISA 2015 Turkey sample. Journal of Measurement and Evaluation in Education and Psychology, 13(1), 69-85. https://doi.org/10.21031/epod.958576
16. DeSimone, J. A., Harms, P. D., & DeSimone, A. J. (2015). Best practice recommendations for data screening. Journal of Organizational Behavior, 36 (2), 171–181. https://doi.org/10.1002/job.1962
17. DeSimone, J. A., & Harms, P. D. (2018). Dirty data: The effects of screening respondents who provide low-quality data in survey research. Journal of Business and Psychology, 33, 559-577. https://doi.org/10.1007/s10869-017-9514-9
18. Dönmez, O., & Akbulut, Y. (2016). Social desirability bias in cyberbullying research. Educational Technology: Theory and Practice, 6(2), 1-18. https://doi.org/10.17943/etku.19033
19. Dönmez, O., Akbulut, Y., Zabzun, G., & Köseoğlu, B. (2025). Effects of survey order on subjective measures of cognitive load: A randomized controlled trial. Applied Cognitive Psychology, 39(2), e70039. https://doi.org/10.1002/acp.70039
20. Erdem-Kara, B., & Akbulut, Y. (2025). Prevalence and psychometric implications of careless responses in an online student survey. Journal of Psychoeducational Assessment, 07342829251328132. https://doi.org/10.1177/07342829251328132
21. Goodman, J., Anson, R., & Belcheir, M. (2015). The effect of incentives and other instructor-driven strategies to increase online student evaluation response rates. Assessment & Evaluation in Higher Education,  40(7), 958-970. https://doi.org/10.1080/02602938.2014.960364
22. Grieve, R., & Elliott, J. (2013). Cyberfaking: I can, so I will? Intentions to fake in online psychological testing. Cyberpsychology, Behavior, and Social Networking, 16(5), 364–369. https://doi.org/10.1089/cyber.2012.0271
23. Hambleton, R. K., & Traub, R. E. (1974). The effects of item order on test performance and stress. The Journal of Experimental Education, 43(1), 40-46. https://doi.org/10.1080/00220973.1974.10806302
24. Heggestad, E. D., Scheaf, D. J., Banks, G. C., Monroe Hausfeld, M., Tonidandel, S., & Williams, E. B. (2019). Scale adaptation in organizational science research: A review and best-practice recommendations. Journal of Management, 45(6), 2596-2627. https://doi.org/10.1177/0149206319850280
25. Huang, F. L., & Cornell, D. G. (2016). Question order affects the measurement of bullying victimization among middle school students. Educational and Psychological Measurement, 76(5), 724–740. https://doi.org/10.1177/0013164415622664
26. Iaconelli, R., & Wolters, C. A. (2020). Insufficient effort responding in surveys assessing self-regulated learning: Nuisance or fatal flaw?. Frontline Learning Research, 8(3), 104-125. https://doi.org/10.14786/flr.v8i3.521
27. King, M. F., & Bruner, G. C. (2000). Social desirability bias: A neglected aspect of validity testing. Psychology and Marketing, 17(2), 79-103. https://doi.org/10.1002/(SICI)1520-6793(200002)17:2<79::AID-MAR2>3.0.CO;2-0
28. Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213-36. https://doi.org/10.1002/acp.2350050305
29. Krosnick, J. A., & Alwin, D. F. (1987). An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opinion Quarterly, 51(2), 201-219. https://doi.org/10.1086/269029
30. Newman, D. L., Kundert, D. K., Lane Jr, D. S., & Bull, K. S. (1988). Effect of varying item order on multiple-choice test scores: Importance of statistical and cognitive difficulty. Applied Measurement in Education, 1(1), 89-97. https://doi.org/10.1207/s15324818ame0101_8
31. Örün, Ö., & Akbulut, Y. (2019). Effect of multitasking, physical environment and electroencephalography use on cognitive load and retention. Computers in Human Behavior, 92, 216-229. https://doi.org/10.1016/j.chb.2018.11.027
32. Rasinski, K. A., Lee, L., & Krishnamurty, P. (2012). Question order effects. In H. Cooper, P. M. Camic, D. L. Long, A. T. Panter, D. Rindskopf, & K. J. Sher (Eds.), APA handbook of research methods in psychology, Vol. 1. Foundations, planning, measures, and psychometrics (pp. 229–248). American Psychological Association. https://doi.org/10.1037/13619-014
33. Roos, A. L., Goetz, T., Voracek, M., Krannich, M., Bieg, M., Jarrell, A., & Pekrun, R. (2021). Test anxiety and physiological arousal: A systematic review and meta-analysis. Educational Psychology Review, 33, 579-618. https://doi.org/10.1007/s10648-020-09543-z
34. Siddiq, F., & Scherer, R. (2019). Is there a gender gap? A meta-analysis of the gender differences in students' ICT literacy. Educational Research Review, 27, 205-217. https://doi.org/10.1016/j.edurev.2019.03.007
35. Skulmowski, A. (2023). Learners emphasize their intrinsic load if asked about it first: communicative aspects of cognitive load measurement. Mind, Brain, and Education, 17(3), 161-218. https://doi.org/10.1111/mbe.12369
36. Stark, T. H., Silber, H., Krosnick, J. A., Blom, A. G., Aoyagi, M., Belchior, A., Bosnjak, M., Clement, S. L., John, M., Jónsdóttir, G.A., Lawson, K., Lynn, P., Martinsson, J., Shamshiri-Petersen, D., Tvinnereim, E., & Yu, R. R. (2020). Generalization of classic question order effects across cultures. Sociological Methods & Research, 49(3), 567-602. https://doi.org/10.1177/0049124117747304
37. Statista. (2023, September 29). Turkey: most popular sports 2013. https://www.statista.com/statistics/677503/most-followed-sports-turkey/
38. Stefkovics, Á., & Kmetty, Z. (2022). A comparison of question order effects on item-by-item and grid formats: visual layout matters. Measurement Instruments for the Social Sciences, 4(8) 1-12. https://doi.org/10.1186/s42409-022-00036-z
39. Steger, D., Jankowsky, K., Schroeders, U., & Wilhelm, O. (2023). The road to hell is paved with good intentions: How common practices in scale construction hurt validity. Assessment, 30(6), 1811-1824. https://doi.org/10.1177/10731911221124846
40. Şahin, M. D. (2021). Effect of item order on certain psychometric properties: A demonstration on a cyberloafing scale. Frontiers in Psychology, 12, 590545.https://doi.org/10.3389/fpsyg.2021.590545
41. Thorndike, F. P., Carlbring, P., Smyth, F. L., Magee, J. C., Gonder-Frederick, L., Ost, L. G., & Ritterband, L. M. (2009). Web-based measurement: Effect of completing single or multiple items per webpage. Computers in Human Behavior, 25(2), 393-401. https://doi.org/10.1016/j.chb.2008.05.006
42. Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157), 1124-1131. https://doi.org/10.1126/science.185.4157.1124
43. Weinberg, M. K., Seton, C., & Cameron, N. (2018). The measurement of subjective wellbeing: Item-order effects in the personal wellbeing index—adult. Journal of Happiness Studies, 19 (1), 315-332. https://doi.org/10.1007/s10902-016-9822-1.
44. Wise, S. L., Kuhfeld, M. R., & Lindner, M. A. (2024). Don’t test after lunch: The relationship between disengagement and the time of day that low-stakes testing occurs. Applied Measurement in Education, 37(1), 14–28. https://doi.org/10.1080/08957347.2024.2311925
45. Yedukondalu, J., Sunkara, K., Radhika, V., Kondaveeti, S., Anumothu, M., & Murali Krishna, Y. (2025). Cognitive load detection through EEG lead wise feature optimization and ensemble classification. Scientific Reports, 15(1), 842. https://doi.org/10.1038/s41598-024-84429-6
46. Yörük-Açıkel, B., Turhan, U., & Akbulut, Y. (2018). Effect of multitasking on simulator sickness and performance in 3D aerodrome control training. Simulation & Gaming, 49 (1), 27-49. https://doi.org/10.1177/1046878117750417
47. Xue, Y., Wang, K., & Qiu, Y. (2024). Enhancing online learning: A multimodal approach for cognitive load assessment. International Journal of Human-Computer Interaction. Advance online publication. https://doi.org/10.1080/10447318.2024.2327198
48. Zhou, M., & Winne, P.H. (2012). Modeling academic achievement by self-reported versus traced goal orientation. Learning and Instruction, 22(6), 413-419. https://doi.org/10.1016/j.learninstruc.2012.03.004
49. Zhu, W., Wang, F., Mayer, R. E., & Liu, T. (2024). Effects of explaining a science lesson to others or to oneself: A cognitive neuroscience approach. Learning and Instruction, 91, 101897. https://doi.org/10.1016/j.learninstruc.2024.101897