Psikometri literatüründeki eğilimlerin yapısal konu modellemesi ile belirlenmesi

Öz

Bilginin dijitalleşmesi, ilgili bilgiyi bulmayı ve keşfetmeyi giderek zorlaştırmakta; bu durum, büyük miktarda veriyi düzenlemeye, aramaya ve anlamaya yardımcı olacak hesaplama araçlarının geliştirilmesini gerekli kılmaktadır. Psikometri gibi büyük veri kümeleri içeren alanlarda, araştırma eğilimlerinin kapsamlı bir şekilde incelenmesi ve bu araçlar aracılığıyla çeşitli temaların önemini ve zaman içindeki değişimini anlamak, alanın dinamik yapısını değerlendirmek açısından büyük önem taşımaktadır. Bu çalışma, psikometri alanında önde gelen on bir dergide yayımlanan araştırmalarda ele alınan temaları keşfetmeyi ve bu temaların genel dağılımını belirlemeyi amaçlamaktadır. Bu doğrultuda, yapısal konu modellemesi yöntemi kullanılmıştır. Web of Science veri tabanından elde edilen 8.523 makale özetinin analizi, yayınlarda on dört farklı konunun varlığını ortaya koymuştur. "Ölçek Geliştirme ve Geçerlik" en baskın konu olarak öne çıkarken, "Değişen Madde Fonksiyonu" en az öne çıkan konu olmuştur. Konuların akademik dergiler arasındaki dağılımı, dergilerin psikometri araştırmalarının gelişimini ve evrimini şekillendirmedeki kritik rolünü vurgulamaktadır. Ayrıca, konu korelasyonlarının detaylı incelenmesiyle gelecekteki olası araştırma yönleri ve disiplinler arası çalışma alanları belirlenmiştir. Bu çalışma, psikometri alanındaki güncel gelişmeleri takip etmek isteyen araştırmacılar için önemli bir kaynak niteliği taşımakta olup, elde edilen bulgular bu alanda gelecekte yapılacak araştırmalara rehberlik edecek değerli içgörüler sunmaktadır.

Anahtar Kelimeler

• Yapısal Konu Modellemesi
• Psikometri
• Trend Analizi
• Gizli Dirichlet Tahsisi
• Metin Madenciliği

Exploring trends in psychometrics literature through a structural topic model

Abstract

The digitalization of knowledge has made it increasingly challenging to find and discover relevant information, leading to the development of computational tools to assist in organizing, searching, and comprehending vast amounts of information. In fields like psychometrics, which involve large datasets, a comprehensive examination of research trends, as well as understanding the prominence of various themes and their evolution over time through these tools, is essential for assessing the dynamic structure of the field. This study aims to explore the themes addressed in publications from eleven leading journals in psychometrics and to determine the overall distribution of topics. To achieve this, structural topic modelling has been employed. A comprehensive analysis of 8,523 article abstracts sourced from the Web of Science database revealed the existence of fourteen topics within the publications. “Scale Development and Validation” emerged as the most prominent topic, whereas “Differential Item Functioning” was the least well-known. The distribution of topics across academic journals emphasized the key role journals play in shaping the development and evolution of psychometric research. Through further exploration of topic correlations, potential future research directions and between-topic research areas were revealed. This study serves as a valuable resource for researchers aiming to keep up with the latest advancements in psychometrics. The findings provide crucial insights to guide and shape future research in the field.

Keywords

• Structural topic modelling
• Psychometrics
• Trend analysis
• Latent dirichlet allocation
• Text mining

References

1. Ackerman, T.A., Bandalos, D.L., Briggs, D.C., Everson, H.T., Ho, A.D., Lottridge, S.M., Madison, M.J., Sinharay, S., Rodriguez, M.C., Russell, M., Von Davier, A.A., & Wind, S.A. (2023). Foundational competencies in educational measurement. Educational Measurement Issues and Practice, 43(3), 7–17. https://doi.org/10.1111/emip.12581
2. Anderson, D., Rowley, B., Stegenga, S., Irvin, P.S., & Rosenberg, J.M. (2020). Evaluating content‐related validity evidence using a text‐based machine learning procedure. Educational Measurement: Issues and Practice, 39(4), 53 64. https://doi.org/10.1111/emip.12314
3. Bai, X., Zhang, X., Li, K.X., Zhou, Y., & Yuen, K.F. (2021). Research topics and trends in the maritime transport: A structural topic model. Transport Policy, 102, 11 24. https://doi.org/10.1016/j.tranpol.2020.12.013
4. Banks, G.C., Woznyj, H.M., Wesslen, R.S., & Ross, R.L. (2018). A review of best practice recommendations for text analysis in R (and a User-Friendly app). Journal of Business and Psychology, 33(4), 445–459. https://doi.org/10.1007/s10869-017-9528-3
5. Bastola, M. N., & Hu, G. (2021). Chasing my supervisor all day long like a hungry child seeking her mother!: Students’ perceptions of supervisory feedback. Studies in Educational Evaluation, 70, 101055. https://doi.org/10.1016/j.stueduc.2021.101055
6. Blanca, M.J., Alarcón, R., Arnau, J., Bono, R., & Bendayan, R. (2018). Non-normal data: Is ANOVA still a valid option? Psicothema, 30(4), 552 557. https://doi.org/10.7334/psicothema2018.245
7. Blei, D.M., Ng, A.Y., & Jordan, M.I. (2003). Latent dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022. https://doi.org/10.5555/944919.944937
8. Blei, D.M. (2012). Probabilistic topic models. Communications of the ACM, 55(4), 77–84. https://doi.org/10.1145/2133806.2133826

9. Boon-Itt, S., & Skunkan, Y. (2020). Public perception of the COVID-19 pandemic on Twitter: Sentiment analysis and topic modeling study. JMIR Public Health and Surveillance, 6(4), e21978.
10. Briggs, D.C. (2024). Strive for measurement, set new standards, and try not to be evil. Journal of Educational and Behavioral Statistics, 49(5), 694 701. https://doi.org/10.3102/10769986241238479
11. Brooks, C., Burton, R., Van Der Kleij, F., Carroll, A., Olave, K., & Hattie, J. (2020). From fixing the work to improving the learner: An initial evaluation of a professional learning intervention using a new student-centred feedback model. Studies in Educational Evaluation, 68, 100943. https://doi.org/10.1016/j.stueduc.2020.100943
12. Borsboom, D. (2005). Measuring the mind: Conceptual issues in contemporary psychometrics. Cambridge University Press. https://doi.org/10.1017/CBO9780511613980
13. Buckhalt, J.A. (1999). Defending the science of mental ability and its central dogma. Review of Jensen on Intelligence g Factor. Psycoloquy, 10(23). http://www.cogsci.ecs.soton.ac.uk/cgi/psyc/newpsy?10.47
14. Buzick, H.M., Casabianca, J.M., & Gholson, M.L. (2023). Personalizing Large‐Scale Assessment in practice. Educational Measurement Issues and Practice, 42(2), 5–11. https://doi.org/10.1111/emip.12551
15. Chen, S., & Lei, P. (2005). Controlling item exposure and test overlap in computerized adaptive testing. Applied Psychological Measurement, 29(3), 204 217. https://doi.org/10.1177/0146621604271495
16. Chen, J., Chen, C., & Shih, C. (2013). Improving the control of type I error rate in assessing differential item functioning for hierarchical generalized linear model when impact is presented. Applied Psychological Measurement, 38(1), 18 36. https://doi.org/10.1177/0146621613488643
17. Choi, J.Y., Hwang, H., Yamamoto, M., Jung, K., & Woodward, T.S. (2016). A unified approach to functional principal component analysis and functional Multiple-Set canonical correlation. Psychometrika, 82(2), 427–441. https://doi.org/10.1007/s11336-015-9478-5
18. Cizek, G.J., Bowen, D., & Church, K. (2010). Sources of Validity Evidence for Educational and Psychological Tests: a Follow-Up Study. Educational and Psychological Measurement, 70(5), 732–743. https://doi.org/10.1177/0013164410379323
19. Cohn, S., & Huggins-Manley, A.C. (2019). Applying unidimensional models for semiordered data to scale data with neutral responses. Educational and Psychological Measurement, 80(2), 242–261. https://doi.org/10.1177/0013164419861143
20. Jones, L.V., & Thissen, D.M. (2006). A history and overview of psychometrics. In Handbook of statistics (pp. 1–27). https://doi.org/10.1016/s0169-7161(06)26001-2
21. Gao, X., & Sazara, C. (2023). Discovering mental health research topics with topic modeling. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2308.13569
22. Göral, S., Özkan, S., Sercekus, P., & Alataş, E. (2021). The validity and reliability of the Turkish version of the Attitudes to Fer-Tility and Childbearing Scale (AFCS). International Journal of Assessment Tools in Education, 8(4), 764 774. https://doi.org/10.21449/ijate.773132
23. Gregson, T. (1991). The separate constructs of communication satisfaction and job satisfaction. Educational and Psychological Measurement, 51(1), 39 48. https://doi.org/10.1177/0013164491511003
24. Groenen, P.J.F., & van der Ark, L.A. (2006). Visions of 70 years of psychometrics: the past, present, and future. Statistica Neerlandica, 60(2), 135–144. https://doi.org/10.1111/j.1467-9574.2006.00318.x
25. Guo, J., & Luh, W. (2008). Approximate sample size formulas for testing group mean differences when variances are unequal in One-Way ANOVA. Educational and Psychological Measurement, 68(6), 959–971. https://doi.org/10.1177/0013164408318759
26. Hidalgo, M.D., & LÓPez-Pina, J.A. (2004). Differential Item Functioning Detection and Effect Size: A Comparison between Logistic Regression and Mantel-Haenszel Procedures. Educational and Psychological Measurement, 64(6), 903 915. https://doi.org/10.1177/0013164403261769
27. Huynh, H. (1996). Decomposition of a Rasch partial credit item into independent binary and indecomposable trinary items. Psychometrika, 61(1), 31 39. https://doi.org/10.1007/bf02296957 Hwang, S., Flavin, E., & Lee, J.E. (2023). Exploring research trends of technology use in mathematics education: A scoping review using topic modeling. Education and Information Technologies, 28, 10753–10780. https://doi.org/10.1007/s10639-023-11603-0
28. Jiang, Y., Von Davier, A.A., & Chen, H. (2012). Evaluating equating results: percent relative error for chained kernel equating. Journal of Educational Measurement, 49(1), 39–58. https://doi.org/10.1111/j.1745-3984.2011.00159.x Jiang, X., & Ironsi, S.S. (2024). Do learners learn from corrective peer feedback? Insights from students. Studies in Educational Evaluation, 83, 101385. https://doi.org/10.1016/j.stueduc.2024.101385
29. Kiers, H.A.L. (1997). Three-mode orthomax rotation. Psychometrika, 62(4), 579–598. https://doi.org/10.1007/bf02294644
30. Kim, S. (2001). An evaluation of a Markov Chain Monte Carlo method for the Rasch model. Applied Psychological Measurement, 25(2), 163 176. https://doi.org/10.1177/01466210122031984
31. Kim, S. (2006). A comparative study of IRT fixed parameter calibration methods. Journal of Educational Measurement, 43(4), 355–381. https://doi.org/10.1111/j.1745-3984.2006.00021.x
32. Lederman, J. (2023). Validity and racial justice in educational assessment. Applied Measurement in Education, 36(3), 242 254. https://doi.org/10.1080/08957347.2023.2214654
33. Liu, J., & Low, A.C. (2008). A Comparison of the Kernel Equating Method with Traditional Equating Methods Using SAT® Data. Journal of Educational Measurement, 45(4), 309–323. https://doi.org/10.1111/j.1745-3984.2008.00067.x
34. MacDonald, P.L., & Gardner, R.C. (2000). Type I Error Rate Comparisons of Post Hoc Procedures for I j Chi-Square Tables. Educational and Psychological Measurement, 60(5), 735–754. https://doi.org/10.1177/00131640021970871
35. Martin, C.R., & Savage-McGlynn, E. (2013). A ‘good practice’ guide for the reporting of design and analysis for psychometric evaluation. Journal of Reproductive and Infant Psychology, 31(5), 449–455. https://doi.org/10.1080/02646838.2013.835036
36. Meeter, M. (2022). Predicting Retention in Higher Education from high-stakes Exams or School GPA. Educational Assessment, 28(1), 1 10. https://doi.org/10.1080/10627197.2022.2130748
37. Michell, J. (2022). The art of imposing measurement upon the mind: Sir Francis Galton and the genesis of the psychometric paradigm. Theory & Psychology, 32(3), 375 400. https://doi.org/10.1177/09593543211017671
38. Pan, Y., Livne, O., Wollack, J.A., & Sinharay, S. (2023). Item selection algorithm based on collaborative filtering for item exposure control. Educational Measurement Issues and Practice, 42(4), 6–18. https://doi.org/10.1111/emip.12578
39. Park, S., Steiner, P.M., & Kaplan, D. (2018). Identification and sensitivity analysis for average causal mediation effects with time-varying treatments and mediators: Investigating the underlying mechanisms of kindergarten retention policy. Psychometrika, 83(2), 298–320. https://doi.org/10.1007/s11336-018-9606-0
40. Plake, B.S., & Wise, L.L. (2014). What is the role and importance of the revised AERA, APA, NCME standards for educational and psychological testing? Educational Measurement Issues and Practice, 33(4), 4–12. https://doi.org/10.1111/emip.12045
41. Polatgil, M. (2023). Analyzing comments made to the Duolingo mobile application with topic modeling. International Journal of Computing and Digital Systems, 13(1), 223–230.
42. Randall, J., Slomp, D., Poe, M., & Oliveri, M.E. (2022). Disrupting White Supremacy in Assessment: Toward a Justice-Oriented, Antiracist validity framework. Educational Assessment, 27(2), 170–178. https://doi.org/10.1080/10627197.2022.2042682
43. Richardson, G.M., Bowers, J., Woodill, A.J., Barr, J.R., Gawron, J.M., & Levine, R.A. (2014). Topic models: A tutorial with R. International Journal of Semantic Computing, 8(01), 85-98.
44. Roberts, M.E., Stewart, B.M., Tingley, D., Lucas, C., Leder‐Luis, J., Gadarian, S.K., Albertson, B., & Rand, D.G. (2014). Structural topic models for Open‐Ended Survey Responses. American Journal of Political Science, 58(4), 1064 1082. https://doi.org/10.1111/ajps.12103
45. Roberts, M.E., Stewart, B.M., & Tingley, D. (2019). stm: An R package for structural topic models. Journal of Statistical Software, 91(2). https://doi.org/10.18637/jss.v091.i02
46. Rupp, A.A. (2018). Designing, evaluating, and deploying automated scoring systems with validity in mind: Methodological design decisions. Applied Measurement in Education, 31(3), 191–214. https://doi.org/10.1080/08957347.2018.1464448
47. Schuster, C. (2004). A Note on the Interpretation of Weighted Kappa and its Relations to Other Rater Agreement Statistics for Metric Scales. Educational and Psychological Measurement, 64(2), 243–253. https://doi.org/10.1177/0013164403260197
48. Shih, C., & Wang, W. (2009). Differential Item Functioning Detection Using the Multiple Indicators, Multiple Causes Method with a Pure Short Anchor. Applied Psychological Measurement, 33(3), 184–199. https://doi.org/10.1177/0146621608321758
49. Silge, J., & Robinson, D. (2016). tidytext: Text Mining and Analysis Using Tidy Data Principles in R. The Journal of Open-Source Software, 1(3), 37. https://doi.org/10.21105/joss.00037
50. Singh, J., & Gupta, V. (2017). A systematic review of text stemming techniques. Artificial Intelligence Review, 48(2), 157–217. https://doi.org/10.1007/s10462-016-9498-2
51. Sireci, S.G. (2013). Agreeing on validity arguments. Journal of Educational Measurement, 50(1), 99–104. https://doi.org/10.1111/jedm.12005
52. Sijtsma, K., & Pfadt, J.M. (2021). Part II: On the use, the misuse, and the very limited usefulness of Cronbach’s alpha: Discussing lower bounds and correlated errors. Psychometrika, 86(4), 843-860.
53. Tharenou, P., & Terry, D.J. (1998). Reliability and validity of scores on scales to measure managerial aspirations. Educational and Psychological Measurement, 58(3), 475–492. https://doi.org/10.1177/0013164498058003008
54. Talloen, W., Moerkerke, B., Loeys, T., De Naeghel, J., Van Keer, H., & Vansteelandt, S. (2016). Estimation of indirect effects in the presence of unmeasured confounding for the Mediator–Outcome relationship in a multilevel 2-1-1 mediation model. Journal of Educational and Behavioral Statistics, 41(4), 359 391. https://doi.org/10.3102/1076998616636855
55. Tonidandel, S., Summerville, K.M., Gentry, W.A., & Young, S.F. (2021). Using structural topic modeling to gain insight into challenges faced by leaders. The Leadership Quarterly, 33(5), 101576. https://doi.org/10.1016/j.leaqua.2021.101576
56. Tunç, E.B., Parlak, S., Uluman, M., & Eryiğit, D. (2021). Development of the Hostility in Pandemic Scale (HPS): A Validity and Reliability study. International Journal of Assessment Tools in Education, 8(3), 475–486. https://doi.org/10.21449/ijate.837616
57. Wheeler, J.M., Cohen, A.S., & Wang, S. (2024). A comparison of latent semantic analysis and latent Dirichlet allocation in educational measurement. Journal of Educational and Behavioral Statistics, 49(5), 848–874. https://doi.org/10.3102/10769986231209446
58. Van Der Ark, L.A. (2005). Stochastic ordering of the latent trait by the sum score under various polytomous IRT models. Psychometrika, 70(2), 283–304. https://doi.org/10.1007/s11336-000-0862-3
59. Van der Linden, W.J., & Glas, C.A.W. (2000). Computerized adaptive testing: Theory and practice. Springer. https://doi.org/10.1007/978-1-4757-3224-0
60. Vitoratou, S., & Pickles, A. (2017). Psychometric analysis of the Mental Health Continuum-Short Form. Journal of Clinical Psychology, 73(10), 1307 1322. https://doi.org/10.1002/jclp.22422
61. Xiong, J., & Li, F. (2023). Bilevel topic model-based multitask learning for constructed-response multidimensional automated scoring and interpretation. Educational Measurement: Issues and Practice, 42(2), 42–61. https://doi.org/10.1111/emip.12550
62. Yavuz, S., Odabaş, M., & Özdemir, A. (2016). Öğrencilerin sosyoekonomik düzeylerinin TEOG matematik başarısına etkisi [Effect of socio-economic status on student’s TEOG mathematics achievement]. Journal of Measurement and Evaluation in Education and Psychology, 7(1), 85–95. https://doi.org/10.21031/epod.86531
63. Zagaria, A., & Lombardi, L. (2024). Bayesian versus frequentist approaches in psychometrics: a bibliometric analysis. Discover Psychology, 4, 61. https://doi.org/10.1007/s44202-024-00164-z
64. Zhan, P., Man, K., Wind, S.A., & Malone, J. (2022). Cognitive diagnosis modeling incorporating response times and fixation counts providing comprehensive feedback and accurate diagnosis. Journal of Educational and Behavioral Statistics, 47(6), 736–776. https://doi.org/10.3102/10769986221111085