Examination of Statistical Methods Used in Group Comparisons in Visual Analogue Scale (VAS) Data in Terms of Type I Error Rate and Power of Test: Monte Carlo simulation study for VAS data

Abstract

In this study, how the application changes with the size of the scale, type I error rate and power of test values were examined in different sample sizes. The material of this study is the random numbers generated according to different sample sizes and different group means of standard deviation out of populations that hold multinomial distribution. In the study, permutation test, F-test and Kruskal-Wallis (KW) test were examined using combinations with different sample sizes (n = 5, 10, 20, 30, 50) and different p values (probability of occurrence of events which are 0.25, 0.50, 0.75) first interms of Type I error rate and then power of the test using different group averages (Δ =1.0, 1.5, 2.0). As a result of the simulations, it is seen that with small sample sizes, Kruskal-Wallis test was unable to maintain Type I error rate at 0.05 level. In the evaluation of such Likert-type data, it can be stated that, permutation test, one of the distribution free tests, is more practical than other tests in terms of maintaining the Type I error rate at 0.05 level and high power of test values. The permutation test is satisfactory in terms of both the type I error rate and the power of the test. Also, permutation test is a distribution-free test. Therefore, can be used without prerequisites. In almost every combination (π, Δ, k, n), permutation test had similar or superior type 1 error and power values than the F and KW tests. It was observed that compared to the 5 visual analogue scale, when the data is measured in 10 and 20 visual analogue scale, the power of test values decreased. In other words, if more than 5 visual analogue scale are made on the visual analogue scale data, the results would negatively be affected.

Keywords

• quantitized data
• likert-type data
• permutation test
• monte carlo simulation
• power of test values

Görsel Analog Ölçek (VAS) Verilerindeki Grup Karşılaştıralarında Kullanılan İstatistik Metotların Değerlendirilmesi: VAS verileri için Monte Carlo Simülasyon Çalışması

Abstract

Bu çalışmada farklı örneklem büyüklüklerinde uygulamanın ölçek büyüklüğü ile nasıl değiştiği, I. tip hata oranı ve test değerlerinin gücü incelenmiştir. Bu çalışmanın materyalini multinomial dağılıma sahip popülasyonlardan farklı örneklem büyüklüklerine ve farklı grup standart sapma ortalamalarına göre üretilen rastgele sayılar oluşturmaktadır. Çalışmada farklı örneklem büyüklükleri (n=5, 10, 20, 30, 50) ve farklı p değerlerine (olayların gerçekleşme olasılığı olan 0.25, 0.50, 0.75) sahip kombinasyonlar kullanılarak permütasyon testi, F-testi ve Kruskal-Wallis (KW) testi farklı grup ortalamaları (Δ =1.0, 1.5, 2.0) kullanılarak önce I. Tip hata oranı, sonra da testin gücü açısından incelenmiştir. Simülasyonlar sonucunda, küçük örneklem büyüklüklerinde Kruskal-Wallis testinin Tip I hata oranını 0.05 seviyesinde tutamadığı görülmüştür. Bu tür Likert tipi verilerin değerlendirilmesinde, dağılımdan bağımsız testlerden biri olan permütasyon testinin, Tip I hata oranını 0.05 düzeyinde tutması ve test değerlerinin gücünün yüksek olması açısından diğer testlere göre daha pratik olduğu söylenebilir. Permütasyon testi hem I. tip hata oranı hem de testin gücü açısından tatmin edicidir. Ayrıca permütasyon testi dağılım içermeyen bir testtir. Bu nedenle, önkoşul olmaksızın kullanılabilir. Hemen hemen her kombinasyonda (π, Δ, k, n) permütasyon testi, F ve KW testlerine benzer veya daha üstün tip 1 hata ve güç değerlerine sahiptir. Veriler 5 görsel analog ölçeğine kıyasla 10 ve 20 görsel analog ölçeğinde ölçüldüğünde test değerlerinin gücünün azaldığı gözlemlenmiştir. Başka bir deyişle, görsel analog ölçeği verileri üzerinde 5'ten fazla görsel analog ölçeği yapılırsa sonuçlar olumsuz etkilenecektir.

Keywords

• derecelendirilmiş veri
• likert tip veri
• permütasyon testi
• monte carlo simülasyonu
• testing gücü değerleri

References

1. [1] D. L. Clason and T. J. Dormody, ″Analyzing data measured by individual likert-type items,″ J. Agric. Educ., vol. 35 no.4, pp. 31-35, 1994.
2. [2] K. A. A. Small, ″Discrete choice model for ordered alternatives,″ Econometrica, vol. 55 no. 2, pp. 409-424, 1987.
3. [3] M. A. Garcia-Perez, ″Are the steps on likert scales equidistant? Responses on visual analog scales allow estimating their distances,″ Educ. Psychol. Meas., 1-32, 2023.
4. [4] S. Sinharay, Discrete Probability Distributions, McGaw Group Pty, 2010.
5. [5] O. Kavuncu, İstatistik Teorisi ve Teorik Dağılımlar, T.C. Ziraat Bank Publishing, 1995.
6. [6] F. X. Lesage, S. Berjot, and F. Deschamps, ″Clinical stress assessment using a visual analogue scale,″ Occupational medicine, vol. 62 no. 8, pp. 600-605, 2012.
7. [7] M. Kanda, M. Matsuhashi, N. Sawamoto, T. Oga, T. Mima, T. Nagamine and H. Shibasaki, ″Cortical potentials related to assessment of pain intensity with visual analogue scale (VAS),″ Clinical Neurophysiology, vol. 113 no. 7, pp. 1013-1024, 2002.
8. [8] A. Reich, M. Heisig, N. Q. Phan, K. Taneda, K. Takamori, S. Takeuchi and J. C. Szepietowski, ″Visual analogue scale: evaluation of the instrument for the assessment of pruritus,″ Acta Dermato Venereologica, vol. 92 no. 5, pp. 497, 2012.

9. [9] F. Dexter and D. H. Chestnut, ″Analysis of statistical tests to compare visual analog scale measurements among groups,″ Anesthesiology, vol. 82, pp. 896-902, 1995.
10. [10] M. Kliger, S. Stahl, M. Haddad, E. Suzan, R. Adler and E. Eisenberg, ″Measuring the intensity of chronic pain: are the visual analogue scale and the verbal rating scale interchangeable?″ Pain Practice, vol. 15, no. 6, pp. 538-547, 2015.
11. [11] M. J. Hjermstad, P. M. Fayers, D. F. Haugen, A. Caraceni, G. W. Hanks, J. H. Loge, R. Fainsinger, N. Aass and S. Kaasa, ″Studies comparing numerical rating scales, verbal rating scales, and visual analogue scales for assessment of pain intensity in adults: a systematic literature review,″ Journal of Pain and Symptom Management, vol. 41 no. 6, pp. 1073-1093, 2011.
12. [12] H. Van Laerhoven, H. J. Van der Zaag‐Loonen and B. H. Derkx, ″A comparison of likert scale and visual analogue scales as response options in children's questionnaires,″ Acta paediatrica, vol. 93 no. 6, pp. 830-835, 2004.
13. [13] J. Bielewicz, B. Daniluk, and P. Kamieniak, ″VAS and NRS, same or different? Are visual analog scale values and numerical rating scale equally viable tools for assessing patients after microdiscectomy?″ Pain Res Manag, (vol,page), 2022.
14. [14] A. Butler, P. Rothery and D. Roy, ″Minitab macros for resampling methods,″. Teach. Stat., vol. 25, no. 1, pp. 22-25, 2003.
15. [15] E. Stark and M. Abeles, ″Applying resampling methods to neurophysiological data,″ J Neurosci Methods, vol. 145, pp. 133–14, 2005. doi: 10.1016/j.jneumeth.2004.12.005
16. [16] D. S. Collingridge, ″A primer on quantitized data analysis and permutation testing,″ J Mix Methods Res, vol. 7 no. 1, pp. 81-97, 2012.
17. [17] A. Figueiredo, ″Resampling methods in ANOVA for data from the von Mises-Fisher distribution,″ Commun. Stat. Simul. Comput, pp. 1-15, 2021.
18. [18] Ö. Koşkan and F. Gürbüz, ″Comparison of F test and resampling approach for type I error rate and test power by simulation method,″ Journal of Agricultural Sciences, vol. 15, no. 1, pp. 105-111, 2009.
19. [19] D. J. Sheskin, Handbook of Parametric and Nonparametric Statistical Procedures, Chapman and Hall/CRC, 1987.
20. [20] nT. Kesici and Z. Kocabaş, Biyoistatistik, Ankara University Press, 1998.
21. [21] D. J. K. Mewhort, ″A comparison of the randomization test with the F test when error is skewed,″ Behav. Res. Methods, vol. 37 no.3, pp. 426-435, 2005.
22. [22] M. Weber, Robustness and Power of the t, Permutation T and Wilcoxon Tests(Doctoral thesis, University of Wayne State University, Detroit, Michigan), 2006. Retrieved from https://www.proquest.com/docview/304968464?pq-origsite=gscholar&fromopenview=true&sourcetype=Dissertations%20&%20Theses
23. [23] R. D. Routledge, ″P-values from permutation and F tests,″ Comput Stat Data Anal, vol. 24, pp. 379-386, 1997.
24. [24] E. Başpınar, and F. Gürbüz, ″The power of the test in the samples of various sample sizes were taken from the binary combinations of the normal, beta, gamma, and weibull distributions,″ Journal of Agricultural Sciences, vol. 6, no. 1, pp. 116-127, 2000.
25. [25] M. Mendeş and B. Tekindal, ″Determining the appropriate sample size in testing the difference between the means in normal and non-normal populations,″ The Journal of the Industrial Arts Education Faculty of Gazi University, vol. 111, pp. 25–38, 2002.
26. [26] G. Heller and E. S. Venkatraman, ″Resampling procedures to compare survival distributions in the presence of right-censored data,″ Biometrics, vol. 52, no. 4, pp. 1204-1213, 1996. doi: 10.2307/2532836.
27. [27] J. Ludbrook and H. Dudley, ″Why permutation tests are superior to t and F tests in biomedical research,″ Am. Stat., vol. 52, no. 2, pp. 127-132, 1998. doi: 10.1080/00031305.1998.10480551.
28. [28] P. Good, Permutation Tests: Practical Guide to Resampling Methods for Testing Hypotheses, Springer Verlag, New York, USA, 2000. doi: 10.1007/978-1-4757-2346-5 . [29] G. K. Balasubramani, S. R. Wisniewski, H. Zhang, and H. F. Eng, ″Development of an efficient SAS macro to perform permutation test for two independent samples,″ Comput. Methods Programs Biomed., vol. 79, pp. 179-187, 2005. doi: 10.1016/j.cmpb.2005.03.010.
29. [30] F. Pesarin and L. Salmaso, ″Permutation test for univariate and multivariate ordered categorical data,″Austrian J. Sta.t, vol. 35, no. 2, pp. 315-324, 2006. doi: 10.17713/ajs.v35i2&3.378.
30. [31] S. Keskin and M. Mendeş, ″Comparison of the power of the test of one-way ANOVA method and some approximation tests for the samples drawn from the exponential distributed population,″ J. Agric. Sci., vol. 8, no. 4, pp. 293-299, 2002.
31. [32] P. Vidoni, ″Improved prediction limits for continuous and discrete observations in generalised linear models,″ Biometrica, vol. 88, no. 3, pp. 881-887, 2001.