Comparison of Student – t, Welch’s t, and Mann – Whitney U Tests in Terms of Type I Error Rate and Test Power

Year 2023, Volume: 37 Issue: 2, 223 - 231, 30.08.2023

Malik Ergin , Ozgur Koskan

Abstract

In this study, we compared the Student's t-test, Welch's t-test, and Mann-Whitney U test, in terms of their type I error rate and statistical power when the assumptions of parametric tests are violated in different situations. Materials used in this study, consisted of random numbers generated using the Numpy library in the Python programming language. All random numbers were generated from a normal distribution with N (0, 1) parameters. Balanced and unbalanced experimental conditions were simulated 50 000 times for each combination. The study revealed that, in comparison to other tests, Welch’s t - test was particularly more conservative in terms of type I error rate. It was discovered that the Student-t test had higher power values than the Mann-Whitney U test, mainly when only a small sample size of observations was used for the analysis. This simulation study indicated that Welch’s t - test is robust for preserving type I error rate when the distribution is normal. Therefore, in practice, the use of Welch t-test is recommended based on the findings of this study. One of the recommendations of this study is that the tests in question should also be evaluated in cases where observations have different distributions.

Keywords

Type I error rate, Test power, Student – t test, Mann – Whitney U test, Welch’s – t test, Simulation

References

• Ahad NA, Yahaya SSS (2014). Sensitivity analysis of Welch’s t -test. AIP Conference Proceedings. American Institute of Physics Inc. C., 1605(1): 888-893. Doi:10.1063/1.4887707 Aslan E, Koşkan Ö, Altay Y (2021). Determination of the sample size on different independent K group comparisons by power analysis. Türkiye Tarımsal Araştırmalar Dergisi, 8(1): 34-41. Doi:10.19159/tutad.792694
• Bindak R (2014). Comparision Mann-Whitney U Test and Students’ t Test in terms of type 1 error rate and test power: a Monte Carlo simulation study. Afyon Kocatepe University Journal of Sciences and Engineering, 14(1): 5-11. Doi:10.5578/fmbd.7380
• Bradley JV (1978). Robustness. British Journal of Mathematical and Statistical Psychology, 31(2):144-152. Doi:10.1111/j.2044-8317.1978.tb00581.x
• Delacre M, Lakens D, Leys C (2017). Why psychologists should by default use welch’s t-Test instead of student’s t-Test. International Review of Social Psychology, 30(1): 92-101. Doi:10.5334/irsp.82
• Derrick B, Toher D, White, P (2016). Why Welch’s test is type 1 error robust. The Quantitative Methods in Psychology, 12(1). Doi:10.20982/tqmp.12.1.p030
• Harris CR, Millman KJ, van der Walt SJ, Gommers R, Virtanen P, Cournapeau D, Oliphant TE (2020). Array programming with NumPy. Nature, 585(7825): 357-362. Doi:10.1038/s41586-020-2649-2
• Kasuya E (2001). Mann-Whitney U test when variances are unequal. Animal Behaviour, 61: 1247-1249. Doi:10.1006/anbe.2001.1691
• Keselman HJ, Keselman JC, Shaffer JP (1991). Multiple pairwise comparisons of repeated measures means under violation of multisample sphericity. Psychological Bulletin, 110(1): 162. Doi:10.1037/0033-2909.110.1.162
• Keselman HJ, Othman AR, Wilcox RR, Fradette K (2004). The new and improved two-sample t test. Psychological Science, 15(1): 47-51. Doi:10.1111/j.0963-7214.2004.01501008.x
• Koskan O, Koknaroglu H, Altay Y (2022). Determination of minimum number of animals in comparing treatment means by power analysis. MVZ Córdoba, 27(2): 1-11. Doi:10.21897/rmvz.2572
• McKnight PE, Najab J (2010). Mann‐Whitney U Test. The Corsini encyclopedia of psychology, 1(1). Doi:10.1002/9780470479216.CORPSY0524
• Murphy KR, Myors B, Wolach A (2014). Statistical Power Analysis: A Simple And General Model For Traditional And Modern Hypothesis Tests. Routledge, New York, USA. p. 244. Doi: 10.4324/9781315773155
• Ruxton GD (2006). The unequal variance t-test is an underused alternative to Student’s t-test and the Mann-Whitney U test. Behavioral Ecology, 17(4): 688-690. Doi:10.1093/beheco/ark016
• Welch BL (1947). The generalization of “Student’s” problem when several different population variances are involved. Biometrika, 34(1-2): 28-35. Doi:10.1093/biomet/34.1-2.28
• Winter JCF (2013). Using the Student’s t-test with extremely small sample sizes. Practical Assessment, Research, and Evaluation Practical Assessment, 18(1): 10. Doi:10.7275/e4r6-dj05
• Zimmerman DW (2004). Conditional probabilities of rejecting h0 by pooled and separate-variances t tests given heterogeneity of sample variances. Communications in Statistics Part B: Simulation and Computation, 33(1): 69-81. Doi:10.1081/SAC-120028434
• Zimmerman DW, Zumbo BD (1993). Rank transformations and the power of the Student t test and Welch t test for non-normal populations with unequal variances. Canadian Journal of Experimental Psychology, 47(3): 523. Doi:10.1037/h0078850