A Comprehensive Monte Carlo Simulation Study on Multiple Comparison Methods after ANOVA

Abstract

Multiple comparison methods are applied to control the type I error rate at the nominal level. In this study, we investigate the performance of multiple comparison methods after analysis of variance (ANOVA) is implemented under different conditions. We include Bonferroni, Holm, Hochberg, Hommel, Benjamini-Hochberg (BH), and Benjamini-Yekutieli (BY) correction methods. Monte-Carlo simulation study is applied to assess their performances under different patterns, including sample size and group number combinations. Wide inferences are drawn on considered methods, and suggestions are provided for selecting appropriate methods. Moreover, the methods are implemented on three different types of real-life data sets to emphasize the importance of these correction methods in the research.

Keywords

• ANOVA
• Monte Carlo simulation
• pairwise comparison
• type I error rate
• variance homogeneity

References

1. S. K. Sarkar, C. K. Chang, The Simes method for multiple hypotheses testing with positively dependent test statistics, Journal of the American Statistical Association 92 (440) (2012) 1601–1608.
2. O. J. Dunn, Multiple comparisons among means, Journal of the American Statistical Association 56 (293) (1961) 52–64.
3. S. Holm, A simple sequentially rejective multiple test procedure, Scandinavian Journal of Statistics 6 (2) (1979) 65–70.
4. G. Hommel, A stagewise rejective multiple test procedure based on a modified Bonferroni test, Biometrika 75 (2) (1988) 383–386.
5. Y. Hochberg, A sharper Bonferroni procedure for multiple tests of significance, Biometrika 75 (4) (1988) 800–802.
6. Y. Benjamini, Y. Hochberg, Controlling the false discovery rate-a practical and powerful approach to multiple testing, Journal of the Royal Statistical Society Series B (Methodological) 57 (1) (1995) 289–300.
7. Y. Benjamini, D. Yekutieli, The control of the false discovery rate in multiple testing under dependency, The Annals of Statistics 29 (4) (2001) 1165–1188.
8. S. Lee, D. K. Lee, What is the proper way to apply the multiple comparison test?, Korean Journal of Anesthesiology 71 (5) (2018) 353–360.

9. R. Bender, S. Lange, Adjusting for multiple testing-when and how?, Journal of Clinical Epidemiology 54 (4) (2001) 343–349.
10. P. H. Westfall, J. F. Troendle, Multiple testing with minimal assumptions, Biometrical Journal 50 (5) (2008) 745–755.
11. R. E. Blakesley, S. Mazumdar, M. A. Dew, P. R. Houck, G. Tang, C. F. Reynolds, M. A. Butters, Comparisons of methods for multiple hypotheses testing in neuropsychological research, Neuropsychology 23 (2) (2009) 255–264.
12. V. B. Felix, A. F. B. Menezes, Comparisons of ten corrections methods for t-test in multiple comparisons via Monte Carlo study, Electronic Journal of Applied Statistical Analysis 11 (01) (2018) 74–91.
13. S. J. Staffa, D. Zurakowski, Strategies in adjusting for multiple comparisons: A primer for pediatric surgeons, Journal of Pediatric Surgery 55 (9) (2020) 1699–1705.
14. D. A. Dimitriev, E. V. Saperova, O. S. Indeykina, A. D. Dimitriev, Heart rate variability in mental stress: The data reveal regression to the mean, Data in Brief 22 (2019) 245–250.
15. S. S. Kharola, D. Gupta, A. Agrawal, Heart rate variability in mental stress: The data reveal regression to the mean, Indian Statistical Institute Bangalore Centre (2023) 19 pages.
16. A. A. Musicus, L. A. Gibson, S. L. Bellamy, J. A. Orr, D. Hammond, K. Glanz, K. G. Volpp, M. B. Schwartz, A. Bleakley, A. A. Strasser, C. A. Roberto, Effects of sugary beverage text and pictorial warnings: A randomized trial, American Journal of Preventive Medicine 64 (5) (2023) 716–727.
17. M. Giacalone, Z. Agata, P.C. Cozzucoli, A. Alibrandi, Bonferroni-Holm and permutation tests to compare health data: Methodological and applicative issues, BMC Medical Research Methodology 18 (81) (2018) 1–9.
18. S. Chen, Z. Feng, X. Yi, A general introduction to adjustment for multiple comparisons, Journal of Thoracic Disease 9 (6) (2017) 1725–1729.
19. R. J. Simes, An improved Bonferroni procedure for multiple tests of significance, Biometrika 73 (1986) 751–754.
20. O. Dag, N. A. B. Dolgun, N. M. Konar, Onewaytests: An R package for one-way tests in independent groups designs, The R Journal 10 (1) (2018) 175–199.
21. T. Hothorn, F. Bretz, P. Westfall, Simultaneous inference in general parametric models, Biometrical Journal 50 (3) (2008) 346–363.
22. P. Mair, R. Wilcox, Robust statistical methods in R using the WRS2 package, Behavior Research Methods 52 (2020) 464–488.
23. J. Fox, S. Weisberg, B. Price, carData: Companion to Applied Regression Data Sets, 2022.