Çok başlı dikkat mekanizmalarında anahtar ve değer boyutlarının rolünün değerlendirilmesi

Abstract

Bu çalışma, dönüştürücü tabanlı sinirsel makine çevirisi modelleri üzerinde kritik hiperparametrelerin etkisini analiz etmektedir. Çalışmanın temel amacı, altı temel mimari parametrenin—kaynak ve hedef gömme boyutları, anahtar ve değer boyutları, dikkat başı sayısı ve dizi uzunluğu—ana etkilerini değerlendirmektir. Çalışma, çoklu dikkat mekanizması içindeki asimetrik anahtar ve değer boyutlarının etkisine özellikle vurgu yapmaktadır. Bulgular, eşdeğer anahtar ve değer boyutlarının oluşturulmasının istikrarlı bir varsayılan değer olarak işlev görmesine rağmen, dilsel bağlamlar genelinde optimum performansı garanti etmediğini göstermektedir. Modelin asimetrik yapılandırmalara verdiği tepki, dile özgüydü ve anahtar boyutu ile dikkat başı sayısı arasındaki etkileşimde dikkate değer bir değişkenlik gözlemlendi. Ayrıca, hedef gömme boyutunun artırılması genellemeyi tutarlı bir şekilde iyileştirirken, daha büyük bir kaynak gömme boyutu tüm dil çiftlerinde aşırı uyuma yol açmıştır. Bu etkiler, eğitim doğruluğunu artıran ancak aynı zamanda doğrulama performansını düşüren belirli parametre kombinasyonları yoluyla ortaya çıkmış ve dile bağlı genelleme sınırlamalarını yansıtmıştır. Bu sonuçlar, Transformer'ın optimal performansının evrensel bir hiperparametre yapılandırmasıyla elde edilemeyeceğini, bunun yerine her veri setinin dilbilimsel özelliklerine göre uyarlanması gerektiğini vurgulamaktadır.

Keywords

• Transformer Mimarisi
• Çok Başlı Dikkat
• Anahtar Boyut
• Değer Boyutu
• Hiperparametre Optimizasyonu
• Asimetrik Boyutlar
• Derin Öğrenme

Supporting Institution

Sivas Bilim ve Teknoloji Üniversitesi Bilimsel Araştırma Projeleri Koordinasyon Birimi

Project Number

2024-DTP-Müh-0004

Ethical Statement

Bu çalışma için etik kurul onayı gerekmemiştir, çünkü çalışma ne hayvanlar ne de insanlar üzerinde gerçekleştirilmiştir.

Thanks

Bu çalışma, Sivas Bilim ve Teknoloji Üniversitesi Bilimsel Araştırma Projeleri Koordinasyon Birimi tarafından desteklenmiştir. Proje Numarası: 2024-DTP-Müh-0004. Bu çalışmada kullanılan hesaplama kaynakları, 5020092024 numaralı hibe kapsamında Türkiye Ulusal Yüksek Performanslı Hesaplama Merkezi (UHeM) tarafından sağlanmıştır. Araştırmada, TUBİTAK ULAKBİM Yüksek Performanslı ve Grid Hesaplama Merkezi (TRUBA) ile Sivas Bilim ve Teknoloji Üniversitesi Lütfi Albay Yapay Zeka ve Robotik Laboratuvarı tarafından sağlanan hesaplama kaynakları kullanılmıştır.

Assessing the role of key and value dimensionality in multi-head attention mechanisms

Abstract

Context—The transformer architecture has occupied a prominent position in recent research due to its attention-driven design; however, the role of asymmetric key and value dimensions within its attention mechanism remains largely underexplored in the literature despite their potential impact on model behavior and performance.

Objective—This study aims to systematically analyze the effects of key architectural parameters in the Transformer architecture on neural machine translation performance. These parameters include asymmetric key and value dimensions within the multi-head attention mechanism, embedding dimension, number of attention heads and sentence length.

Method—This study analyses the architectural components of the Transformer model across three neural machine translation datasets: English–Turkish, French–Turkish, and German–Turkish. Six fundamental hyperparameters—source and target embedding dimensions, key and value dimensions, the number of attention heads, and sentence length—were assessed at two levels each. To address the computational complexity of the Transformer's modular structure, a fractional factorial experimental design was employed, reducing the 64 possible configurations to 16 experimental runs. This approach significantly reduced the use of computational resources while ensuring a comprehensive analysis of both individual parameters and their pairwise interactions. All models were trained for 150 epochs using PyTorch with Distributed Data Parallel (DDP) and Automated Mixed Precision (AMP) in an HPC environment with NVIDIA A100 GPUs. Finally, statistical evaluation and performance visualization were conducted using Minitab.

Results—The results show that while establishing equivalent key and value dimensions serves as a stable default, it does not guarantee optimal performance across linguistic contexts. The model’s response to asymmetrical configurations was highly language-specific, with notable variability observed in the interaction between the key dimension and the number of attention heads. Furthermore, increasing the target embedding dimension consistently improved generalization, whereas a larger source embedding dimension led to overfitting across all language pairs. These effects resulted from specific combinations of parameters that improved training accuracy but reduced validation performance, reflecting limitations in the generalization of language-dependent processes.

Conclusion—These results emphasize that optimal Transformer performance cannot be achieved through a universal hyperparameter configuration and must instead be tailored to the linguistic characteristics of each dataset.

Keywords

• Asymmetrical Dimensions
• Deep Learning
• Hyperparameter Optimization
• Key dimension
• Multi-Head Attention
• Neural machine translation
• Transformer Architecture
• Value dimension

Supporting Institution

This work has been supported by the Scientific Research Projects Coordination Unit of the Sivas University of Science and Technology.

Project Number

2024-DTP-Müh-0004

Ethical Statement

Ethics committee approval was not required for this study because of there was no study on animals or humans.

Thanks

This work has been supported by the Scientific Research Projects Coordination Unit of the Sivas University of Science and Technology. Project Number: 2024-DTP-Müh-0004. Computing resources used in this work were provided by the National Center for High Performance Computing of Turkey (UHeM) under grant number 5020092024. The research utilized computational resources provided by the TUBITAK ULAKBIM High Performance and Grid Computing Center (TRUBA) and the Lütfi Albay Artificial Intelligence and Robotics Laboratory at Sivas University of Science and Technology.

References

1. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, “Attention is all you need”, Proceedings of the 31st Annual Conference on Neural Information Processing Systems (NIPS), Long Beach, CA, USA, 04-09 December 2017.
2. Y. Zhang, M. X. Tuo, Q. Y. Yin, L. Qi, X. X. Wang, T. Liu, “Keywords extraction with deep neural network model”, Neurocomputing, 383, 113–121, 2020. https://doi.org/10.1016/j.neucom.2019.11.083.
3. J. Devlin, M. W. Chang, K. Lee, K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, Proceedings of the 17th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Minneapolis, MN, USA, 02-07 June 2019, 4171-4186.
4. Y. Guan, J. Whitehill, “Transformer-Encoder Trees for Efficient Multilingual Machine Translation and Speech Translation”, arXiv, 2025. https://doi.org/10.48550/arXiv.2509.17930.
5. J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, S. Sanghai, “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints”, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), Singapore, Singapore, 06-10 December 2023, 4895-4901. https://doi.org/10.18653/v1/2023.emnlp-main.298.
6. N. Kalchbrenner, P. Blunsom, “Recurrent continuous translation models”, Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP 2013), Seattle, Washington, USA, 18-21 October 2013, 1700-1709. https://doi.org/10.18653/v1/d13-1176.
7. I. Sutskever, O. Vinyals, Q. V. Le, “Sequence to sequence learning with neural networks”, Proceedings of the 28th International Conference on Neural Information Processing Systems (NIPS 2014), Montreal, Canada, 08-13 December 2014.
8. D. Bahdanau, K. Cho, Y. Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate”, arXiv, 2014. https://doi.org/10.48550/arXiv.1409.0473.

9. D. Friedman, A. Wettig, D. Chen, “Learning Transformer Programs”, arXiv, 2023. https://doi.org/10.48550/arXiv.2306.01128.
10. P. Michel, O. Levy, G. Neubig, “Are sixteen heads really better than one?”, Proceedings of the 33rd International Conference on Neural Information Processing Systems (NIPS 2019), Vancouver, Canada, 08-14 December 2019.
11. E. Voita, D. Talbot, F. Moiseev, R. Sennrich, I. Titov, “Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned”, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, 28 July-02 August 2019, 5797-5808. https://doi.org/10.18653/v1/P19-1580.
12. L. Meng, M. Goodwin, A. Yazidi, P. Engelstad, “A Manifold Representation of the Key in Vision Transformers”, Proceedings of the 41st Computer Graphics International Conference (CGI Annual), Geneve, Switzerland, 01-05 July 2024, 311-322. https://doi.org/10.1007/978-3-031-82021-2_22.
13. Y. Lu, X. Zhou, W. He, J. Zhao, T. Ji, T. Gui, Q. Zhang, X. J. Huang, “LongHeads: Multi-Head Attention is Secretly a Long Context Processor”, arXiv, 2024. https://doi.org/10.48550/arXiv.2402.10685.
14. P. Jin, B. Zhu, L. Yuan, S. C. Yan, “MoH: Multi-Head Attention as Mixture-of-Head Attention”, arXiv, 2025. https://doi.org/10.48550/arXiv.2410.11842.
15. T. Ji, B. Guo, Y. Wu, Q. P. Guo, L. X. Shen, Z. Chen, X. P. Qiu, Q. Zhang, T. Gui, “Towards Economical Inference: Enabling DeepSeek’s Multi-Head Latent Attention in Any Transformer-based LLMs”, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL Annual), Vienna, Austria, 27 July-01 August 2025, 33313–33328. https://doi.org/10.18653/v1/2025.acl-long.1597.
16. B. İşlek, R. Katırcı, H. Celik, “Enhancing Question Answering Systems Through Optimal Hyperparameter Tuning in GRU”, 8th International Artificial Intelligence and Data Processing Symposium (IDAP), Malatya, Türkiye, September, 21-22 September 2024. https://doi.org/10.1109/IDAP64064.2024.10710732.
17. G. A. Lujan-Moreno, P. R. Howard, O. G. Rojas, D. C. Montgomery, “Design of experiments and response surface methodology to tune machine learning hyperparameters, with a random forest case-study”, Expert Systems with Applications, 109, 195–205, 2018. https://doi.org/10.1016/j.eswa.2018.05.024.
18. W. Pannakkong, K. Thiwa-anont, K. Singthong, P. Parthanadee, J. Buddhakulsomsiri, “Hyperparameter Tuning of Machine Learning Algorithms Using Response Surface Methodology: A Case Study of ANN, SVM , and DBN”, Mathematical Problems in Engineering, 2022, 8513719, 2022. https://doi.org/10.1155/2022/8513719.
19. S. Islam, H. Elmekki, A. Elsebai, J. Bentahar, N. Drawel, G. Rjoub, W. Pedrycz, “A comprehensive survey on applications of transformers for deep learning tasks”, Expert Systems with Applications, 241, 122666, 2024. https://doi.org/10.1016/j.eswa.2023.122666.
20. H. Çelik, R. Katırcı, B. İşlek, “Effect Of Parameters On Performance In Question-Answer Model With Simple Rnn Deep Learning Method”, International Conference on Scientific and Innovation Research-III, Sivas, Türkiye, 03-05 May 2024. https://doi.org/10.5281/zenodo.11320381.
21. R. Katırcı, H. Çelik, “Learning Rate Sensitivity In Transformer Models: A Case Study In Neural Machine Translation”, International Conference on Scientific and Innovation Research-IV, Sivas, Türkiye, 30-31 May 2025. https://doi.org/10.5281/zenodo.15769066.
22. N. Shazeer, M. Stern, “Adafactor: Adaptive learning rates with sublinear memory cost”, Proceedings of 35th International Conference on Machine Learning (ICML), Stockholm, Sweden, 10-15 July 2018.
23. L. Y. Liu, H. M. Jiang, P. C. He, W. Z. Chen, X. D. Liu, J. F. Gao, J. W. Han, “On the Variance of the Adaptive Learning Rate and Beyond”, The Eighth International Conference on Learning Representations (ICLR 2020), Virtual Only Conference, 26 April-1 May 2020. https://doi.org/10.48550/arXiv.1908.03265.
24. K. Shen, J. L. Guo, X. Tan, S. L. Tang, R. Wang, J. Bian, “A Study on ReLU and Softmax in Transformer”, arXiv, 2023. https://doi.org/10.48550/arXiv.2302.06461.
25. H. S. Fang, J. U. Lee, N. S. Moosavi, I. Gurevych, “Transformers with Learnable Activation Functions”, Proceeding of the 17th Conference of the European-Chapter of the Association-for-Computational-Linguistics (EACL), Dubrovnik, Croatia, 02-06 May 2023, 2337-2353. https://doi.org/10.18653/v1/2023.findings-eacl.181.