QLoRA vs. LoRA: An Empirical Study of Performance–Efficiency Trade-offs in Fine-Tuning Mistral‑7B on Instruction‑Following Data

Abdullah Talha Kabakuş

doi:10.54287/gujsa.1937530

QLoRA vs. LoRA: An Empirical Study of Performance–Efficiency Trade-offs in Fine-Tuning Mistral‑7B on Instruction‑Following Data

Abstract

Parameter‑Efficient Fine‑Tuning (PEFT) approaches have emerged as a key strategy for efficiently adapting Large Language Models (LLMs) in resource‑constrained environments. Within this family of methods, LoRA and its quantized extension QLoRA provide effective alternatives by substantially decreasing the number of trainable parameters required for adaptation. In this study, a systematic comparison of LoRA and QLoRA is presented using a controlled experimental setup with identical base models and training data, with all experiments repeated across three random seeds and statistical significance assessed via paired t‑tests. Both approaches are evaluated across multiple dimensions, including generation quality (ROUGE‑L, Exact Match, token‑level F1, and the semantic metric BERTScore), training efficiency, and GPU memory consumption. Baseline (untrained) model performance is reported to quantify the gain from fine‑tuning, and loss curves are provided to confirm convergence. The results show that QLoRA not only reduces memory usage by approximately 70% but also achieves superior performance compared to LoRA, with improvements in token‑level F1 and BERTScore F1 reaching statistical significance (p < 0.05), while other lexical metrics showed consistent but non‑significant improvements. Furthermore, the performance–efficiency trade‑offs are analyzed, and it is demonstrated that QLoRA offers a more favorable balance between computational cost and model quality. These findings highlight QLoRA as a highly effective and practical fine‑tuning strategy for resource‑constrained environments, supported by a reproducible, statistically robust evaluation pipeline.

Keywords

References

Ben Zaken, E., Goldberg, Y., & Ravfogel, S. (2022). BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. In: S. Muresan, P. Nakov, & A. Villavicencio (Eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol. 2, pp. 1–9), (22-27 May 2022), Dublin, Ireland. https://doi.org/10.18653/v1/2022.acl-short.1
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., … Amodei, D. (2020). Language models are few-shot learners. In: H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, H. Lin (Eds.), Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS 2020) (pp. 1877–1901), (6-12 December 2020), Vancouver BC Canada. https://doi.org/10.48550/arXiv.2005.14165
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tai, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., … Wei, J. (2024). Scaling Instruction-Finetuned Language Models. Journal of Machine Learning Research, 25(1), 3381–3433. https://doi.org/10.48550/arXiv.2210.11416
Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. In: S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, & A. Oh (Eds.), Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS 2022) (pp. 30318 - 30332), (28 November - 9 December 2022), New Orleans LA USA. https://doi.org/10.48550/arXiv.2208.07339
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. In: A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Proceedings of the 37th International Conference on Neural Information Processing Systems (NeurIPS 2023), (pp. 10088–10115), (10-16 December 2023), New Orleans LA USA. https://doi.org/10.48550/arXiv.2305.14314
Ding, N., Qin, Y., Yang, G., Wei, F., Yang, Z., Su, Y., Hu, S., Chen, Y., Chan, C.-M., Chen, W., Yi, J., Zhao, W., Wang, X., Liu, Z., Zheng, H.-T., Chen, J., Liu, Y., Tang, J., Li, J., & Sun, M. (2023). Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3), 220–235. https://doi.org/10.1038/s42256-023-00626-4
Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2023). OPTQ: Accurate Quantization for Generative Pre-trained Transformers. In: Proceedings of the 11th International Conference on Learning Representations (ICLR 2023) (pp. 1–16), (1-5 May 2023), Kigali Rwanda.
Guan, Z. K., & Wang, Y. (2025). Evolving Paradigms in Task-Based Search and Learning: A Comparative Analysis of Traditional Search Engine with LLM-Enhanced Conversational Search System. https://doi.org/10.48550/arXiv.2512.00313

Hayou, S., Ghosh, N., & Yu, B. (2024). LoRA+: Efficient Low Rank Adaptation of Large Models. In: R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, & F. Berkenkamp (Eds.), Proceedings of the 41st International Conference on Machine Learning (ICML’24), (pp. 17783–17806), (21-27 July 2024), Vienna Austria. https://doi.org/10.48550/arXiv.2402.12354
He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., & Neubig, G. (2022). Towards a unified view of parameter-efficient transfer learning. In: Proceedings of the 10th International Conference on Learning Representations (ICLR 2022) (pp. 1–15), (25-29 April 2022). https://doi.org/10.48550/arXiv.2110.04366
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019). Parameter-efficient transfer learning for NLP. In: K. Chaudhuri, & R. Salakhutdinov (Eds.), Proceedings of the 36th International Conference on Machine Learning (ICML 2019) (pp. 1–10), (9-15 June 2019), Long Beach, California, USA. https://doi.org/10.48550/arXiv.1902.00751
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2022). LoRA: Low-Rank Adaptation of Large Language Models. In: Proceedings of the 10th International Conference on Learning Representations (ICLR 2022) (pp. 1–13), (25-29 April 2022). https://doi.org/10.48550/arXiv.2106.09685
Hu, K. (2023, February 2). ChatGPT sets record for fastest-growing user base - analyst note. Reuters. https://www.reuters.com/technology/chatgpt-sets-record-fastest-growing-user-base-analyst-note-2023-02-01/
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. de las, Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., & Sayed, W. El. (2023). Mistral 7B. https://doi.org/10.48550/arXiv.2310.06825
Lester, B., Al-Rfou, R., & Constant, N. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning. In: M.-F. Moens, X. Huang, L. Specia, & S. W.-t. Yih (Eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021) (pp. 3045–3059), (7-11 November 2021), Punta Cana, Dominican Republic. https://doi.org/10.18653/v1/2021.emnlp-main.243
Li, X. L., & Liang, P. (2021). Prefix-tuning: Optimizing continuous prompts for generation. In: C. Zong, F. Xia, W. Li, & R. Navigli (Eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021) (Vol. 1, pp. 4582–4597), (1-6 August 2021). https://doi.org/10.18653/v1/2021.acl-long.353
Liao, B., Meng, Y., & Monz, C. (2023). Parameter-Efficient Fine-Tuning without Introducing New Latency. In: A. Rogers, J. Boyd-Graber, & N. Okazaki (Eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023) (Volume 1: Long Papers), 1, (pp. 4242–4260), (10-12 July 2023), Toronto, Canada. https://doi.org/10.18653/v1/2023.acl-long.233
Lin, C.-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. In: Proceedings of the Workshop on Text Summarization Branches Out (WAS 2004), (pp. 74–81), (25–26 July 2004) Barcelona, Spain.
Loshchilov, I., & Hutter, F. (2019). Decoupled Weight Decay Regularization. In: Proceedings of the Seventh International Conference on Learning Representations (ICLR 2019), (pp. 1–18), (6-9 May 2019), New Orleans. https://doi.org/10.48550/arXiv.1711.05101
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback. In: S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, & A. Oh (Eds.), Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS 2022) (pp. 27730 - 27744), (28 November - 9 December 2022), New Orleans. https://doi.org/10.48550/arXiv.2203.02155
Pfeiffer, J., Kamath, A., Rücklé, A., Cho, K., & Gurevych, I. (2021). AdapterFusion: Non-destructive task composition for transfer learning. In: P. Merlo, J. Tiedemann, & R. Tsarfaty (Eds.), Proceedings of 16th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference (EACL 2021) (pp. 487–503), (19-23 April 2021). https://doi.org/10.18653/v1/2021.eacl-main.39
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21(1), 5485–5551. https://doi.org/10.48550/arXiv.1910.10683
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. In: J. Su, K. Duh, & X. Carreras (Eds.), Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2016) (pp. 2383–2392), (1-5 November 2016), Austin, Texas, USA. https://doi.org/10.18653/v1/d16-1264
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., & Hashimoto, T. B. (2023). Stanford Alpaca: An Instruction-following LLaMA Model. In GitHub repository. GitHub. https://github.com/tatsu-lab/stanford_alpaca
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., & Lample, G. (2023). LLaMA: Open and Efficient Foundation Language Models. https://doi.org/10.48550/arXiv.2302.13971
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. In: U. von Luxburg, I. Guyon, S. Bengio, H. Wallach, & R. Fergus (Eds.), Proceedings of the 31st Annual Conference on Neural Information Processing Systems (NIPS 2017), (4-9 December 2017), Long Beach California USA. https://doi.org/10.48550/arXiv.1706.03762
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., & Han, S. (2023). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. In: A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, & J. Scarlett (Eds.), Proceedings of the 40th International Conference on Machine Learning (ICML’23) (pp. 38087 – 38099), (23-29 July 2023), Honolulu, Hawaii, USA. https://doi.org/10.48550/arXiv.2211.10438
Xu, L., Xie, H., Qin, S.-Z. J., Tao, X., & Wang, F. L. (2026). Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(6), 6107–6126. https://doi.org/10.1109/TPAMI.2026.3657354
Zhang, Q., Chen, M., Bukharin, A., He, P., Cheng, Y., Chen, W., & Zhao, T. (2023). Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. In: Proceedings of the 11th International Conference on Learning Representations (ICLR 2023) (pp. 1–17), (1-5 May 2023), Kigali Rwanda. https://doi.org/10.48550/arXiv.2303.10512
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2020). BERTScore: Evaluating Text Generation with BERT. In: Proceedings of the 8th International Conference on Learning Representations (ICLR 2020) (pp. 1–43), (26-30 April 2020), Addis Ababa, Ethiopia. https://doi.org/10.48550/arXiv.1904.09675

Details

Primary Language

English

Subjects

Deep Learning, Natural Language Processing

Journal Section

Research Article

Authors

Abdullah Talha Kabakuş ^*
0000-0003-2181-4292
Türkiye

Early Pub Date

September 23, 2026

Publication Date

September 30, 2026

Submission Date

April 24, 2026

Acceptance Date

July 16, 2026

Published in Issue

Year 2026 Volume: 13 Number: 3

DOI

https://doi.org/10.54287/gujsa.1937530

IZ

https://izlik.org/JA26YF59KF

Cite

RIS / Bibtex

APA

Kabakuş, A. T. (2026). QLoRA vs. LoRA: An Empirical Study of Performance–Efficiency Trade-offs in Fine-Tuning Mistral‑7B on Instruction‑Following Data. Gazi University Journal of Science Part A: Engineering and Innovation, 13(3), 1100-1118. https://doi.org/10.54287/gujsa.1937530

AMA

1.Kabakuş AT. QLoRA vs. LoRA: An Empirical Study of Performance–Efficiency Trade-offs in Fine-Tuning Mistral‑7B on Instruction‑Following Data. GU J Sci, Part A. 2026;13(3):1100-1118. doi:10.54287/gujsa.1937530

Chicago

Kabakuş, Abdullah Talha. 2026. “QLoRA Vs. LoRA: An Empirical Study of Performance–Efficiency Trade-Offs in Fine-Tuning Mistral‑7B on Instruction‑Following Data”. Gazi University Journal of Science Part A: Engineering and Innovation 13 (3): 1100-1118. https://doi.org/10.54287/gujsa.1937530.

EndNote

Kabakuş AT (September 1, 2026) QLoRA vs. LoRA: An Empirical Study of Performance–Efficiency Trade-offs in Fine-Tuning Mistral‑7B on Instruction‑Following Data. Gazi University Journal of Science Part A: Engineering and Innovation 13 3 1100–1118.

IEEE

[1]A. T. Kabakuş, “QLoRA vs. LoRA: An Empirical Study of Performance–Efficiency Trade-offs in Fine-Tuning Mistral‑7B on Instruction‑Following Data”, GU J Sci, Part A, vol. 13, no. 3, pp. 1100–1118, Sept. 2026, doi: 10.54287/gujsa.1937530.

ISNAD

Kabakuş, Abdullah Talha. “QLoRA Vs. LoRA: An Empirical Study of Performance–Efficiency Trade-Offs in Fine-Tuning Mistral‑7B on Instruction‑Following Data”. Gazi University Journal of Science Part A: Engineering and Innovation 13/3 (September 1, 2026): 1100-1118. https://doi.org/10.54287/gujsa.1937530.

JAMA

1.Kabakuş AT. QLoRA vs. LoRA: An Empirical Study of Performance–Efficiency Trade-offs in Fine-Tuning Mistral‑7B on Instruction‑Following Data. GU J Sci, Part A. 2026;13:1100–1118.

MLA

Kabakuş, Abdullah Talha. “QLoRA Vs. LoRA: An Empirical Study of Performance–Efficiency Trade-Offs in Fine-Tuning Mistral‑7B on Instruction‑Following Data”. Gazi University Journal of Science Part A: Engineering and Innovation, vol. 13, no. 3, Sept. 2026, pp. 1100-18, doi:10.54287/gujsa.1937530.

Vancouver

1.Abdullah Talha Kabakuş. QLoRA vs. LoRA: An Empirical Study of Performance–Efficiency Trade-offs in Fine-Tuning Mistral‑7B on Instruction‑Following Data. GU J Sci, Part A. 2026 Sep. 1;13(3):1100-18. doi:10.54287/gujsa.1937530