TB-SMGAN: A GAN Based Hybrid Data Augmentation Framework on Chest X-ray Images and Reports

Abstract

Data augmentation is a common practice in image classification, employing methods such as reflection, random cropping, re-scaling, and transformations to enhance training data. These techniques are prevalent when working with extended real-world datasets, focusing on improving classification accuracy through increased diversity. The use of Generative Adversarial Networks (GANs), known for their high representational power, enables learning the distribution of real data and generating samples with previously unseen discriminative features. However, intra-class imbalances in augmentations are problematic for conventional GAN augmentations. Hence, we propose a framework named Text-Based Style-Manipulated GAN augmentation framework (TB-SMGAN) aims to leverage the generative capabilities of StyleGAN2-ADA. In this framework, we utilize StyleCLIP to control disentangled feature manipulations and intra-class imbalances. We enhance the efficiency of StyleCLIP by fine-tuning CLIP with x-ray images and information extractions from corresponding medical reports. Our proposed framework demonstrates an improvement in terms of mean PR-AUC score when employing the text-based manipulated GAN augmentation technique compared to conventional GAN augmentation.

Keywords

• Generative Augmentation
• Latent Space Manipulation
• Generative Adversarial Networks
• Chest x-ray Data Augmentation

References

1. Alqahtani, H., Kavakli-Thorne, M., & Kumar, G. (2021). Applications of generative adversarial networks (gans): An updated review. Archives of Computational Methods in Engineering, 28, 525-552. https://doi.org/10.1007/s11831-019-09388-y
2. Altwaijry, N. (2023). Probability-based synthetic minority oversampling technique. IEEE Access, 11, 28831-28839. https://doi.org/10.1109/ACCESS.2023.3260723
3. Benčević, M., Habijan, M., Galić, I., & Pizurica, A. (2022, August 29 - September 02). Self-supervised Learning as a Means to Reduce the Need for Labeled Data in Medical Image Analysis. In: Proceedings of the 30th European Signal Processing Conference (EUSIPCO 2022) (pp. 1328-1332). Belgrade, Serbia. https://doi.org/10.23919/EUSIPCO55093.2022.9909542
4. Bowles, C., Chen, L., Guerrero, R., Bentley, P., Gunn, R., Hammers, A., Dickie, D. A., Hernández, M. V., Wardlaw, J., & Rueckert, D. (2018). Gan augmentation: Augmenting training data using generative adversarial networks. https://doi.org/10.48550/arXiv.1810.10863
5. Dao, H. N., Quang, T. N., & Paik, I. (2022, October 26-28). Transfer Learning for Medical Image Classification on Multiple Datasets using PubMedCLIP. In: Proceedings of the 2022 IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia) (pp. 1-4). Yeosu, Korea, Republic of. https://doi.org/10.1109/ICCE-Asia57006.2022.9954669
6. Dao, H. N., Nguyen, T., Mugisha, C., & Paik, I. (2023). A Multimodal Transfer Learning Approach Using PubMedCLIP for Medical Image Classification. IEEE Access, 12, 75496-75507. https://doi.org/10.1109/ACCESS.2024.3401777
7. Deepshikha, K., & Naman, A. (2020). Removing Class Imbalance using Polarity-GAN: An Uncertainty Sampling Approach. https://doi.org/10.48550/arXiv.2012.04937
8. Fetty, L., Bylund, M., Kuess, P., Heilemann, G., Nyholm, T., Georg, D., & Löfstedt, T. (2020). Latent space manipulation for high-resolution medical image synthesis via the StyleGAN. Zeitschrift für Medizinische Physik, 30(4), 305-314. https://doi.org/10.1016/j.zemedi.2020.05.001

9. Frid-Adar, M., Diamant, I., Klang, E., Amitai, M., Goldberger, J., & Greenspan, H. (2018). GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification. Neurocomputing, 321, 321-331. https://doi.org/10.1016/j.neucom.2018.09.013
10. Hochberg, D. C., Greenspan, H., & Giryes, R. (2022). A self supervised StyleGAN for image annotation and classification with extremely limited labels. IEEE Transactions on Medical Imaging, 41(12), 3509-3519. https://doi.org/10.1109/TMI.2022.3187170
11. Honnibal, M., Montani, I., Van Landeghem, S., & Boyd, A. (2020). spaCy: Industrial-strength natural language processing in python.
12. Islam, S. M., & Mondal, H. S. (2019, July 06-08). Image Enhancement Based Medical Image Analysis. In: Proceedings of the 10th International Conference on Computing, Communication and Networking Technologies (ICCCNT) (pp. 1-5). Kanpur, India. https://doi.org/10.1109/ICCCNT45670.2019.8944910
13. Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., Seekins, J., Mong, D. A., Halabi, S. S., Sandberg, J. K., Jones, R., Larson, D. B., Langlotz, C. P., Patel, B. N., Lungren, M. P., & Ng, A. Y. (2019, January 27 - February 1). Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In: Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 33, No. 01, pp. 590-597). Honolulu, Hawaii, USA. https://doi.org/10.1609/aaai.v33i01.3301590
14. Jablonski, J. A., Angadi, S. S., Sharma, S., & Brown, D. E. (2022, March 10-11). Enabling Clinically Relevant and Interpretable Deep Learning Models for Cardiopulmonary Exercise Testing. In: Proceedings of the 2022 IEEE Healthcare Innovations and Point of Care Technologies (HI-POCT) (pp. 50-53). Houston, TX, USA. https://doi.org/10.1109/HI-POCT54491.2022.9744068
15. Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C., Mark, R. G., & Horng, S. (2019). MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data, 6(1), 317. https://doi.org/10.1038/s41597-019-0322-0
16. Kariuki, P. W., Gikunda, P. K., & Wandeto, J. M. (2023, April 14-16). Deep Transfer Learning Optimization Techniques for Medical Image Classification: A Review. In: Proceedings of the 2022 International Conference on Intelligent Computing and Machine Learning (2ICML) (pp. 7-15). Qingdao, China. https://doi.org/10.1109/2ICML58251.2022.00013
17. Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., & Aila, T. (2020, December 6-12). Training generative adversarial networks with limited data. In: H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, & H. Lin (Eds.) Proceedings of the 34th International Conference on Neural Information Processing Systems (NIPS'20) (pp. 12104-12114). Vancouver BC Canada.
18. Ke, B., Lu, H., Huo, W., & Wang, Y. (2022, July 22-24). Semi-supervised Medical Image Classification Combining Metric Pseudo-Label and Classification Pseudo-Label. In: P. Lin, & Y. Yang (Eds.) Proceedings of the 2022 International Conference on Computer Engineering and Artificial Intelligence (ICCEAI) (pp. 794-799). Shijiazhuang, China. https://doi.org/10.1109/ICCEAI55464.2022.00166
19. Kora Venu, S., & Ravula, S. (2020). Evaluation of Deep Convolutional Generative Adversarial Networks for Data Augmentation of Chest X-ray Images. Future Internet, 13(1), 8. https://doi.org/10.3390/fi13010008
20. Lacan, A., Sebag, M., & Hanczar, B. (2023). GAN-based data augmentation for transcriptomics: survey and comparative assessment. Bioinformatics, 39(S1), i111-i120. https://doi.org/10.1093/bioinformatics/btad239
21. Li, Z., Xia, P., Tao, R., Niu, H., & Li, B. (2022). A New Perspective on Stabilizing GANs Training: Direct Adversarial Training. IEEE Transactions on Emerging Topics in Computational Intelligence, 7(1), 178-189. https://doi.org/10.1109/TETCI.2022.3193373
22. Liu, L., Zhang, Y., & Sun, L. (2023). Medimatrix: innovative pre-training of grayscale images for rheumatoid arthritis diagnosis revolutionises medical image classification. Health Information Science and Systems, 11(1), 44. https://doi.org/10.1007/s13755-023-00246-7
23. Neumann, M., King, D., Beltagy, I., & Ammar, W. (2019, August 1). ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing. In: Dina D.-F., Kevin B. C., Sophia A., & J. Tsujii (Eds.) Proceedings of the 18th BioNLP Workshop and Shared Task (pp. 319-327). Florence, Italy. https://doi.org/10.18653/v1/W19-5034
24. Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., & Lischinski, D. (2021, October 10-17). StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 2065-2074). Montreal, QC, Canada. https://doi.org/10.1109/ICCV48922.2021.00209
25. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021, July 18-24). Learning transferable visual models from natural language supervision. In: M. Meila, & T. Zhang (Eds.) Proceedings of the 38th International Conference on Machine Learning, PMLR (pp. 8748-8763). https://doi.org/10.48550/arXiv.2103.00020
26. Shorten, C., & Khoshgoftaar, T. M. (2019). A survey on image data augmentation for deep learning. Journal of Big Data, 6(1), 60. https://doi.org/10.1186/s40537-019-0197-0
27. Sundaram, S., & Hulkund, N. (2021). GAN-based Data Augmentation for Chest X-ray Classification. https://doi.org/10.48550/arXiv.2107.02970
28. Tarawneh, A. S., Hassanat, A. B., Altarawneh, G. A., & Almuhaimeed, A. (2022). Stop oversampling for class imbalance learning: A review. IEEE Access, 10, 47643-47660. https://doi.org/10.1109/ACCESS.2022.3169512
29. Tov, O., Alaluf, Y., Nitzan, Y., Patashnik, O., & Cohen-Or, D. (2021). Designing an encoder for StyleGAN image manipulation. ACM Transactions on Graphics (TOG), 40(4), 133. https://doi.org/10.1145/3450626.3459838
30. Wang, Y., Ge, X., Ma, H., Qi, S., Zhang, G., & Yao, Y. (2021). Deep Learning in Medical Ultrasound Image Analysis: A Review. IEEE Access, 9, 54310-54324. https://doi.org/10.1109/ACCESS.2021.3071301
31. Wang, X., & Qi, G.-J. (2022). Contrastive Learning With Stronger Augmentations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5), 5549-5560. https://doi.org/10.1109/TPAMI.2022.3203630
32. Yuan, Z., Yan, Y., Sonka, M., & Yang, T. (2021, October 10-17). Large-scale Robust Deep AUC Maximization: A New Surrogate Loss and Empirical Studies on Medical Image Classification. In: Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (pp. 3040-3049). Montreal, QC, Canada. https://doi.org/10.1109/ICCV48922.2021.00303