Virtual Hematoxylin and Eosin (H&E) Staining of Medical Images using Generative Adversarial Networks

Year 2024, Volume: 36 Issue: 4, 326 - 336

Musa Aydın

https://doi.org/10.7240/jeps.1530421

Abstract

Virtual staining of medical images is an essential approach in digital pathology. Traditional tissue staining is a time-consuming, specialized, and intensive process where staining varies from expert to expert. By using a deep learning approach, virtual staining improves image quality and reduces the costs associated with manual staining. In this study, we investigate the application of a deep neural network based on the Conditional Generative Adversarial Network (cGAN) for virtual staining of unstained whole slide images (WSI-Whole Slide Images) to Hematoxylin and Eosin (H&E) stained image pairs. Using a publicly available dataset, we compare our results with a reference work using a larger dataset. Using only seven WSIs, our approach performs competitively in terms of Structural Similarity Index (SSIM), Peak Signal-to-Noise Ratio (PSNR), and Pearson Correlation Coefficient (PCC) compared to the reference work using 68 WSIs. In addition, in our study, the hybrid loss function we proposed in the training process of the adversarial generative network was used to evaluate synthetic and real images. The average SSIM, PSNR, and PCC values obtained in our study are 0.668, 21.487, and 0.872, respectively, while in the reference study, these values are 0.724, 22.609, and 0.903, respectively. The results demonstrate the potential of GANs to acquire high-quality virtual staining images, reducing the need for extensive datasets and thus increasing efficiency and reproducibility for digital pathology.

Keywords

Hematoxylin and Eosin (H&E), Deep Neural Networks, Medical Images Virtual Staining, Generative Adversarial Networks (GANs)

Çekişmeli Üretici Ağlar Kullanılarak Medikal Görüntülerin Sanal Hematoksilen ve Eozin (H&E) Boyanması

Abstract

Tıbbi görüntülerin sanal boyanması işlemi dijital patolojide önemli bir yaklaşım olarak görülmektir. Geleneksel doku boyama zaman alan, uzmanlık gerektiren, boyamanın uzmandan uzmana değişkenlik gösterdiği yoğun bir süreçtir. Derin öğrenme yaklaşımı kullanılarak sanal boyama ile görüntü kalitesinin iyileştirilmesi ve manuel boyamadan kaynaklı maliyetlerin azaltılması sağlanmaktadır. Bu çalışmada, boyamasız tüm slayt görüntülerin (WSI-Whole Slide Images) Hematoksilen ve Eozin (H&E) boyalı görüntü çiftlerini sanal olarak boyamak için koşullu çekişmeli üretici ağ (cGAN- The Conditional Generative Adversarial Network) tabanlı bir derin sinir ağının uygulanmasını araştırmaktadır. Açık kaynak olarak sunulan bir veri setini kullanarak, sonuçlarımızı daha büyük bir veri seti kullanan bir referans çalışmayla karşılaştırıyoruz. Sadece yedi adet WSI kullanan yaklaşımımız, 68 WSI kullanan referans çalışmayla karşılaştırıldığında Yapısal Benzerlik İndeksi (SSIM), Tepe Sinyal-Gürültü Oranı (PSNR) ve Pearson Korelasyon Katsayısı (PCC) açısından rekabetçi bir performans göstermektedir. Ayrıca çalışmamızda çekişmeli üretici ağın eğitim sürecinde önerdiğimiz hibrit kayıp fonksiyonu ile sentetik görüntüler ve gerçek görüntülerin değerlendirilmesi sağlanmıştır. Çalışmamızda elde edilen sonuçlar SSIM, PSNR ve PCC değerlerinin ortalaması sırasıyla 0,668, 21,487 ve 0,872 iken, referans çalışmada bu değerler sırasıyla 0,724, 22,609 ve 0,903 olarak hesaplanmıştır. Elde edilen sonuçlar, GAN'ların yüksek kaliteli sanal boyama görüntülerini elde etme potansiyelini ortaya koyarak, kapsamlı veri kümelerine olan ihtiyacı azaltmaktadır ve böylece dijital patoloji için verimlilik ve tekrar edilebilirliği artırmaktadır.

Keywords

Hematoksilen ve Eozin (H&E), Derin Sinir Ağları, Medikal Görüntü Sanal Boyama, Çekişmeli Üreti Ağlar (GANs)

References

• J. D. Martina, C. Simmons, and D. M. Jukic, “High-definition hematoxylin and eosin staining in a transition to digital pathology,” J Pathol Inform, vol. 2, no. 1, 2011, doi: 10.4103/2153-3539.86284.
• M. G. Hanna et al., “Integrating digital pathology into clinical practice,” 2022. doi: 10.1038/s41379-021-00929-0.
• K. Bera, K. A. Schalper, D. L. Rimm, V. Velcheti, and A. Madabhushi, “Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology,” Nat Rev Clin Oncol, vol. 16, no. 11, 2019, doi: 10.1038/s41571-019-0252-y.
• V. Baxi, R. Edwards, M. Montalto, and S. Saha, “Digital pathology and artificial intelligence in translational medicine and clinical practice,” 2022. doi: 10.1038/s41379-021-00919-2.
• R. Colling et al., “Artificial intelligence in digital pathology: a roadmap to routine use in clinical practice,” Journal of Pathology, vol. 249, no. 2, 2019, doi: 10.1002/path.5310.
• B. Acs and J. Hartman, “Next generation pathology: artificial intelligence enhances histopathology practice,” 2020. doi: 10.1002/path.5343.
• H. Reza Tizhoosh and L. Pantanowitz, “Artificial intelligence and digital pathology: Challenges and opportunities,” J Pathol Inform, vol. 9, no. 1, 2018, doi: 10.4103/jpi.jpi_53_18.
• T. M. Abraham et al., “Label- and slide-free tissue histology using 3D epi-mode quantitative phase imaging and virtual hematoxylin and eosin staining,” Optica, vol. 10, no. 12, 2023, doi: 10.1364/optica.502859.
• S. Koivukoski, U. Khan, P. Ruusuvuori, and L. Latonen, “Unstained Tissue Imaging and Virtual Hematoxylin and Eosin Staining of Histologic Whole Slide Images,” Laboratory Investigation, vol. 103, no. 5, 2023, doi: 10.1016/j.labinv.2023.100070.
• G. Wolflein, I. H. Um, D. J. Harrison, and O. Arandjelovic, “HoechstGAN: Virtual Lymphocyte Staining Using Generative Adversarial Networks,” in Proceedings - IEEE Winter Conference on Applications of Computer Vision, WACV 2023,doi: 10.1109/WACV56688.2023.00497.
• A. Yilmaz, T. Aydin, and R. Varol, “Virtual staining for pixel-wise and quantitative analysis of single cell images,” Sci Rep, vol. 13, no. 1, 2023, doi: 10.1038/s41598-023-45150-y.
• S. Biswas and S. Barma, “Feature Fusion GAN Based Virtual Staining on Plant Microscopy Images,” IEEE/ACM Trans Comput Biol Bioinform, 2024, doi: 10.1109/TCBB.2024.3380634.
• K. Sun, Z. Chen, G. Wang, J. Liu, X. Ye, and Y. G. Jiang, “Bi-directional Feature Fusion Generative Adversarial Network for Ultra-high Resolution Pathological Image Virtual Re-staining,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2023. doi: 10.1109/CVPR52729.2023.00380.
• A. Golfe, R. del Amor, A. Colomer, M. A. Sales, L. Terradez, and V. Naranjo, “ProGleason-GAN: Conditional progressive growing GAN for prostatic cancer Gleason grade patch synthesis,” Comput Methods Programs Biomed, vol. 240, 2023, doi: 10.1016/j.cmpb.2023.107695.
• K. de Haan et al., “Deep learning-based transformation of H&E stained tissues into special stains,” Nat Commun, vol. 12, no. 1, 2021, doi: 10.1038/s41467-021-25221-2.
• X. Meng, X. Li, and X. Wang, “A Computationally Virtual Histological Staining Method to Ovarian Cancer Tissue by Deep Generative Adversarial Networks,” Comput Math Methods Med, vol. 2021, 2021, doi: 10.1155/2021/4244157.
• Y. Zhang, K. de Haan, Y. Rivenson, J. Li, A. Delis, and A. Ozcan, “Digital synthesis of histological stains using micro-structured and multiplexed virtual staining of label-free tissue,” Light Sci Appl, vol. 9, no. 1, 2020, doi: 10.1038/s41377-020-0315-y.
• J. Boschman et al., “The utility of color normalization for AI-based diagnosis of hematoxylin and eosin-stained pathology images,” Journal of Pathology, vol. 256, no. 1, 2022, doi: 10.1002/path.5797.
• T. A. Azevedo Tosta, P. R. de Faria, L. A. Neves, and M. Z. do Nascimento, “Computational normalization of H&E-stained histological images: Progress, challenges and future potential,” 2019. doi: 10.1016/j.artmed.2018.10.004.
• S. Vijh, M. Saraswat, and S. Kumar, “A new complete color normalization method for H&E stained histopatholgical images,” Applied Intelligence, vol. 51, no. 11, 2021, doi: 10.1007/s10489-021-02231-7.
• A. Janowczyk, A. Basavanhally, and A. Madabhushi, “Stain Normalization using Sparse AutoEncoders (StaNoSA): Application to digital pathology,” Computerized Medical Imaging and Graphics, vol. 57, 2017, doi: 10.1016/j.compmedimag.2016.05.003.
• M. Z. Hoque, A. Keskinarkaus, P. Nyberg, and T. Seppänen, “Stain normalization methods for histopathology image analysis: A comprehensive review and experimental comparison,” Information Fusion, vol. 102, 2024, doi: 10.1016/j.inffus.2023.101997.
• J. Vasiljević, F. Feuerhake, C. Wemmert, and T. Lampert, “HistoStarGAN: A unified approach to stain normalisation, stain transfer and stain invariant segmentation in renal histopathology,” Knowl Based Syst, vol. 277, 2023, doi: 10.1016/j.knosys.2023.110780.
• U. Khan, S. Koivukoski, M. Valkonen, L. Latonen, and P. Ruusuvuori, “The effect of neural network architecture on virtual H&E staining: Systematic assessment of histological feasibility,” Patterns, vol. 4, no. 5, 2023, doi: 10.1016/j.patter.2023.100725.
• I. J. Goodfellow et al., “Generative adversarial nets,” in Advances in Neural Information Processing Systems, 2014. doi: 10.1007/978-3-658-40442-0_9.
• R. Yan et al., “Unpaired virtual histological staining using prior-guided generative adversarial networks,” Computerized Medical Imaging and Graphics, vol. 105, 2023, doi: 10.1016/j.compmedimag.2023.102185.
• H. Zhang, V. Sindagi, and V. M. Patel, “Image De-Raining Using a Conditional Generative Adversarial Network,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 11, 2020, doi: 10.1109/TCSVT.2019.2920407.
• Z. Wang, et al., “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, 2004, doi: 10.1109/TIP.2003.819861.
• D. R. I. M. Setiadi, “PSNR vs SSIM: imperceptibility quality assessment for image steganography,” Multimed Tools Appl, vol. 80, no. 6, 2021, doi: 10.1007/s11042-020-10035-z.
• H. Rahadian, S. Bandong, A. Widyotriatmo, and E. Joelianto, “Image encoding selection based on Pearson correlation coefficient for time series anomaly detection,” Alexandria Engineering Journal, vol. 82, 2023, doi: 10.1016/j.aej.2023.09.070.
• O. Ciga, T. Xu, et al., “Overcoming the limitations of patch-based learning to detect cancer in whole slide images,” Sci Rep, vol. 11, no. 1, 2021, doi: 10.1038/s41598-021-88494-z.
• Y. Tian, D. Su, S. Lauria, and X. Liu, “Recent advances on loss functions in deep learning for computer vision,” 2022. doi: 10.1016/j.neucom.2022.04.127.
• A. Salar and A. Ahmadi, “Improving loss function for deep convolutional neural network applied in automatic image annotation,” Visual Computer, vol. 40, no. 3, 2024, doi: 10.1007/s00371-023-02873-3.
• O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Lecture Notes in Computer Science, 2015. doi: 10.1007/978-3-319-24574-4_28.
• Bijie Bai, et al. Label-Free Virtual HER2 Immunohistochemical Staining of Breast Tissue using Deep Learning. BME Front. 2022; DOI:10.34133/2022/9786242