Deep learning microscopy image enhancement for a line scanning confocal microscope

Year 2025, Volume: 67 Issue: 2, 135 - 152, 24.12.2025

Amirmohammad Ketabchi Nima Bavili Berna Morova

https://doi.org/10.33769/aupse.1621490

Abstract

Confocal microscopy provides images with a higher resolution and signal-to-noise ratio in comparison with wide-field illumination microscopy by eliminating the out-of-focus background. Besides the advances in confocal microscopy, new techniques like light-sheet microscopy, structured illumination microscopy, line-scanning confocal microscopy, or deep learning enhanced microscopy draw attention due to their advantages including higher image acquisition speed, higher contrast, and better resolution, and being more cost- effective. In this work, we first demonstrate the operation of a line-scanning confocal microscope developed using a digital light projector (DLP) and a CMOS camera with a rolling shutter. In this method, a series of illumination lines are projected on a sample by a DLP and focusing objective (50X, NA=0.55). The reflected light is imaged with a rolling shutter CMOS camera. The line-scanning confocal imaging is achieved by overlapping the illumination lines and the rolling shutter of the sensor. Significant improvement of ∼52% in image contrast is obtained using this technique. Then by using this setup, a dataset including 500 pairs of images of fibers of a tissue paper is prepared. This dataset is employed for training a Convolutional Neural Network (CNN) and a conditional Generative Adversarial Network (cGAN) for deep learning enhanced microscopy. 95% of the dataset is used for the training of the net- work, while the remaining 5% is used for the validation. ∼45% and ∼47% contrast improvement was measured in the test images for CNN and cGAN, respectively, comparable to the ground-truth images.

Keywords

Microscopy , deep learning , confocal microscopy , CNN , cGAN

Project Number

118F529

References

• Jonkman J, et al. Tutorial: guidance for quantitative confocal microscopy. Nature Protocols 2020; 15(5): p. 1585-1611. https://doi.org/10.1038/s41596-020-0313-9.
• Paddock SW. Principles and practices of laser scanning confocal microscopy. Molecular Biotechnology 2000; 16(2): p. 127-149. https://doi.org/10.1385/MB:16:2:127.
• Oreopoulos J, Berman R, and Browne M. Spinning-disk confocal microscopy: present technology and future trends. Methods in Cell Biology 2014; 123: p. 153-175. https://doi.org/10.1016/B978-0-12-420138-5.00009-4.
• Gareau DS , Abeytunge S, and Rajadhyaksha M, Line-scanning reflectance confocal microscopy of human skin: comparison of full-pupil and divided-pupil configurations. Optics Letters 2009; 34(20): p. 3235-3237. https://doi.org/10.1364/OL.34.003235.
• Muller MS, Elsner AE, Ozawa GY. Non-Mydriatic Confocal Retinal Imaging Using a Digital Light Projector. Proceedings of SPIE-The International Society for Optical Engineering 2013 Mar 26;8567. https://doi.org/10.1117/12.2001305.
• Deng L, Hinton G, Kingsbury B. New types of deep neural network learning for speech recognition and related applications: an overview. 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada; 2013. pp. 8599-8603, https://doi.org/10.1109/ICASSP.2013.6639344.
• Zeiler MD, Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012. https://doi.org/10.48550/arXiv.1212.5701.
• Renuka DK, Hamsapriya T, Chakkaravarthi MR, Surya PL. Spam Classification Based on Supervised Learning Using Machine Learning Techniques. 2011 International Conference on Process Automation, Control and Computing, Coimbatore, India; 2011. pp. 1-7, https://doi.org/10.1109/PACC.2011.5979035.
• Gardner-Medwin AR, Barlow HB. The limits of counting accuracy in distributed neural representations. Neural Computation 2001 Mar; 13(3):477-504. https://doi.org/10.1162/089976601300014420.
• Kaelbling LP, Littman ML, Moore AW. Reinforcement learning: A survey. Journal of Artificial Intelligence Research 1996; 4: pp.237-285. https://doi.org/10.1613/jair.301.
• Lee WH, Ozger M, Challita U, Sung KW. Noise Learning-Based Denoising Autoencoder. IEEE Communications Letters 2021; 25(9): pp. 2983-2987. https://doi.org/10.1109/LCOMM.2021.3091800.
• Vincent P, Larochelle H, Bengio Y, and Manzagol PA. Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th International Conference on Machine Learning (ICML) 2008; pp. 1096-1103. https://doi.org/10.1145/1390156.1390294.
• Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, and Bengio Y. Generative adversarial networks. Communications of the ACM 2016; 63(11), 139- 144. https://doi.org/10.1145/3422622.
• Creswell A, White T, Dumoulin V, Arulkumaran K, Sengupta B, Bharath AA. Generative Adversarial Networks: An Overview, in IEEE Signal Processing Magazine 2018; 35(1), pp. 53-65. https://doi.org/10.1109/MSP.2017.2765202.
• Li S, Li W, Cook C, Zhu C, Gao Y. Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition 2018; pp. 5457-5466. https://doi.org/10.1109/CVPR.2018.00572.
• Yu Y, Si X, Hu C, Zhang J. A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures. Neural Computation 2019; 31(7):1235-1270. https://doi.org/10.1162/neco a 01199.
• De K, Masilamani V. Image Sharpness Measure for Blurred Images in Frequency Domain. Procedia Engineering 2013; 64: pp. 149-158. https://doi.org/10.1016/j.proeng.2013.09.086. DEEP LEARNING MICROSCOPY IMAGE ENHANCEMENT 151
• Sultana F, Sufian A, Dutta P. Advancements in Image Classification using Convolutional Neural Network. 2018 Fourth International Conference on Research in Computational Intelligence and Communication Networks (ICRCICN), Kolkata, India; 2018. pp. 122-129, https://doi.org/10.1109/ICRCICN.2018.8718718.
• Ghiasi-Shirazi K. Competitive Cross-Entropy Loss: A Study on Training Single-Layer Neural Networks for Solving Nonlinearly Separable Classification Problems. Neural Processing Letters 2019; 50: p.1115–1122. https://doi.org/10.1007/s11063-018-9906-5.
• Bera S, Shrivastava VK. Analysis of various optimizers on deep convolutional neural network model in the application of hyperspectral remote sensing image classification. International Journal of Remote Sensing 2019; 41(7): pp.2664–2683. https://doi.org/10.1080/01431161.2019.1694725.
• Webb S. Deep learning for biology. Nature 2018; 554(7690): p. 555-558. https://doi.org/10.1038/d41586-018-02174-z.
• Agrawal A, Choudhary A. Deep materials informatics: Applications of deep learning in materials science. MRS Communications 2019; 9:779–792. https://doi.org/10.1557/mrc.2019.73.
• Wang F, Casalino LP, Khullar D. Deep Learning in Medicine—Promise, Progress, and Challenges. JAMA Intern Medicine 2019; 179(3):293–294. https://doi.org/10.1001/jamainternmed.2018.7117.
• Li J, Tong G, Pan Y, Yu Y. Spatial and temporal super-resolution for fluorescence microscopy by a recurrent neural network. Optics Express 2021; 29: 15747-15763. https://doi.org/10.1364/OE.423892.
• Wang H, Rivenson Y, Jin Y, Wei Z, Gao R, Gunaydin H, Bentolila LA, Kural C, Ozcan A. Deep learning enables cross-modality super-resolution in fluorescence microscopy. Nature Methods 2019; 16(1):103-110. https://doi.org/10.1038/s41592-018-0239-0.
• Rivenson Y, Göröcs Z, Günaydin H, Zhang Y, Wang H, Ozcan A. Deep learning microscopy. Optica 2017; 4:1437-1443. https://doi.org/10.1364/OPTICA.4.001437.
• Christensen CN, Ward EN, Lu M, Lio P, Kaminski CF. ML-SIM: universal reconstruction of structured illumination microscopy images using transfer learning. Biomedical Optics Express 2021; 12(5):2720-2733. https://doi.org/10.1364/BOE.414680.
• Midtvedt B, Helgadottir A, Argun A, Pineda J, Midtvedt D, Volpe G. Quantitative digital microscopy with deep learning. Applied Physics Reviews 2021; 8(1): 011310. https://doi.org/10.1063/5.0034891.
• Ketabchi AM, Morova B, Uysalli Y, Aydin M, Eren F, Bavili N, Pysz D, Buczynski R, Kiraz A. Enhancing resolution and contrast in fibre bundle-based fluorescence microscopy using generative adversarial network. Journal of Microscopy 2024; 295, 236–242. https://doi.org/10.1111/jmi.13296.
• Ketabchi AM, Morova B, Bavili N, Kiraz A. Contrast improvement through a generative adversarial network (GAN) by utilizing a dataset obtained from a line-scanning confocal microscope. SPIE Photonics Europe 2024; 12998. https://doi.org/10.1117/12.3016368.
• Guo M, Wu Y, Hobson CM et al. Deep learning-based aberration compensation improves contrast and resolution in fluorescence microscopy. Nature Communications 2025; 16, 313. https://doi.org/10.1038/s41467-024-55267-x.
• Cui Z, Xing Y, Chen Y, Zheng X, Liu W, Kuang C, Chen Y. Real-Time Resolution Enhancement of Confocal Laser Scanning Microscopy via Deep Learning. Photonics 2024; 11, 983. https://doi.org/10.3390/photonics11100983.
• Shuai Liu, Wenzhen Zou, Hao Sha, Xiaochen Feng, Bin Chen, Jian Zhang, Sanyang Han, Xiu Li, and Yongbing Zhang, ”Deep learning-enhanced snapshot hyperspectral confocal microscopy imaging system, Optics Express 2024; 32, 13918-13931. https://doi.org/10.1364/OE.519045. 152 A. KETABCHI, N. BAVILI, AND B. MOROVA
• Muller MS, Green JJ, Baskaran K, Ingling AW, Clendenon JL, Gast TJ, Elsner AE. Non-mydriatic confocal retinal imaging using a digital light projector. Proceedings of SPIE - The International Society for Optical Engineering 2015;9376:93760E. https://doi.org/10.1117/12.2077704.
• Isola P, Zhu JY, Zhou T, Efros AA. Image-to-Image Translation with Conditional Adversarial Networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA; 2017. pp. 5967-5976. https://doi.org/10.1109/CVPR.2017.632.
• Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y. Generative adversarial networks. Communications of the ACM 2020; 63(11):139–144. https://doi.org/10.1145/3422622.
• Morova B et al., Fabrication and characterization of large numerical aperture, high-resolution optical fiber bundles based on high-contrast pairs of soft glasses for fluorescence imaging. Optics Express 2019; 27(7): p.9502-9515. https://doi.org/10.1364/OE.27.009502.