Analysis of the Impact of RGB-to-Achromatic Color Space Transformations on Single-Image Superresolution Performance

Abstract

Super-resolution techniques are employed to enhance the quality of digital images. Color spaces are developed to model colors in various digital environments. In the literature, several studies suggest that applying color space transformations and subsequently employing super-resolution techniques on the transformed images improve image quality. This study analyzes the impact of color space trans-formations on super-resolution applications. The analysis is conducted by performing the super-resolution process entirely in the RGB color space, followed by converting the obtained result into a different color space and comparing the quality metrics. The findings reveal that it is possible to achieve higher scores by converting RGB images into YCbCr or CIELab color spaces, despite no actual improvement in perceived image quality. Our experiments involve applying image enhancement techniques solely within the RGB color space, converting the results into alternative color spaces, and comparing them with ground truth images in Set5, Set14, BSDS100, Urban100, and DIV2K. Working in color spaces other than RGB does not lead to significant visual quality improvement. Our experiments demonstrate that solely through color space conversion, traditional metrics such as PSNR and SSIM, as well as deep learning-based metrics like DISTS and A-DISTS, can yield higher scores. Therefore, the observed improvements in quality metrics resulting from color space transformations may be misleading and may not reflect actual enhancements in image fidelity. With the A-DISTS metric that evaluates human perception, our study examines not only the impact of transformations from RGB to alternative color spaces on metrics but also evaluates the alignment of these metrics with human perception, an area that has received limited attention in the literature.

Keywords

• Single-image super-resolution
• Achromatic axis
• Color space conversion
• Image enhancement

References

1. Agustsson E, Timofte R. 2017. Ntire 2017 challenge on single image super-resolution: Dataset and study. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, Honolulu, HI, USA, July 21-26, pp: 126–135.
2. Bevilacqua M, Roumy A, Guillemot C, Alberi-Morel ML. 2012. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: Proceedings of the 23rd British Machine Vision Conference (BMVC), Surrey, UK, September 3-7, pp: 1–10.
3. Bosse S, Maniry D, Müller KR, Wiegand T, Samek W. 2017. Deep neural networks for no-reference and full-reference image quality assessment. IEEE Trans Image Process, 27(1): 206–219.
4. Burger W, Burge MJ. 2008. Introduction to spectral techniques. In: Principles of Digital Image Processing: Core Algorithms. Springer, London, UK, 1st ed, pp: 313–342.
5. Candès EJ, Fernandez-Granda C. 2014. Towards a mathematical theory of super-resolution. Commun Pure Appl Math, 67(6): 906–956.
6. Conde MV, Choi UJ, Burchi M, Timofte R. 2023. Swin2SR: Swinv2 transformer for compressed image super-resolution and restoration. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops, Tel Aviv, Israel, October 23–27, pp: 669–687.
7. Ding K, Liu Y, Zou X, Wang S, Ma K. 2021. Locally Adaptive Structure and Texture Similarity for Image Quality Assessment. In: Proceedings of the 29th ACM International Conference on Multimedia, Virtual Event, China, October 20-24, pp: 2483–2491.
8. Ding K, Ma K, Wang S, Simoncelli EP. 2020. Image quality assessment: Unifying structure and texture similarity. IEEE Trans Pattern Anal Mach Intell, 4(5): 2567–2581.

9. Dong C, Loy CC, He K, Tang X. 2016. Image Super-Resolution Using Deep Convolutional Networks. IEEE Trans Pattern Anal Mach Intell, 38(2): 295–307.
10. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv Prepr ArXiv201011929.
11. Dosselmann R, Yang XD. 2005. Existing and emerging image quality metrics. In: Canadian Conference on Electrical and Computer Engineering, Saskatoon, SK, Canada, May 1-4, pp: 1906–1913.
12. Erdemir E, Dragotti P.L, Gündüz D. 2020. Privacy-aware time-series data sharing with deep reinforcement learning. IEEE Trans Inf Forensics Secur, 16: 389–401.
13. Fadnavis S. 2014. Image interpolation techniques in digital image processing: an overview. Int J Eng Res Appl, 4(10): 70–73.
14. Getreuer P. 2011. Linear methods for image interpolation. Image Process Line, 1: 238–259.
15. Gong R, Wang Y, Cai Y, Shao X. 2017. How to deal with color in super resolution reconstruction of images. Opt Express, 25(10): 11144–11156.
16. Han D. 2013. Comparison of commonly used image interpolation methods. In: Proceedings of the 2nd International Conference on Computer Science and Electronics Engineering, Hangzhou, China, March 22-23, pp: 1556–1559.
17. Huang JB, Singh A, Ahuja N. 2015. Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Boston, MA, USA, pp: 5197–5206.
18. John N, Viswanath A, Sowmya V, Soman KP. 2016. Analysis of various color space models on effective single image super resolution. In: Berretti S, Thampi S.M, Srivastava P.R, editors. Intelligent Systems Technologies and Applications. Springer, Cham, Switzerland, 1st ed, pp: 529–540.
19. Jolicoeur-Martineau A. 2018. The relativistic discriminator: a key element missing from standard GAN. ArXiv Prepr ArXiv180700734.
20. Keles O, Yilmaz, MA, Tekalp AM, Korkmaz C, Dogan Z. 2021. On the Computation of PSNR for a Set of Images or Video. In: Picture Coding Symposium, Bristol, UK, June 29 - July 2, pp: 1–5.
21. Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp: 4681–4690.
22. Liang J, Cao J, Sun G, Zhang K, Van Gool L, Timofte R. 2021. SwinIR: Image restoration using swin transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada, October 11-17, pp: 1833–1844.
23. Liu Z, Hu H, Lin Y, Yao Z, Xie Z, Wei Y. 2022. Swin transformer v2: Scaling up capacity and resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, New Orleans, LA, USA, June 18-24, pp: 12009–12019.
24. Ma J, Tang L, Fan F, Huang J, Mei X, Ma Y. 2022. SwinFusion: Cross-domain long-range learning for general image fusion via swin transformer. IEEECAA J Autom Sin, 9(7): 1200–1217.
25. Martin D, Fowlkes C, Tal D, Malik J. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings Eighth IEEE International Conference on Computer Vision, Vancouver, BC, Canada, pp: 416–423.
26. Nilsson J, Akenine Möller T. 2020. Understanding ssim. ArXiv Prepr ArXiv200613846.
27. Popat K, Picard RW. 1997. Cluster-based probability model and its application to image and texture processing. IEEE Trans Image Process, 6(2): 268–284.
28. Ronneberger O, Fischer P, Brox T. 2015. U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, October 5-9, pp: 234–241.
29. Sheikh H.R, Bovik A.C. 2006. Image information and visual quality. IEEE Trans Image Process, 15(2): 430–444.
30. Su H, Li Y, Xu Y, Fu X, Liu S. 2024. A review of deep-learning-based super-resolution: From methods to applications. Pattern Recognit, 157: 110935.
31. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN. 2017. Attention is all you need. Adv Neural Inf Process Syst, 30.
32. Wang X, Xie L, Dong C, Shan Y. 2021. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada, October 11-17, pp: 1905–1914.
33. Wang X, Yu K, Wu S, Gu J, Liu Y, Dong C. 2018. Esrgan: Enhanced super-resolution generative adversarial networks. In: Proceedings of the European conference on computer vision (ECCV) workshops, Munich, Germany, September 8-14, pp: 0–0.
34. Wang Y, Liu W, Sun W, Meng X, Yang G, Ren K. 2023. A progressive feature enhancement deep network for large-scale remote sensing image super-resolution. IEEE Trans Geosci Remote Sens, 61(1): 1–13.
35. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process, 13(4): 600–612.
36. Wang Z, Chen J, Hoi SCH. 2021. Deep Learning for Image Super-Resolution: A Survey. IEEE Trans Pattern Anal Mach Intell, 43(10): 3365–3387.
37. Xu Z, Ma Q, Yuan F. 2020. Single color image super-resolution using sparse representation and color constraint. J Syst Eng Electron, 31(2): 266–271.
38. Xue W, Zhang L, Mou X, Bovik AC. 2013. Gradient magnitude similarity deviation: A highly efficient perceptual image quality index. IEEE Trans Image Process, 23(2): 684–695.
39. Yang Y, Yuhua P, Zhaoguang L. 2007. A fast algorithm for YCbCr to RGB conversion. IEEE Trans Consum Electron, 53(4): 1490–1493.
40. Yilmaz İ, Güllü M, Baybura T, Erdoğan AO. 2002. Renk Uzayları ve Renk Dönüşüm Programı (RDP). Afyon Kocatepe Üniversitesi Fen Ve Mühendis Bilim Derg, 2(2): 19–35.
41. Zeyde R, Elad M, Protter M. 2012. On single image scale-up using sparse-representations. In: Curves and Surfaces: 7th International Conference, Avignon, France, pp: 711–730.
42. Zhang Lin, Zhang Lei, Mou X, Zhang D. 2011. FSIM: A feature similarity index for image quality assessment. IEEE Trans Image Process, 20(8): 2378–2386.
43. Zhang R, Isola P, Efros AA, Shechtman E, Wang O. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Salt Lake City, UT, USA, June 18-23, pp: 586–595.