Deep Learning Based Color and Style Transfer: A Review and Challenges

Abstract

Deep learning methods have been applied in many fields in recent years, and successful results have been obtained. Image processing is one of these areas. One of the image processing applications using deep learning is color and style transfer. Color and style transfer is aimed at transferring the color and texture from the source image to another image (the target image). In color transfer, the colors in the source image are transferred, while in style transfer, texture is transferred as well as color. In the literature, color transfer has been studied for many years, and traditional methods such as PCA have been used in addition to deep learning. On the other hand, studies on style transfer are relatively new and mostly use deep learning methods. In this study, color and style transfer studies in the literature were examined. The methods used in these studies are mentioned, and the current problems in this field are shared.

Keywords

• Color Transfer
• Neural Style Transfer
• Image Colorization
• Image Recoloring
• CNN.

References

1. [1] Q. Cai, M. Ma, C. Wang, and H. Li, “Image Neural Style Transfer: A Review,” Computers and Electrical Engineering, vol. 108, p. 108723, 2023.
2. [2] L. Jiao and J. Zhao, “A survey on the new generation of deep learning in image processing,” IEEE Access, vol. 7, pp. 172231–172263, 2019.
3. [3] “Neural Style Transfer, Image Color Transfer.” Accessed: Sep. 15, 2024. [Online]. Available: https://www.scopus.com/search/
4. [4] L. A. Gatys, A. S. Ecker, and M. Bethge, “Image Style Transfer Using Convolutional Neural Networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2414–2423.
5. [5] Y. Jing, Y. Yang, Z. Feng, J. Feng, Y. Yu, and M. Song, “Neural Style Transfer: A Review,” IEEE Transactions on Visualization and Computer Graphics, vol. 26, no. 11, pp. 3365–3385, 2019.
6. [6] A. Singh, V. Jaiswal, G. Joshi, A. Sanjeeve, S. Gite, and K. K., “Neural Style Transfer: A Critical Review,” IEEE Access, vol. 9, pp. 131583–131613, 2021.
7. [7] G. Sohaliya and K. Sharma, “An Evolution of Style Transfer from Artistic to Photorealistic: A Review,” in 2021 Asian Conference on Innovation in Technology, ASIANCON 2021, Institute of Electrical and Electronics Engineers Inc., Aug. 2021. doi: 10.1109/ASIANCON51346.2021.9544924.
8. [8] J. W. Johnson, “Towards the Algorithmic Detection of Artistic Style,” International Journal of Advanced Computer Science and Applications, vol. 10, no. 1, pp. 76–81, 2019.

9. [9] L. A. Gatys, A. S. Ecker, and M. Bethge, “A Neural Algorithm of Artistic Style,” arXiv preprint arXiv:1508.06576, 2015.
10. [10] L. Zhang, Y. Ji, X. Lin, and C. Liu, “Style Transfer for Anime Sketches with Enhanced Residual U-Net and Auxiliary Classifier GAN,” in 2017 4th IAPR Asian Conference on Pattern Recognition (ACPR), 2017, pp. 506–511.
11. [11] B. Karadağ, A. Arı, and M. Karadağ, “Derin Öğrenme Modellerinin Sinirsel Stil Aktarımı Performanslarının Karşılaştırılması,” Journal of Polytechnic, vol. 24, no. 4, pp. 1611–1622, 2021, doi: 10.2339/politeknik.885838.
12. [12] J. Lian and J. Cui, “Anime Style Transfer with Spatially-Adaptive Normalization,” in 2021 IEEE International Conference on Multimedia and Expo (ICME), 2021, pp. 1–6.
13. [13] L. JinKua, C. Yang, and H. B. Abdalla, “Enhanced Style Transfer with Colorization and Super-Resolution,” in 2022 7th International Conference on Communication, Image and Signal Processing (CCISP), 2022, pp. 166–172.
14. [14] Z. Ke, Y. Liu, L. Zhu, N. Zhao, and R. W. Lau, “Neural Preset for Color Style Transfer,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 14173–14182.
15. [15] J. J. Virtusio, J. J. Ople, D. S. Tan, T. M., N. Kumar, and K. L. Hua, “Neural Style Palette: A Multimodal and Interactive Style Transfer from a Single Style Image,” IEEE Transactions on Multimedia, vol. 23, pp. 2245–2258, 2021.
16. [16] Y. Deng et al., “Stytr2: Image Style Transfer with Transformers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 11326–11336.
17. [17] X. Fu, “Digital Image Art Style Transfer Algorithm Based on CycleGAN,” Computational Intelligence and Neuroscience, vol. 2022, no. 1, p. 6075398, 2022.
18. [18] S.-H. Huang, J.-A. An, D. Wei, J. Luo, and H. Pfister, “QuantArt: Quantizing Image Style Transfer Towards High Visual Fidelity,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 5947–5956.
19. [19] T. T. Fang, D. M. Vo, A. Sugimoto, and S.-H. Lai, “Stylized-Colorization for Line Arts,” in 2020 25th International Conference on Pattern Recognition (ICPR), 2021, pp. 2033–2040.
20. [20] C. Zheng and Y. Zhang, “Two-Stage Color Ink Painting Style Transfer via Convolution Neural Network,” in 2018 15th International Symposium on Pervasive Systems, Algorithms and Networks (I-SPAN), 2018, pp. 193–200.
21. [21] F. Luan, S. Paris, E. Shechtman, and K. Bala, “Deep Photo Style Transfer,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 4990–4998.
22. [22] X. Wang, G. Oxholm, D. Zhang, and Y.-F. Wang, “Multimodal Transfer: A Hierarchical Deep Convolutional Neural Network for Fast Artistic Style Transfer,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 5239–5247.
23. [23] J. Cui, “Image Style Migration Algorithm Based on HSV Color Model,” in 2022 IEEE International Conference on Advances in Electrical Engineering and Computer Applications (AEECA), 2022, pp. 111–114.
24. [24] Y. Liao and Y. Huang, “Deep Learning-Based Application of Image Style Transfer,” Mathematical Problems in Engineering, vol. 2022, no. 1, p. 1693892, 2022.
25. [25] X. Liu, X. Li, M.-M. Cheng, and P. Hall, “Geometric Style Transfer,” arXiv preprint arXiv:2007.05471, 2020.
26. [26] X. Han, Y. Wu, and R. Wan, “A Method for Style Transfer from Artistic Images Based on Depth Extraction Generative Adversarial Network,” Applied Sciences, vol. 13, no. 2, p. 867, 2023.
27. [27] C. Lv, D. Zhang, S. Geng, Z. Wu, and H. Huang, “Color Transfer for Images: A Survey,” ACM Transactions on Multimedia Computing, Communications and Applications, vol. 20, no. 8, pp. 1–29, 2024.
28. [28] L. Bao, K. Panetta, and S. Agaian, “Fast Color Transfer for Camouflage Applications,” in 2017 IEEE International Symposium on Technologies for Homeland Security (HST), 2017, pp. 1–5.
29. [29] S. Liu, “An Overview of Color Transfer and Style Transfer for Images and Videos,” arXiv preprint arXiv:2204.13339, 2022.
30. [30] E. Reinhard, M. Adhikhmin, B. Gooch, and P. Shirley, “Color Transfer Between Images,” IEEE Computer Graphics and Applications, vol. 21, no. 5, pp. 34–41, 2001.
31. [31] A. Abadpour and S. Kasaei, “An Efficient PCA-Based Color Transfer Method,” Journal of Visual Communication and Image Representation, vol. 18, no. 1, pp. 15–34, 2007.
32. [32] X. An and F. Pellacini, “User-Controllable Color Transfer,” in Computer Graphics Forum, 2010, pp. 263–271.
33. [33] B. Arbelot, R. Vergne, T. Hurtut, and J. Thollot, “Local Texture-Based Color Transfer and Colorization,” Computer & Graphics, vol. 62, pp. 15–27, 2017.
34. [34] B. Xu, X. Liu, C. Lu, T. Hong, and Y. Zhu, “Transferring the Color Imagery from an Image: A Color Network Model for Assisting Color Combination,” Color Research and Application, vol. 44, no. 2, pp. 205–220, 2019.
35. [35] C. Gu, X. Lu, and C. Zhang, “Example-Based Color Transfer with Gaussian Mixture Modeling,” Pattern Recognition, vol. 129, p. 108716, 2022.
36. [36] X. Xiao and L. Ma, “Gradient-Preserving Color Transfer,” in Computer Graphics Forum, 2009, pp. 1879–1886.
37. [37] J. Lee, H.-Y. Son, G. Lee, J. Lee, S.-H. Cho, and S. Lee, “Deep Color Transfer Using Histogram Analogy,” Visual Computer, vol. 36, pp. 2129–2143, 2020.
38. [38] J. Yin, Y.-C. Huang, B.-H. Chen, and S.-Z. Ye, “Color Transferred Convolutional Neural Networks for Image Dehazing,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 11, pp. 3957–3967, 2019.
39. [39] D. Liu, Y. Jiang, M. Pei, and S. Liu, “Emotional image color transfer via deep learning,” Pattern Recognition Letters, vol. 110, pp. 16–22, Jul. 2018, doi: 10.1016/j.patrec.2018.03.015.
40. [40] M. Zhang, J. Liao, and J. Yu, “Deep exemplar-based color transfer for 3d model,” IEEE Trans Vis Computers & Graphics, vol. 28, no. 8, pp. 2926–2937, 2020.
41. [41] Q. C. Tian and L. D. Cohen, “Histogram-Based Color Transfer for Image Stitching,” Journal of Imaging, vol. 3, no. 3, p. 38, 2017.
42. [42] Y. Qian, D. Liao, and J. Zhou, “Manifold Alignment Based Color Transfer for Multiview Image Stitching,” in Proceedings of the IEEE International Conference on Image Processing, 2013.
43. [43] Z. Ding, P. Li, Q. Yang, S. Li, and Q. Gong, “Regional Style and Color Transfer,” in 2024 5th International Conference on Computer Vision, Image and Deep Learning (CVIDL), 2024, pp. 593–597.
44. [44] I. Luengo, E. Flouty, P. Giataganas, P. Wisanuvej, J. Nehme, and D. Stoyanov, “SurReal: Enhancing surgical simulation realism using style transfer,” arXiv preprint arXiv:1811.02946, 2018.
45. [45] W. H. Png, Y. Aun, and M. L. Gan, “FeaST: Feature-guided Style Transfer for high-fidelity art synthesis,” Computers & Graphics, p. 103975, 2024.