Single-Image Super-Resolution Analysis in DCT Spectral Domain

Abstract

Advances in deep learning techniques have lead to drastic changes in contemporary methods used for a variety of computer vision problems. Single-image super-resolution is one of these problems that has been significantly and positively influenced by these trends. The mainstream state-of-the-art methods for super-resolution learn a non-linear mapping from low-resolution images to high-resolution images in the spatial domain, parameterized through convolution and transposed-convolution layers. In this paper, we explore the use of spectral representations for deep learning based super-resolution. More specifically, we propose an approach that operates in the space of discrete cosine transform based spectral representations. Additionally, to reduce the artifacts resulting from spectral processing, we propose to use a noise reduction network as a post-processing step. Notably, our approach allows using a universal super-resolution model for a range of scaling factors. We evaluate our approach in detail through quantitative and qualitative results.

Keywords

• deep learning
• Super resolution
• image process

References

1. R. Timofte, V. De Smet, and L. Van Gool, “A+: Adjusted anchored neighborhood regression for fast super-resolution,” in Asian Conference on Computer Vision. Springer, 2014, pp. 111–126.
2. J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE International Conference on Image Processing, vol. 19, no. 11, pp. 2861–2873, 2010.
3. S. Schulter, C. Leistner, and H. Bischof, “Fast and accurate image up- scaling with super-resolution forests,” in IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3791–3799.
4. C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 2, pp. 295–307, 2016.
5. J. Kim, J. Kwon Lee, and K. Mu Lee, “Accurate image super-resolution using very deep convolutional networks,” in IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 1646–1654.
6. W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Deep laplacian pyramid networks for fast and accurate super-resolution,” arXiv preprint arXiv:1704.03915, 2017.
7. S. Anwar, S. Khan, and N. Barnes, “A deep journey into super- resolution: A survey,” arXiv preprint arXiv:1904.07523, 2019.
8. O. Rippel, J. Snoek, and R. P. Adams, “Spectral representations for convolutional neural networks,” in Advances in Neural Information Processing Systems, 2015, pp. 2449–2457.

9. Y. Wang, C. Xu, S. You, D. Tao, and C. Xu, “Cnnpack: Packing convolutional neural networks in the frequency domain,” in Advances in Neural Information Processing Systems, 2016, pp. 253–261.
10. N. Kumar, R. Verma, and A. Sethi, “Convolutional neural networks for wavelet domain super resolution,” Pattern Recognition Letters, vol. 90, pp. 65–71, 2017.
11. J.Li, S.You, and A.Robles-Kelly, “A frequency domain neural network for fast image super-resolution,” in International Joint Conference on Neural Networks. IEEE, 2018, pp. 1–8.
12. S. Xue, W. Qiu, F. Liu, and X. Jin, “Faster image super-resolution by improved frequency-domain neural networks,” Signal, Image and Video Processing, pp. 1–9, 2019.
13. C. Dong, C. C. Loy, and X. Tang, “Accelerating the super-resolution convolutional neural network,” in European Conference on Computer Vision. Springer, 2016, pp. 391–407.
14. C. Ledig, L. Theis, F. Husza ́r, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang et al., “Photo-realistic single image super-resolution using a generative adversarial network,” arXiv preprint arXiv:1609.04802, 2016.
15. T. Dai, J. Cai, Y. Zhang, S.-T. Xia, and L. Zhang, “Second-order atten- tion network for single image super-resolution,” in IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 11 065–11 074.
16. Y. Wang, F. Perazzi, B. McWilliams, A. Sorkine-Hornung, O. Sorkine- Hornung, and C. Schroers, “A fully progressive approach to single-image super-resolution,” in IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 864–873.
17. A. V. Oppenheim, Discrete-time signal processing. Pearson Education India, 1999.
18. K. R. Rao and P. Yip, Discrete cosine transform: algorithms, advantages, applications. Academic press, 2014.
19. R. Clarke, “Relation between the karhunen loeve and cosine transforms,” in IEEE Proceedings (Communications, Radar and Signal Processing), vol. 128, no. 6. IET, 1981, pp. 359–360.
20. N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting.” Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, 2014.
21. X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in International Conference on Artificial Intelligence and Statistics, 2010, pp. 249–256.
22. D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
23. C. Dong, Y. Deng, C. Change Loy, and X. Tang, “Compression artifacts reduction by a deep convolutional network,” in IEEE International Conference on Computer Vision, 2015, pp. 576–584.
24. M. Bevilacqua, A. Roumy, C. Guillemot, and M. L. Alberi-Morel, “Low- complexity single-image super-resolution based on nonnegative neighbor embedding,” 2012.
25. W. Shi, J. Caballero, F. Husza ́r, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-time single image and video super- resolution using an efficient sub-pixel convolutional neural network,” in IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 1874–1883.
26. P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, “Contour detection and hierarchical image segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 5, pp. 898–916, 2010.
27. J.-B. Huang, A. Singh, and N. Ahuja, “Single image super-resolution from transformed self-exemplars,” in IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 5197–5206.
28. Zhou Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, April 2004.