Structure-Texture Decomposition of RGB-D Images

Abstract

In this paper, we study the problem of separating texture from structure in RGB-D images. Our structure preserving image smoothing operator is based on the region covariance smoothing (RCS) method in [16] that we present a number of modifications to this framework to make it depth-aware and increase its effectiveness. In particular, we propose to incorporate three geometric depth features, namely height above ground, angle with gravity and horizontal disparity to the pool of image features used in that study. We also suggest to use a new kernel function based on KL-divergence between the distributions of extracted features. We demonstrate our approach on challenges images from NYU-Depth v2 Dataset [24], achieving more accurate decompositions than the state-of-the-art approaches which do not utilize any depth information.

Keywords

• RGB-D images,structure-preserving smoothing,image decomposition,region covariances

References

1. B. Arbelot, R. Vergne, T. Hurtut, and J. Thollot, “Automatic texture guided color transfer and colorization”, in Proc. Expresive'16, 2016
2. J.-F. Aujol, G. Gilboa, T. Chan, and S. Osher, “Structure-texture image decomposition–modeling, algorithms, and parameter selection”, International Journal of Computer Vision, vol. 67, issue 1, Apr. 2006, pp. 111–136.
3. A. Buades, B. Coll, and J.-M. Morel, “A non-local algorithm for image denoising”, in Proc. CVPR’05, vol. 2, 2005, pp. 60–65.
4. M. Camplani, S. Hannuna, M. Mirmehdi, D. Damen, A. Paiement, L. Tao, and T. Burghardt, “Real-time RGB-D tracking with depth scaling kernelised correlation filters and occlusion handling”, in Proc. BMVC’15, 2015.
5. H. Cho, Hyunjoon Lee, H. Kang, and S. Lee, “Bilateral texture filtering”, ACM Transactions on Graphics (TOG) – Proc. of ACM SIGGRAPH 2014, vol. 33, issue 4, July 2014.
6. Z. Deng, S. Todorovic, and L.J. Latecki, “Semantic Segmentation of RGBD Images with Mutex Constraints”, in Proc. ICCV’15, 2015.
7. K. Desingh, K.M. Krishna, D. Rajan, and C. Jawahar, “Depth really matters: Improving visual salient region detection with depth”, in Proc. BMVC’13, 2013.
8. P. Dollar, and C. L. Zitnick, “Structured Forests for Fast Edge Detection”, in Proc. ICCV'13, 2013.

9. F. Durand, and J. Dorsey, “Fast bilateral filtering for the display of high-dynamic-range images”. ACM Transactions on Graphics (TOG) – Proc. of ACM SIGGRAPH 2001, vol. 21, issue 3, Jul. 2002, 257–266.
10. Z. Farbman, R. Fattal, D. Lischinski, and R. Szeliski, “Edge-preserving decompositions for multi-scale tone and detail manipulation”, ACM Transactions on Graphics (TOG) – Proc. of ACM SIGGRAPH 2008, vol. 27, issue 3, Aug. 2008.
11. S. Gupta, R. Girshick, P. Arbeláez, J. Malik, “Learning rich features from RGB-D Images for object detection and segmentation”, in Proc. ECCV’14, 2014.
12. K. He, J. Sun, and X. Tang, “Guided image filtering”, in Proc. ECCV’10, 2010.
13. X. Hong, H. Chang, S. Shan, X. Chen, and W. Gao, “Sigma set: A small second order statistical region descriptor”, in Proc. CVPR’09, 2009, pp. 1802–1809.
14. J. Jeon, S. Cho, X. Tong, and S. Lee, “Intrinsic image decomposition using structure-texture separation and surface normals”, in Proc. ECCV’14, 2014.
15. L. Karacan, A. Erdem, and E. Erdem, “Image Matting with KL-Divergence Based Sparse Sampling”, in Proc. ICCV’15, 2015, pp. 424-432.
16. L. Karacan, E. Erdem, and A. Erdem, “Structure-preserving image smoothing via region covariances”, ACM Transactions on Graphics (TOG) – Proc. of ACM SIGGRAPH Asia 2013, vol. 32, issue 6, Nov. 2013.
17. M. Kass, and J. Solomon, “Smoothed local histogram filters”, ACM Transactions on
18. Graphics (TOG) – Proc. of ACM SIGGRAPH 2010, vol. 29, issue 4, July 2010.
19. Z. Ma, K. He, Y. Wei, J, Sun, and E. Wu, “Constant time weighted median filtering for stereo matching and beyond”, in Proc. ICCV’13, 2013.
20. Y. Meyer, Oscillating patterns in image processing and nonlinear evolution equations: the fifteenth Dean Jacqueline B. Lewis memorial lectures, vol. 22. American Mathematical Society, 2001.
21. S. Paris and F. Durand, “A Fast Approximation of the Bilateral Filter using a Signal Processing Approach”, in Proc. ECCV’06, 2006.)
22. H. Peng, B. Li, W. Xiong, W. Hu, and R. Ji, “RGBD salient object detection: A benchmark and algorithms”, in Proc. ECCV’14, 2014, pp. 92–109.
23. P. Perona and J. Malik, “Scale-space and edge detection using anisotropic diffusion”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12, Jul. 1990, pp. 629–639.
24. L. Rudin, S. Osher and E. Fatemi, “Nonlinear total variation based noise removal algorithms”, Physica D: Nonlinear Phenomena. vol. 60, Nov. 1992, pp. 259–268.
25. N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor Segmentation and Support Inference from RGBD Images”, in Proc. ECCV'12, 2012.
26. S. Song and J. Xiao, “Tracking revisited using RGBD camera: Unified benchmark and baselines”, in Proc. ICCV’13, 2013.
27. S. Song and J. Xiao, “Deep sliding shapes for amodal 3D object detection in RGB-D images”, In Proc. CVPR’16, 2016.
28. K. Subr, C. Soler, and F. Durand, “Edge-preserving multiscale image decomposition based on local extrema”, ACM Transactions on Graphics (TOG) – Proc. of ACM SIGGRAPH Asia 2009, vol. 28, issue 5, Dec. 2009.
29. C. Tomasi, and R. Manduchi, “Bilateral filtering for gray and color images”, in Proc. ICCV’98, 1998, pp. 839–846.
30. O. Tuzel, F. Porikli, and P. Meer, “Region covariance: A fast descriptor for detection and classification”, in Proc. ECCV’06, 2006, pp. 589– 600.
31. J. van de Weijer, and R. Van den Boomgaard, “Local mode filtering”, in Proc. CVPR’01, 2001, pp. II–428-433.
32. B. Weiss, “Fast median and bilateral filtering”, ACM Transactions on
33. Graphics (TOG) – Proc. of ACM SIGGRAPH 2006, vol. 25, issue 3, 2006, pp. 519–526.
34. L. Xu, C. Lu, Y. Xu, and J. Jia, “Image smoothing via L0 gradient minimization”, ACM Transactions on Graphics (TOG) – Proc. of ACM SIGGRAPH Asia 2011, vol. 30, issue 6, Dec. 2011.
35. L. Xu, J. SJ. Ren, Q. Yan, R. Liao, and J. Jia, “Deep edge-aware filters”, in Proc. ICML’15, 2015.
36. L. Xu, Q. Yan, Y. Xia, J. Jia, “Structure extraction from texture via relative total variation”, ACM Transactions on Graphics (TOG) – Proc. of ACM SIGGRAPH Asia 2012, vol. 31, issue 6, Nov. 2012.
37. Q. Yang, “Semantic filtering”, in Proc. CVPR'16, 2016, pp. 4517-4526.
38. Q. Yang, K.-H. Tan, and N. Ahuja, “Real-time O(1) bilateral filtering”, in Proc. CVPR’09, 2009, pp. 557–564.
39. Q. Yang, S. Wang, and N. Ahuja, “SVM for edge-preserving filtering”, in Proc. CVPR’10, 2010, pp. 1775– 1782.
40. Q. Zhang, L. Xu, and J. Jia, “100+ times faster weighted median filter”, in Proc. CVPR’14, 2014.
41. Q. Zhang, L. Xu, and J. Jia, “Rolling guidance filter”, in Proc. ECCV’14, 2014.