Deep Learning-Based Material Classification Using Thermal and Texture-Based Hybrid Images

Abstract

This study aimed to investigate the performance of deep learning-based material classification using thermal and texture-based hybrid images of different materials. For this purpose, a controlled experimental setup was established, and images of eight different materials, namely aluminium foil, paper, cardboard, fabric, metal, nylon, chipboard, and wood, were obtained. The materials were imaged at room temperature, after 10 seconds of heating, and after 20 seconds of heating. In addition, each condition was recorded under both illuminated and non-illuminated environments. The obtained images were combined with texture images to include thermal information at ratios of 25%, 50%, 75%, and 100%, and four different datasets were created by dividing them into 128×128-pixel sub-images. In the classification stage, VGG16, VGG19, DenseNet201, InceptionV3, MobileNetV2, and ResNet50V2 architectures were applied using the transfer learning approach, and the accuracy values of the models were compared. The findings showed that the thermal component ratio had a significant effect on classification performance. The highest accuracy value was obtained as 0.95 with the VGG19 architecture on the dataset consisting of hybrid images containing a 25% thermal component. The results indicate that the use of low-ratio thermal information together with texture information can improve material classification performance.

Keywords

• Machine learning
• Hybrid imaging
• Material classification
• Deep learning
• Transfer learning

Supporting Institution

Scientific and Technological Research Council of Türkiye

Project Number

1919B012420870

Ethical Statement

This study did not involve human participants, animal experiments, or personal/clinical data; therefore, ethical approval was not required. All experimental procedures were conducted in accordance with relevant scientific and ethical standards.

Thanks

The authors thank the Scientific and Technological Research Council of Türkiye (TÜBİTAK) for supporting this study under project number 1919B012420870 within the scope of 2209-A.

Termal ve Doku Bileşenli Hibrit Görüntüler Kullanılarak Derin Öğrenme Tabanlı Malzeme Sınıflandırması

Abstract

Bu çalışmada, farklı malzemelere ait termal ve doku bileşenli hibrit görüntüler kullanılarak derin öğrenme tabanlı malzeme sınıflandırma performansının araştırılması amaçlanmıştır. Bu amaç doğrultusunda kontrollü bir deney düzeneği oluşturulmuş ve alüminyum folyo, kâğıt, karton, kumaş, metal, naylon, sunta ve tahta olmak üzere sekiz farklı materyale ait görüntüler elde edilmiştir. Materyaller oda sıcaklığında, 10 saniye ısıtma sonrasında ve 20 saniye ısıtma sonrasında görüntülenmiştir. Ayrıca her durum aydınlatmalı ve aydınlatmasız koşullarda kaydedilmiştir. Elde edilen görüntüler; %25, %50, %75 ve %100 oranlarında termal bilgi içerecek şekilde doku görüntüleriyle birleştirilmiş ve 128×128 piksel boyutlarında alt görüntülere ayrılarak dört farklı veri seti oluşturulmuştur. Sınıflandırma aşamasında VGG16, VGG19, DenseNet201, InceptionV3, MobileNetV2 ve ResNet50V2 mimarileri transfer öğrenme yaklaşımıyla uygulanmış ve modellerin doğruluk değerleri karşılaştırılmıştır. Elde edilen bulgular, termal bileşen oranının sınıflandırma başarımı üzerinde belirgin bir etkiye sahip olduğunu göstermiştir. En yüksek doğruluk değeri, %25 termal bileşen içeren hibrit görüntülerden oluşan veri setinde VGG19 mimarisi ile 0,95 olarak elde edilmiştir. Sonuçlar, düşük oranlı termal bilginin doku bilgisiyle birlikte kullanılmasının malzeme sınıflandırma performansını artırabileceğini göstermektedir.

Keywords

• Makine öğrenmesi
• Hibrit görüntüleme
• Malzeme sınıflandırması
• Derin öğrenme
• Transfer öğrenme

Supporting Institution

Türkiye Bilimsel ve Teknolojik Araştırma Kurumu

Project Number

1919B012420870

Ethical Statement

Bu çalışma insan katılımcıları, hayvan deneyleri veya kişisel/klinik veriler içermediğinden etik onay gerekmemiştir. Tüm deneysel işlemler ilgili bilimsel ve etik standartlara uygun olarak yürütülmüştür.

Thanks

Yazarlar bu çalışmayı 2209-A kapsamında 1919B012420870 numaralı proje ile destekleyen Türkiye Bilimsel ve Teknolojik Araştırma Kurumuna (TÜBİTAK) teşekkür ederler.

References

1. R. M. Haralick, K. Shanmugam, and I. Dinstein, “Textural features for image classification,” IEEE Trans. Syst. Man Cybern., vol. SMC-3, no. 6, pp. 610–621, Nov. 1973.
2. R. Mapari, S. Narkhede, A. Navale, and J. Babrah, “Automatic waste segregator and monitoring system,” Int. J. Adv. Comput. Res., vol. 10, no. 49, pp. 172–181, 2020.
3. H. Mahami et al., “Material recognition for automated progress monitoring using deep learning methods,” arXiv preprint, arXiv:2006.16344, 2020.
4. A. Ghosh, M. Ehrlich, L. Davis, and R. Chellappa, “Unsupervised super-resolution of satellite imagery for high fidelity material label transfer,” in IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium, 2019.
5. Y. Cho, N. Bianchi-Berthouze, N. Marquardt, and S. J. Julier, “Deep thermal imaging,” in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018.
6. J. Govindarajan, “A case for joint development of IR cameras in India,” Int. J. Intell. Electron. Syst., vol. 1, no. 1, pp. 26–30, Nov. 2007.
7. C. Hildebrandt, C. Raschner, and K. Ammer, “An overview of recent application of medical infrared thermography in sports medicine in Austria,” Sensors (Basel), vol. 10, no. 5, pp. 4700–4715, May. 2010.
8. R. Usamentiaga, D. F. Garcia, J. Molleda, F. G. Bulnes, and V. G. Orgeira, “Temperature tracking system for sinter material in a rotatory cooler based on infrared thermography,” IEEE Trans. Ind. Appl., vol. 50, no. 5, pp. 3095–3102, Feb. 2014.

9. R. Vadivambal and D. S. Jayas, “Applications of thermal imaging in agriculture and food industry—A review,” Food Bioproc. Tech., vol. 4, no. 2, pp. 186–199, Feb. 2011.
10. M. B. Lopez, C. R. del-Blanco, and N. Garcia, “Detecting exercise-induced fatigue using thermal imaging and deep learning,” in 2017 Seventh International Conference on Image Processing Theory, Tools and Applications (IPTA), 2017.
11. G. D. Boreman, Modulation Transfer Function in Optical and Electro-optical Systems. Bellingham, WA: SPIE Press, 2001.
12. M. Kontos, R. Wilson, and I. Fentiman, “Digital infrared thermal imaging (DITI) of breast lesions: sensitivity and specificity of detection of primary breast cancers,” Clin. Radiol., vol. 66, no. 6, pp. 536–539, Jun. 2011.
13. M. Pergeorelis, T. Rust, and C. Kambhamettu, “Open dataset and enhancement method for long-wave thermal diurnal material classification,” in 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2025, pp. 4595–4601.
14. T.Y. Chen et al., “Artificial intelligence system combining with infrared thermography and visible image for abnormal temperature detection and floor material identification,” IEEE Sens. J., vol. 24, no. 24, pp. 42181–42194, Aug. 2024.
15. J.-R. Riba, T. Canals, and R. Cantero, “Recovered paperboard samples identification by means of mid-infrared sensors,” IEEE Sens. J., vol. 13, no. 7, pp. 2763–2770, Apr. 2013.
16. E. Kerr, T. M. McGinnity, and S. Coleman, “Material classification based on thermal properties — A robot and human evaluation,” in 2013 IEEE International Conference on Robotics and Biomimetics (ROBIO), 2013.
17. L. Wu, S. Liu, and Y. Wu, “Infrared imaging detection of water permeation on field large-scale rock relics,” IEEE Trans. Geosci. Remote Sens., vol. 49, no. 1, pp. 581–590, Jul. 2011.
18. H. Li, X.-J. Wu, and J. Kittler, “Infrared and visible image fusion using a deep learning framework,” in 2018 24th International Conference on Pattern Recognition (ICPR), 2018.
19. H. Xu, J. Ma, J. Jiang, X. Guo, and H. Ling, “U2Fusion: A unified unsupervised image fusion network,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 1, pp. 502–518, Jul. 2022.
20. W. Xiao, Y. Zhang, H. Wang, F. Li, and H. Jin, “Heterogeneous knowledge distillation for simultaneous infrared-visible image fusion and super-resolution,” IEEE Trans. Instrum. Meas., vol. 71, pp. 1–15, Feb. 2022.
21. W. Großmann, H. Horn, and O. Niggemann, “Improving remote material classification ability with thermal imagery,” Sci. Rep., vol. 12, no. 1, p. 17288, Oct. 2022.
22. Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proc. IEEE Inst. Electr. Electron. Eng., vol. 86, no. 11, pp. 2278–2324, Nov. 1998.
23. F. Zhuang et al., “A comprehensive survey on transfer learning,” Proc. IEEE Inst. Electr. Electron. Eng., vol. 109, no. 1, pp. 43–76, Jul. 2021.
24. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv [cs.CV], 2014.
25. G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
26. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
27. M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “MobileNetV2: Inverted residuals and linear bottlenecks,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
28. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778.