Defect detection in apple (Granny Smith) with deep learning techniques

Yıl 2023, , 1119 - 1129, 15.10.2023

Zeynep Ünal , Tefide Kızıldeniz , Mustafa Özden , Hakan Aktaş , Ömer Karagöz

https://doi.org/10.28948/ngumuh.1250012

Öz

During apple (Malus communis L.) harvesting, physical damage that reduces the quality of the product is inevitable. Early detection and separation of damaged fruits is important in terms of increasing their commercial value. Undetected defective products reduce the production volume as well as food loss, since they affect the quality of intact products. The aim of this study is to detect defects in apples using deep learning techniques on images taken from the “Granny Smith” apple cultivar. A technique that does not require special conditions and that will make classification and defect detection cost-effectively has been researched. In the study, the test accuracy of the InceptionV3 model was 100% after 100 epochs, and the test accuracy of the AlexNet model was 98.33%. A method has been developed that can prevent economic losses that may occur after harvesting by detecting and separating the damages that occur on the fruit during harvesting with deep learning techniques.

Anahtar Kelimeler

AlexNet, Apple classification, Bruise detection, Deep learning, Granny Smith

Proje Numarası

TGT 2021/22-BAGEP

Derin öğrenme teknikleri ile elmada (Granny Smith) kusur tespiti

Öz

Elma (Malus communis L.) derimi sırasında ürünün kalitesini düşüren fiziksel zararlanmaların oluşması kaçınılmazdır. Zarar gören meyvelerin erken tespit edilerek ayrılması ticari değerinin artırılması açısından önemlidir. Tespit edilemeyen kusurlu ürünler sağlam ürünlerin kalitesini etkilediğinden dolayı gıda kaybının yanı sıra üretim hacmini de düşürmektedir. Çalışmanın amacı, “Granny Smith” elma çeşidinden alınan görüntüler üzerinde, derin öğrenme teknikleri kullanarak elmalarda kusur tespit etmektir. Özel koşul gerektirmeyen, uygun maliyetle sınıflandırma ve kusur tespiti yapacak bir teknik araştırılırmıştır. Çalışmada, InceptionV3 modelinin 100 çevrim sonunda test doğruluğu %100, AlexNet modelinin ise test doğruluğu %98.33 elde edilmiştir. Derin öğrenme teknikleriyle, derim sırasında meyve üzerinde oluşan zararlar tespit edilerek ayrılmasıyla, derim sonrası oluşabilecek ekonomik kayıpların önüne geçebilecek bir yöntem geliştirilmiştir.

Anahtar Kelimeler

AlexNet, Derin öğrenme, Elma sınıflandırması, Granny Smith, InceptionV3, Kusur Tespiti

Destekleyen Kurum

Niğde Ömer Halisdemir Üniversitesi Bilimsel Araştırma Projeleri Koordinasyon Birimi Koordinatörlüğü

Proje Numarası

TGT 2021/22-BAGEP

Teşekkür

Bu araştırma Niğde Ömer Halisdemir Üniversitesi Bilimsel Araştırma Projeleri Koordinasyon Birimi Koordinatörlüğü’nce desteklenmiştir. Proje Adı: Derin öğrenme teknikleri kullanarak elma sınıflandırma ve sınıflandırılmış elmaların içinde fiziksel zararlanmaların tespiti. Proje No: TGT 2021/22-BAGEP, 2021. Bu nedenle yazarlar, destek sağlayan Niğde Ömer Halisdemir Üniversitesi Bilimsel Araştırma Proje Ofisine teşekkür ederler.

Kaynakça

• FAOStat, http://www.fao.org/faostat/en/#data/QC, Accessed 13 June 2021.
• M. Ünlü, Derim Sonrası Kayıplar, https://arastirma.tarimorman.gov.tr/alata/Belgeler/Diger-belgeler/, Accessed 1 January 2015.
• A. E. Özdemir, E. Çandır, Ö. Dündar ve R. Dilbaz, Üreticiden tüketiciye ulaşıncaya kadar geçen süreçte elmalardaki̇ kayıplar ve önleme yolları. International Journal of Agricultural and Natural Sciences, 2(1), pp. 165-168, 2009.
• H. Özgönen ve H. Ç. Kılıç, Isparta ilinde elmalarda sorun olan hasat sonrası hastalıkların ve yaygınlık oranlarının belirlenmesi. International Journal of Agricultural and Natural Sciences, 2(2), pp. 53-60, 2009.
• Z. Hu, Bruise detection in apples using 3D infrared imaging and machine learning technologies. Michigan Technological University, A Dissertation, Michigan, 2017. http://dx.doi.org/ 10.37099/mtu.dc.etdr/509
• F. Vega ve M. C. Torres, Automatic detection of bruises in fruit using Biospeckle techniques. In Symposium of Signals, Images and Artificial Vision-2013: STSIVA-2013, Bogotá, Colombia, 2013. http://dx.doi.org/10.1109/STSIVA.2013. 6644916
• Y. C. Chiy ve C. H. Chen, Development of on-line apple bruise detection system. Engineering in agriculture, environment, and food, 10(3), pp. 223-232, 2017. http://dx.doi.org/10.1016/j.eaef.2017. 03.003
• Y. Lu, R. Li ve R. Lu, Structured-illumination reflectance imaging (SIRI) for enhanced detection of fresh bruises in apples. Postharvest Biology and Technology, 117, pp. 89-93, 2016. http://dx.doi.org/10.1016/j.postharvbio.2016.02.005
• J. Li, W. Huang, X. Tian, C. Wang, S. Fan ve C. Zhao, Fast detection and visualization of early decay in citrus using Vis-NIR hyperspectral imaging. Computers and Electronics in Agriculture, 127, pp. 582-592, 2016. http://dx.doi.org/ 10.1016/j.compag.2016.07.016
• J. C. Keresztes, M. Goodarzi, W. Saeys, Real-time pixel based early apple bruise detection using short wave infrared hyperspectral imaging in combination with calibration and glare correction techniques. Food Control, 66, pp. 215-226, 2016. http://dx.doi.org/10.1016/j.foodcont.2016.02.007
• W. Che, L. Sun, Q. Zhang, W. Tan, D. Ye, D. Zhang ve Y. Liu, Pixel based bruise region extraction of apple using Vis-NIR hyperspectral imaging. Computers and Electronics in Agriculture, 146., pp. 12-21, 2018. http://dx.doi.org/10.1016/j.compag. 2018. 01.013
• E. Diels, M. van Dael, J. Keresztes, S. Vanmaercke, P. Verboven, B. Nicolai, W. Saeysa, H. Ramona ve B. Smeets, Assessment of bruise volumes in apples using X-ray computed tomography. Postharvest Biology and Technology, 128, pp. 24-32, 2017. http://dx.doi.org/10.1016/j.postharvbio.2017.01.013
• O. Doosti-Irani, M. R. Golzarian, M. H. Aghkhani, H. Sadrnia ve M. Doosti-Irani, Development of multiple regression model to estimate the apple’s bruise depth using thermal maps. Postharvest Biology and Technology, 116, pp. 75-79, 2016. http://dx.doi.org/10.1016/j.postharvbio.2015.12.024
• D. Jawale ve M. Deshmukh, Real time automatic bruise detection in (Apple) fruits using thermal camera. In 2017 International Conference on Communication and Signal Processing (ICCSP)., Tamilnadu, India, 2017. http://dx.doi.org/ 10.1109/iccsp.2017.8286542
• Z. Du, X. Zeng, X. Li, X. Ding, J. Cao ve W. Jiang, Recent advances in imaging techniques for bruise detection in fruits and vegetables. Trends in Food Science & Technology, 99, pp. 133-141, 2020. http://dx.doi.org/10.1016/j.tifs.2020.02.024
• J. Varith, G. M. Hyde, A. L. Baritelle, J. K. Fellman ve T. Sattabongkot, Non-contact bruise detection in apples by thermal imaging. Innovative Food Science & Emerging Technologies, 4(2), pp. 211-218, 2003. http://dx.doi.org/10.1016/s1466-8564(03)00021-3
• R. Lu, Detection of bruises on apples using near-infrared hyperspectral imaging. Transactions of the American Society of Agricultural Engineers, 46 (2), pp. 523-530, pp. 523-530, 2003. http://dx.doi.org/10.13031/2013.12941
• G. ElMasry, N. Wang, C. Vigneault, J. Qiao and A. ElSayed, Early detection of apple bruises on different background colors using hyperspectral imaging. LWT-Food Science and Technology, 41(2), pp. 337-345, 2008. http://dx.doi.org/ 10.1016/j.lwt.2007.02.022
• R. Lu, H. Cen, M. Huang ve D. P. Ariana, Spectral absorption and scattering properties of normal and bruised apple tissue. Transactions of the ASABE, 53(1), pp. 263-269, 2010. http://dx.doi.org/ 10.13031/2013.29491
• W. Huang, J. Li, Q. Wang ve L. Chen, Development of a multispectral imaging system for online detection of bruises on apples. Journal of Food Engineering, 146, pp. 62-71, 2015. http://dx.doi.org/ 10.1016/j.jfoodeng.2014.09.002
• W. Tan, L. Sun, F. Yang, W. Che, D. Ye, D. Zhang ve B. Zou, The feasibility of early detection and grading of apple bruises using hyperspectral imaging. Journal of Chemometrics, 32(10), p. e3067, 2018. http://dx.doi.org/10.1002/cem.3067
• M. Zhang ve G. Li, Visual detection of apple bruises using AdaBoost algorithm and hyperspectral imaging. International Journal of Food Properties, 21(1), pp. 1598-1607, 2018. http://dx.doi.org/10.1080/10942912.2018.1503299
• S. Zhang, X. Wu, S. Zhang, Q. Cheng ve Z. Tan, An effective method to inspect and classify the bruising degree of apples based on the optical properties. Postharvest Biology and Technology, 127, pp. 44-52, 2017. http://dx.doi.org/ 10.1016/j.postharvbio.2016.12.008
• S. Fan, J. Li, Y. Zhang, X. Tian, Q. Wang, X. He, C. Zhang ve W. Huang, On line detection of defective apples using computer vision system combined with deep learning methods. Journal of Food Engineering, 286, p. 110102, 2020. http://dx.doi.org/10.1016/j.jfoodeng.2020.110102
• J. Xing, V. Van Linden, M. Vanzeebroeck ve J. De Baerdemaeker, Bruise detection on Jonagold apples by visible and near-infrared spectroscopy. Food control, 16(4), pp. 357-361, 2005. http://dx.doi.org/10.1016/j.foodcont.2004.03.016
• K. Kayaalp ve S. Metlek, Classification of robust and rotten apples by deep learning algorithm. Sakarya University Journal of Computer and Information Sciences, 3(2), pp. 112-120, 2020. http://dx.doi.org/10.35377/saucis.03.02.717452
• V. Leemans, H. Magein ve F. Destain, On-line fruit grading according to their external quality using machine vision. Biosystem Engineering vol. 83, no. 4, p. 397–404, 2002.
• D. Unay ve B. Gosselin, Automatic defect segmentation of Jonagold apples on multi-spectral images: A comparative study. Postharvest Biology and Technology, 42(3), pp. 271-279, 2006. http://dx.doi.org/10.1016/j.postharvbio.2006.06.010
• D. Unay, B. Gosselin, O. Kleynen, V. Leemans, M. F. Destain ve O. Debeir, Automatic grading of Bi-colored apples by multispectral machine vision. Computers and electronics in agriculture, 75(1), pp. 204-212, 2011. http://dx.doi.org/10.1016/j.compag. 2010.11.006
• X. Luo, T. Takahashi, K. Kyo ve S. Zhang, Wavelength selection in vis/NIR spectra for detection of bruises on apples by ROC analysis. Journal of Food Engineering, 109(3), pp. 457-466, 2012. http://dx.doi.org/10.1016/j.jfoodeng.2011. 10.035
• J. C. Keresztes, E. Diels, M. Goodarzi, N. Nguyen-Do-Trong, P. Goos, B. Nicolai ve W. Saeys, Glare based apple sorting and iterative algorithm for bruise region detection using shortwave infrared hyperspectral imaging. Postharvest biology and technology, 130, pp. 103-115, 2017. http://dx.doi.org/10.1016/j.postharvbio.2017.04.005
• C. Ferrari, G. Foca, R. Calvini ve A. Ulrici, Fast exploration and classification of large hyperspectral image datasets for early bruise detection on apples. Chemometrics and Intelligent Laboratory Systems, 146, pp. 108-119, 2015. http://dx.doi.org/ 10.1016/j.chemolab.2015.05.016
• Q. Zhu, J. Guan, M. Huang, R. Lu ve F. Mendoza, Predicting bruise susceptibility of Golden Delicious apples using hyperspectral scattering technique. Postharvest Biology and Technology, 114, pp. 86-94, 2016. http://dx.doi.org/10.1016/j.postharvbio. 2015. 12.007
• Y. C. Chiu, X. L. Chou, T. E. Grift ve M. Chen, Automated detection of mechanically induced bruise areas in golden delicious apples using fluorescence imagery. Transactions of the ASABE, 58(2), pp. 215-225, 2015. http://dx.doi.org/ 10.13031/trans.58.10578
• R. Li, Y. Lu ve R. Lu, Structured illumination reflectance imaging for enhanced detection of subsurface tissue bruising in apples. Transactions of the ASABE, 61(3), pp. 809-819, 2018. http://dx.doi.org/10.13031/aim.20162460153
• Y. Lu, R. Li ve R. Lu, (2016a). Fast demodulation of pattern images by spiral phase transform in structured-illumination reflectance imaging for detection of bruises in apples. Computers and Electronics in Agriculture, 127, p. 652–658, 2016. http://dx.doi.org/10.1016/j.compag.2016.07.012
• Y. Lu ve R. Lu, Using composite sinusoidal patterns in structured-illumination reflectance imaging (SIRI) for enhanced detection of apple bruise. Journal of food engineering, 199, pp. 54-64, 2017. http://dx.doi.org/10.1016/j.jfoodeng.2016. 12.008
• P. Baranowski, W. Mazurek, B. Witkowska-Walczak ve C. Sławiński, Detection of early apple bruises using pulsed-phase thermography. Postharvest biology and technology, 53(3), pp. 91-100, 2009. http://dx.doi.org/10.1016/j.postharvbio. 2009.04.006
• Y. Lu ve R. Lu, Histogram-based automatic thresholding for bruise detection of apples by structured-illumination reflectance imaging. Biosystems Engineering, 160, pp. 30-41, 2017. http://dx.doi.org/10.1016/j.biosystemseng.2017.05.005
• A. Siedliska, P. Baranowski ve W. Mazurek, Classification models of bruise and cultivar detection on the basis of hyperspectral imaging data. Computers and Electronics in Agriculture, 106, pp. 66-74, 2014. http://dx.doi.org/ 10.1016/j.compag.2014.05.012
• J. F. I. Nturambirwe, H. H. Nieuwoudt, W. J. Perold ve U. L. Opara, Detecting bruise damage and level of severity in apples using a contactless NIR spectrometer. Applied Engineering in Agriculture, 36(3), pp. 257-270, 2020. http://dx.doi.org/ 10.13031/aea.13218
• J. F. Nturambirwe, E. A. Hussein, M. Vaccari, C. Thron, W. J. Perold ve U. L. Opara, Feature reduction for the classification of bruise damage to apple fruit using a contactless FT-NIR spectroscopy with machine learning. Foods, 12(1), p. 210, 2023. http://dx.doi.org/10.3390/foods12010210
• S. Fan, X. Liang, W. Huang, V. J. Zhang, Q. Pang, X. He, L. Li ve C. Zhang, Real-time defects detection for apple sorting using NIR cameras with pruning-based YOLOV4 network. Computers and Electronics in Agriculture, 193, p. 106715, 2022. http://dx.doi.org/10.1016/j.compag. 2022.106715
• A. Beyaz, R. Ozturk ve U. Turker, Assessment of mechanical damage on apples with image analysis. Food, Agriculture & Environment (JFAE), 8(3&4), pp. 476-480, 2010.
• S. Albawi, T. A. Mohammed ve S. Al-Zawi, Understanding of a convolutional neural network. In 2017 international conference on engineering and technology (ICET), Antalya, Turkey, 2017. http://dx.doi.org/10.1109/ICEngTechnol.2017.8308186
• A. Wu, J. Zhu ve T. Ren, Detection of apple defect using laser-induced light backscattering imaging and convolutional neural network. Computers & Electrical Engineering, 81, p. 106454, 2020. http://dx.doi.org/10.1016/j.compeleceng.2019.106454
• A. Kumar, R. C. Joshi, M. K. Dutta, M. Jonak ve R. Burget, Fruit-CNN: An Efficient Deep learning-based fruit classification and quality assessment for precision agriculture. In 2021 13th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT), 2021. http://dx.doi.org/10.1109/ICUMT54235. 2021.9631643
• N. Stasenko, M. Savinov, V. Burlutskiy, M. Pukalchik and A. Somov, Deep Learning for postharvest decay prediction in apples. In IECON 2021–47th Annual Conference of the IEEE Industrial Electronics Society, 2021. http://dx.doi.org/10.1109/iecon48115.2021.9589498
• Y. Xin, S. Ma, Y. Wei, J. Hu, Z. Ding ve F. Wang. Detection of apple surface defect based on YOLOv3. In 2021 ASABE Annual International Virtual Meeting, 2021. http://dx.doi.org/ 10.13031/aim.202100611
• Y. Xue, L. Wang, Y. Zhang and Q. Shen, Defect detection method of apples based on GoogLeNet Deep Transfer Learning. Nongye Jixie Xuebao/Transactions of the Chinese Society of Agricultural Machinery, 51(7), p. 30–35, 2020.
• H. Ayaz, E. Rodríguez-Esparza and M. Ahmad, Classification of apple disease based on non-linear deep features. Applied Sciences, 11(14), p. 6422, 2021. http://dx.doi.org/10.3390/app11146422
• N. D. Lewis, Deep Learning made easy with R, USA: Auscov, 2016.
• I. Goodfellow, Y. Bengio ve A. Courville, Deep Learning, MIT Press, 2016.
• S. S. Mousavi, M. Schukat ve E. Howley, (2016, September). Deep reinforcement learning: an overview. In Proceedings of SAI Intelligent Systems Conference, London, UK, 2016.
• Y. Bengio, Learning deep architectures for AI. Foundations and trends® in Machine Learning, 2(1), 1-127. Boston: Now publishers inc., 2009.
• A. Agrawal, Loss Functions and Optimization Algorithms, https://medium.com/data-science-group-iitr/loss-functions-and-optimization-algorithms-demystified-bb92daff331c, Accessed 9 September 2017
• Y. B. Özçelik ve A. Altan, Diyabetik retinopati teşhisi için Fundus görüntülerinin derin öğrenme tabanlı sınıflandırılması. Avrupa Bilim ve Teknoloji Dergisi, (29), pp. 156-167, 2021. http://dx.doi.org/10.31590/ejosat.1011806
• Z. Ünal, T. Kızıl Deniz Gürbüz, M. Özden ve H. Aktaş, Classification of red apple varieties with deep learning models. IV. International Turkic World Congress on Science and Engineering, Niğde, 2022.
• O. Cömert, M. Hekim ve K. Adem, Faster R-CNN kullanarak elmalarda çürük tespiti. Uluslararası Mühendislik Araştırma ve Geliştirme Dergisi, 11(1), pp. 335-341, 2019.
• Y. LeCun , L. Bottou, Y. Bengio ve P. Haffner, Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), pp. 2278-2324, 1998. http://dx.doi.org/10.1109/ 5.726791
• A. Kausar, M. Sharif, J. Park ve D. R. Shin, Pure-cnn: A framework for fruit images classification. In 2018 International Conference on Computational Science and Computational Intelligence (CSCI), 2018.
• S. Lu, Z. Lu, S. Aok ve L. Graham, Lu, S., Lu, Z., Aok, S., Graham, L. 2018, November. Fruit classification based on six layer convolutional neural network. In 2018 IEEE 23rd International Conference on Digital Signal Processing (DSP), 2018. http://dx.doi.org/10.1109/icdsp.2018. 8631562
• J. Naranjo-Torres, M. Mora, R. Hernández-García, R. J. Barrientos, C. Fredes ve A. Valenzuela, A review of convolutional neural network applied to fruit image processing. Applied Sciences, 10(10), p. 3443, 2020. http://dx.doi.org/10.3390/ app10103443
• A. Krizhevsky, I. Sutskever and G. E. Hinton, Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, pp. 1097-1105., 2012. http://dx.doi.org/10.1145/3065386
• K. Simonyan and A. Zisserman, Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR 2015), San Diego, 2014. https://doi.org/10.48550/arXiv.1409.1556
• K. He, X. Zhang, S. Ren and J. Sun, Identity mappings in deep residual networks. In European conference on computer vision, Amsterdam, The Netherlands, 2016. http://dx.doi.org/10.1007/978-3-319-46493-0_38
• C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens and Z. Wojna, Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, USA, 2016. http://dx.doi.org/10.1109/cvpr.2016.308
• F. Chollet, Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, Honolulu, USA, 2017. http://dx.doi.org/10.1109/cvpr.2017.195
• M. Tan ve Q. Le, Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, 2019.
• J. Davis ve M. Goadrich, The relationship between Precision-Recall and ROC curves. In Proceedings of the 23rd international conference on Machine learning., pp. 233-240, 2006. http://dx.doi.org/ 10.1145/1143844.1143874
• N. Canbaz, Nesneye dayalı yazılımların tasarım kalitesini ölçmek için öğrenme tabanlı bir yöntem. İstanbul Teknik Üniversitesi, Bilişim Enstitüsü, Bilgisayar ve Bilişim Fakültesi, Yüksek Lisans Tezi, İstanbul (Danışman: Doç. Feza Buzluca), 2015.
• R. Halepmollası, Alt sekans profil harıtaları kullanılarak protein katlanması tanıma. İstanbul Teknik Üniversitesi, Fen Bilimleri Enstitüsü, Yüksek Lisans Tezi, İstanbul (Danışman: Yrd. Doç. Dr. Ömer Sinan Saraç)., İstanbul,2016