Transfer Learning based Image Classification of Diseased Tomato Leaves with Optimal Fine-Tuning combined with Heat Map Visualization

Year 2023, Volume: 29 Issue: 4, 1003 - 1017, 06.11.2023

Sandhya Devi Ramıah Subburaj Vijayakumar Vaıthyam Rengarajan Sivakumar Palanıswamy

https://doi.org/10.15832/ankutbd.1230265

Abstract

Plant disease detection and disease classification at initial stages for sensitive commodities like tomatoes and potatoes is highly mandated as the harvest losses have a direct impact on the price fixation of the vegetables. The most identified limitation in the study of plant pathology is the availability of datasets with visual symptoms that covers all the possible diseases of one crop or plant species. Computer Vision systems and advancements in deep learning-based modeling methodologies gained significant attention in smart farming. It is presumed that the implementation of deep learning algorithms demands a large amount of data to learn complex features automatically and this can pose a challenge for applications with lesser data to achieve generalization. In such cases, Transfer Learning with optimum regularization techniques and fine-tuning mechanisms is the solution to overcome the limitations of smaller datasets. The objective of the work is to develop Tomato Disease Classification System using a transfer learning approach for ten tomato disease classes of the PlantVillage dataset downloaded from the Kaggle platform. Inception V3, a pre-trained transfer learning model is used to classify this multi-class, imbalanced, tomato plant disease based on the leaf symptoms such as dark brown lesions, concentric rings, etc. Geometrical data augmentation is used as a regularization technique to expand the size of the dataset. Significant improvement in the performance metrics is observed when the finetuning is optimum. The training accuracy and validation accuracy of the model before and after fine-tuning are 97.08%, 83.52%, and 98.19%, 95.93% respectively. The average accuracy with augmentation and optimal fine-tuning is 98%. In addition, prediction scores in terms of precision, recall, and F1-score are obtained to visualize the rate of mispredictions across the disease classes. It is observed that the misprediction rate is high across the classes early blight, late blight, and Septoria spot due to similar visual symptoms. Further, activations are used to generate an attention map in the form of Heat Maps which are included as a post-processing step before the classification of the output. Plant Leaf Disease Classification- A web application is deployed using Streamlit Python library and Ngrok services.

Keywords

Image classification, Transfer learning, InceptionV3, Attention maps

References

• Adhikari, P., Oh, Y. and Panthee, D.R. (2017). Current status of early blight resistance in tomato: an update. International journal of molecular sciences, 18(10), pp.1-22 https://doi.org/10.3390/ijms18102019 .
• Agarwal, Mohit, Abhishek Singh, Siddhartha Arjaria, Amit Sinha, and Suneet Gupta. "ToLeD: Tomato leaf disease detection using convolution neural network." Procedia Computer Science 167 (2020): 293-301. https://doi.org/10.1016/j.procs.2020.03.225
• Al‐gaashani, Mehdhar SAM, Fengjun Shang, Mohammed SA Muthanna, Mashael Khayyat, and Ahmed A. Abd El‐Latif. "Tomato leaf disease classification by exploiting transfer learning and feature concatenation." IET Image Processing 16, no. 3 (2022): 913-925. https://doi.org/10.1049/ipr2.12397
• Altman, Douglas G., and J. Martin Bland. "Diagnostic tests. 1: Sensitivity and specificity." BMJ: British Medical Journal 308, no. 6943 (1994): 1552. doi: 10.1136/bmj.308.6943.1552 .
• Astani, M., Hasheminejad, M, and Vaghefi, M. (2022). A diverse ensemble classifier for tomato disease recognition. Computers and Electronics in Agriculture, 198, 107054. https://doi.org/10.1016/j.compag.2022.107054
• Basavaiah, Jagadeesh, and Audre Arlene Anthony. "Tomato leaf disease classification using multiple feature extraction techniques." Wireless Personal Communications 115, no. 1 (2020): 633-651
• Bengio, Y., Bastien, F., Bergeron, A., Boulanger–Lewandowski, N., Breuel, T., Chherawala, Y., and Sicard, G. (2011). Deep learners benefit more from out-of-distribution examples. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings, pp. 164-172.
• Dong, N., Zhao, L., Wu, C. H., & Chang, J. F. (2020). Inception v3 based cervical cell classification combined with artificially extracted features. Applied Soft Computing, 93, 106311. https://doi.org/10.1016/j.asoc.2020.106311.
• Feuz, K.D. & Cook, D.J. (2015). Transfer learning across feature-rich heterogeneous feature spaces via feature-space remapping (FSR). ACM Transactions on Intelligent Systems and Technology (TIST), 6(1), pp.1-27. https://doi.org/10.1145/2629528 .
• Gnanasekaran, A& Vijayalakshmi.S. (2012). An economic analysis of tomato cultivation in Dindigul district of Tamil Nadu. International Journal of Scienceand research, vol. 3, pp. 995-997.
• Goh, A. M., & Yann, X. L. (2021). Food-image Classification Using Neural Network Model. Int. J. of Electronics Engineering and Applications, 9(3), pp.12-22.
• Guo, Y., Liu, Y., Georgiou, T., & Lew, M. S. (2018). A review of semantic segmentation using deep neural networks. International journal of multimedia information retrieval, 7,pp. 87-93. https://doi.org/10.1007/s13735-017-0141-z
• Harakannanavar, S. S., Rudagi, J. M., Puranikmath, V. I., Siddiqua, A., and Pramodhini, R. (2022). Plant leaf disease detection using computer vision and machine learning algorithms. Global Transitions Proceedings, 3(1), pp. 305-310. https://doi.org/10.1016/j.gltp.2022.03.016 .
• Hossain, S., Mou, R. M., Hasan, M. M., Chakraborty, S., and Razzak, M. A. (2018). Recognition and detection of tea leaf's diseases using support vector machine. In 2018 IEEE 14th International Colloquium on Signal Processing & Its Applications (CSPA), pp. 150-154. doi: 10.1109/CSPA.2018.8368703
• H. Zhang, L. Zhang and Y. Jiang. (2019). Overfitting and Underfitting Analysis for Deep Learning Based End-to-end Communication Systems. 11th International Conference on Wireless Communications and Signal Processing,doi: 10.1109/WCSP.2019.8927876, pp. 1-6. doi: 10.1109/WCSP.2019.8927876
• Hussain, M., Bird, J.J. and Faria, D.R. (2018). A study on cnn transfer learning for image classification. In UK Workshop on computational Intelligence UK, Springer, Cham, pp. 191-202. https://doi.org/10.1007/978-3-319-97982-3_16
• Ibrahim. H (2019). Susceptibility Studies on Two Varieties of Tomato (Lycopersicon esculentum) to Fungal Leaf Spots. EAS Journal of Nutrition and Food Sciences, vol. 1, no.1, pp. 11-15. doi: 10.36349/easjnfs.2019.v01i01.003
• Jiang, P., Chen, Y., Liu, B., He, D., and Liang, C. (2019). Real-time detection of apple leaf diseases using deep learning approach based on improved convolutional neural networks. IEEE Access, 7, pp. 59069-59080. doi: 10.1109/ACCESS.2019.2914929
• Jiang, Ding, Fudong Li, Yuequan Yang, and Song Yu. "A tomato leaf diseases classification method based on deep learning." In 2020 chinese control and decision conference (CCDC), pp. 1446- 1450. IEEE, 2020. DOI: 10.1109/CCDC49329.2020.9164457
• Jignesh Chowdary, G., Punn, N. S., Sonbhadra, S. K., and Agarwal, S. (2020). Face mask detection using transfer learning of inceptionv3. In Big Data Analytics: 8th International Conference, BDA 2020, Sonepat, India, December 15–18, 2020, Proceedings 8, pp. 81-90. Springer International Publishing. https://doi.org/10.1007/978-3-030-66665-1_6
• Kubota, K., Tsuda, S., Tamai, A. and Meshi, T. (2003). Tomato mosaic virus replication protein suppresses virus-targeted posttranscriptional gene silencing. Journal of virology, 77(20), pp.11016-11026. https://doi.org/10.1128/JVI.77.20.11016-11026.2003
• Liu, J., Chafi, R., Legarrea, S., Alba, J. M., Meijer, T., Menken, S. B., and Kant, M. R. (2020). Spider mites cause more damage to tomato in the dark when induced defenses are lower. Journal of chemical ecology, 46, pp.631-641. https://doi.org/10.1007/s10886-020-01195-1
• Liu, J., and Wang, X. (2021). Plant diseases and pest detection based on deep learning: a review. Plant Methods, 17, pp. 1-18. https://doi.org/10.1186/s13007-021-00722-9
• Liang, Jingsai. "Confusion Matrix: Machine Learning." POGIL Activity Clearinghouse 3, no. 4 (2022). https://pac.pogil.org/index.php/pac/article/view/304
• Mariyappan. D, Ganapathy. T, and Ramalingam. R. (2013). Survey and incidence of tomato leaf curl virus in Tamil Nadu.Pestology, vol. 37, no. 7, pp. 10-15.
• Mazumdar, P., Singh, P., Kethiravan, D., Ramathani, I. and Ramakrishnan, N. (2021). Late blight in tomato: insights into the pathogenesis of the aggressive pathogen Phytophthora infestans and future research priorities. Planta, 253(6), pp.1-24. https://doi.org/10.1007/s00425-021-03636-x
• Moriones, E. & Navas-Castillo, J. (2000). Tomato yellow leaf curl virus, an emerging virus complex causing epidemics worldwide. Virus research, 71(1-2), pp.123-134. https://doi.org/10.1016/S0168-1702(00)00193-3
• Nguyen, L.D., Lin, D., Lin, Z. and Cao, J. (2018). Deep CNNs for microscopic image classification by exploiting transfer learning and feature concatenation. In 2018 IEEE International Symposium on Circuits and Systems (ISCAS). pp. 1-5. doi: 10.1109/ISCAS.2018.8351550
• Oquab, M., Bottou, L., Laptev, I. and Sivic, J. (2014). Learning and transferring mid-level image representations using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1717-1724.
• Osdaghi, E., Jones, J.B., Sharma, A., Goss, E.M., Abrahamian, P., Newberry, E.A., Potnis, N., Carvalho, R., Choudhary, M., Paret, M.L. and Timilsina, S. (2021). A centenary for bacterial spot of tomato and pepper. Molecular Plant Pathology, 22(12), p.1500-1519. d oi: 10.1111/mpp.13125
• Pan, S.J. & Yang, Q. (2009). A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10), pp.1345-1359. doi: 10.1109/TKDE.2009.191
• Potnis, N., Timilsina, S., Strayer, A., Shantharaj, D., Barak, J.D., Paret, M.L., Vallad, G.E. and Jones, J.B. (2015). Bacterial spot of tomato and pepper: diverse X anthomonas species with a wide variety of virulence factors posing a worldwide challenge. Molecular plant pathology, 16(9), pp.907-920. https://doi.org/10.1111/mpp.12244
• Qasim Khan (2022), Tomato Disease Multiple Sources, CC0: Public Domain https://www.kaggle.com/datasets/cookiefinder/tomato-disease-multiple-sources
• Rangarajan, Aravind Krishnaswamy, Raja Purushothaman, and Aniirudh Ramesh. "Tomato crop disease classification using pre-trained deep learning algorithm." Procedia computer science 133 (2018): 1040-1047. https://doi.org/10.1016/j.procs.2018.07.070
• Saito, T. & Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PloS one, 10(3), p.e0118432. https://doi.org/10.1371/journal.pone.0118432
• Saleem, M. H., Potgieter, J., and Arif, K. M. (2020). Plant disease classification: A comparative evaluation of convolutional neural networks and deep learning optimizers. Plants, 9(10), 1319. https://doi.org/10.3390/plants9101319
• Saleem, M. H., Khanchi, S., Potgieter, J., and Arif, K. M. (2020). Image-based plant disease identification by deep learning meta-architectures. Plants, 9(11), 1451. https://doi.org/10.3390/plants9111451
• Shorten, C. & Khoshgoftaar, T.M. (2019). A survey on image data augmentation for deep learning. Journal of Big Data, 6(1), pp.1-48. https://doi.org/10.1186/s40537-019-0197-0
• Sukhija, S., Krishnan, N.C. and Singh, G. (2016). Supervised Heterogeneous Domain Adaptation via Random Forests. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, pp. 2039-2045. http://localhost:8080/xmlui/handle/123456789/2879
• Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. and Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826.
• Wang, C., Chen, D., Hao, L., Liu, X., Zeng, Y., Chen, J. and Zhang, G. (2019). Pulmonary image classification based on inception-v3 transfer learning model. IEEE Access, 7, pp.146533-146541. DOI: 10.1109/ACCESS.2019.2946000
• Wang, K., Gao, X., Zhao, Y., Li, X., Dou, D. and Xu, C.Z. (2019). Pay attention to features, transfer learn faster CNNs. In International Conference on Learning Representations, pp. 1-14.
• Weeraratne, W.A.P.G., Wijerathne, W.M.S.D.K. and Dissanayake, D.M.K.K., 2020. Occurrence of target spot of tomato caused by Corynespora cassiicola in Sri Lanka. Ceylon Journal of Science, 49(5), pp.397-400. http://doi.org/10.4038/cjs.v49i5.7807
• Wiatowski, T.& Bölcskei, H. (2017). A mathematical theory of deep convolutional neural networks for feature extraction. IEEE Transactions on Information Theory, vol. 64, no. 3. pp.1845-1866. DOI: 10.1109/ACCESS.2020.3024111
• Yang, Y., Liu, T., Shen, D., Wang, J., Ling, X., Hu, Z., Chen, T., Hu, J., Huang, J., Yu, W. and Dou, D. (2019). Tomato yellow leaf curl virus intergenic siRNAs target a host long noncoding RNA to modulate disease symptoms. PLoS pathogens, 15(1), pp. 1-22. https://doi.org/10.1371/journal.ppat.1007534
• Yoshida, K., Asano, S., Sushida, H. and Iida, Y. (2021). Occurrence of tomato leaf mold caused by novel race 2.4. 9 of Cladosporium fulvum in Japan. Journal of General Plant Pathology, 87(1), pp.35- 38. https://doi.org/10.1007/s10327-020-00963-x
• Zagoruyko, S. & Komodakis, N. (2016). Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. conference paper at ICLR, arXiv preprint arXiv:1612.03928. https://doi.org/10.48550/arXiv.1612.03928
• Zhang, H., Zhang, L. and Jiang, Y. (2019). Overfitting and underfitting analysis for deep learning based end-to-end communication systems. In 2019 11th International Conference on Wireless Communications and Signal Processing (WCSP), pp. 1-6. DOI: 10.1109/WCSP.2019.8927876
• Zhang, K., Wu, Q., Liu, A. and Meng, X. (2018). Can deep learning identify tomato leaf disease? Advances in Multimedia, Special issue, pp. 1-10. https://doi.org/10.1155/2018/6710865
• Zhou, B., Khosla, A., Lapedriza, A., Oliva, A. and Torralba, A . (2016). Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929.
• Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H. and He, Q. (2020). A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1), pp.43-76. DOI: 10.1109/JPROC.2020.3004555