Transfer Learning based Image Classification of Diseased Tomato Leaves with Optimal Fine-Tuning combined with Heat Map Visualization

Abstract

Plant disease detection and disease classification at initial stages for sensitive commodities like tomatoes and potatoes is highly mandated as the harvest losses have a direct impact on the price fixation of the vegetables. The most identified limitation in the study of plant pathology is the availability of datasets with visual symptoms that covers all the possible diseases of one crop or plant species. Computer Vision systems and advancements in deep learning-based modeling methodologies gained significant attention in smart farming. It is presumed that the implementation of deep learning algorithms demands a large amount of data to learn complex features automatically and this can pose a challenge for applications with lesser data to achieve generalization. In such cases, Transfer Learning with optimum regularization techniques and fine-tuning mechanisms is the solution to overcome the limitations of smaller datasets. The objective of the work is to develop Tomato Disease Classification System using a transfer learning approach for ten tomato disease classes of the PlantVillage dataset downloaded from the Kaggle platform. Inception V3, a pre-trained transfer learning model is used to classify this multi-class, imbalanced, tomato plant disease based on the leaf symptoms such as dark brown lesions, concentric rings, etc. Geometrical data augmentation is used as a regularization technique to expand the size of the dataset. Significant improvement in the performance metrics is observed when the finetuning is optimum. The training accuracy and validation accuracy of the model before and after fine-tuning are 97.08%, 83.52%, and 98.19%, 95.93% respectively. The average accuracy with augmentation and optimal fine-tuning is 98%. In addition, prediction scores in terms of precision, recall, and F1-score are obtained to visualize the rate of mispredictions across the disease classes. It is observed that the misprediction rate is high across the classes early blight, late blight, and Septoria spot due to similar visual symptoms. Further, activations are used to generate an attention map in the form of Heat Maps which are included as a post-processing step before the classification of the output. Plant Leaf Disease Classification- A web application is deployed using Streamlit Python library and Ngrok services.

Keywords

• Image classification
• Transfer learning
• InceptionV3
• Attention maps

References

1. Adhikari, P., Oh, Y. and Panthee, D.R. (2017). Current status of early blight resistance in tomato: an update. International journal of molecular sciences, 18(10), pp.1-22 https://doi.org/10.3390/ijms18102019 .
2. Agarwal, Mohit, Abhishek Singh, Siddhartha Arjaria, Amit Sinha, and Suneet Gupta. "ToLeD: Tomato leaf disease detection using convolution neural network." Procedia Computer Science 167 (2020): 293-301. https://doi.org/10.1016/j.procs.2020.03.225
3. Al‐gaashani, Mehdhar SAM, Fengjun Shang, Mohammed SA Muthanna, Mashael Khayyat, and Ahmed A. Abd El‐Latif. "Tomato leaf disease classification by exploiting transfer learning and feature concatenation." IET Image Processing 16, no. 3 (2022): 913-925. https://doi.org/10.1049/ipr2.12397
4. Altman, Douglas G., and J. Martin Bland. "Diagnostic tests. 1: Sensitivity and specificity." BMJ: British Medical Journal 308, no. 6943 (1994): 1552. doi: 10.1136/bmj.308.6943.1552 .
5. Astani, M., Hasheminejad, M, and Vaghefi, M. (2022). A diverse ensemble classifier for tomato disease recognition. Computers and Electronics in Agriculture, 198, 107054. https://doi.org/10.1016/j.compag.2022.107054
6. Basavaiah, Jagadeesh, and Audre Arlene Anthony. "Tomato leaf disease classification using multiple feature extraction techniques." Wireless Personal Communications 115, no. 1 (2020): 633-651
7. Bengio, Y., Bastien, F., Bergeron, A., Boulanger–Lewandowski, N., Breuel, T., Chherawala, Y., and Sicard, G. (2011). Deep learners benefit more from out-of-distribution examples. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings, pp. 164-172.
8. Dong, N., Zhao, L., Wu, C. H., & Chang, J. F. (2020). Inception v3 based cervical cell classification combined with artificially extracted features. Applied Soft Computing, 93, 106311. https://doi.org/10.1016/j.asoc.2020.106311.

9. Feuz, K.D. & Cook, D.J. (2015). Transfer learning across feature-rich heterogeneous feature spaces via feature-space remapping (FSR). ACM Transactions on Intelligent Systems and Technology (TIST), 6(1), pp.1-27. https://doi.org/10.1145/2629528 .
10. Gnanasekaran, A& Vijayalakshmi.S. (2012). An economic analysis of tomato cultivation in Dindigul district of Tamil Nadu. International Journal of Scienceand research, vol. 3, pp. 995-997.
11. Goh, A. M., & Yann, X. L. (2021). Food-image Classification Using Neural Network Model. Int. J. of Electronics Engineering and Applications, 9(3), pp.12-22.
12. Guo, Y., Liu, Y., Georgiou, T., & Lew, M. S. (2018). A review of semantic segmentation using deep neural networks. International journal of multimedia information retrieval, 7,pp. 87-93. https://doi.org/10.1007/s13735-017-0141-z
13. Harakannanavar, S. S., Rudagi, J. M., Puranikmath, V. I., Siddiqua, A., and Pramodhini, R. (2022). Plant leaf disease detection using computer vision and machine learning algorithms. Global Transitions Proceedings, 3(1), pp. 305-310. https://doi.org/10.1016/j.gltp.2022.03.016 .
14. Hossain, S., Mou, R. M., Hasan, M. M., Chakraborty, S., and Razzak, M. A. (2018). Recognition and detection of tea leaf's diseases using support vector machine. In 2018 IEEE 14th International Colloquium on Signal Processing & Its Applications (CSPA), pp. 150-154. doi: 10.1109/CSPA.2018.8368703
15. H. Zhang, L. Zhang and Y. Jiang. (2019). Overfitting and Underfitting Analysis for Deep Learning Based End-to-end Communication Systems. 11th International Conference on Wireless Communications and Signal Processing,doi: 10.1109/WCSP.2019.8927876, pp. 1-6. doi: 10.1109/WCSP.2019.8927876
16. Hussain, M., Bird, J.J. and Faria, D.R. (2018). A study on cnn transfer learning for image classification. In UK Workshop on computational Intelligence UK, Springer, Cham, pp. 191-202. https://doi.org/10.1007/978-3-319-97982-3_16
17. Ibrahim. H (2019). Susceptibility Studies on Two Varieties of Tomato (Lycopersicon esculentum) to Fungal Leaf Spots. EAS Journal of Nutrition and Food Sciences, vol. 1, no.1, pp. 11-15. doi: 10.36349/easjnfs.2019.v01i01.003
18. Jiang, P., Chen, Y., Liu, B., He, D., and Liang, C. (2019). Real-time detection of apple leaf diseases using deep learning approach based on improved convolutional neural networks. IEEE Access, 7, pp. 59069-59080. doi: 10.1109/ACCESS.2019.2914929
19. Jiang, Ding, Fudong Li, Yuequan Yang, and Song Yu. "A tomato leaf diseases classification method based on deep learning." In 2020 chinese control and decision conference (CCDC), pp. 1446- 1450. IEEE, 2020. DOI: 10.1109/CCDC49329.2020.9164457
20. Jignesh Chowdary, G., Punn, N. S., Sonbhadra, S. K., and Agarwal, S. (2020). Face mask detection using transfer learning of inceptionv3. In Big Data Analytics: 8th International Conference, BDA 2020, Sonepat, India, December 15–18, 2020, Proceedings 8, pp. 81-90. Springer International Publishing. https://doi.org/10.1007/978-3-030-66665-1_6
21. Kubota, K., Tsuda, S., Tamai, A. and Meshi, T. (2003). Tomato mosaic virus replication protein suppresses virus-targeted posttranscriptional gene silencing. Journal of virology, 77(20), pp.11016-11026. https://doi.org/10.1128/JVI.77.20.11016-11026.2003
22. Liu, J., Chafi, R., Legarrea, S., Alba, J. M., Meijer, T., Menken, S. B., and Kant, M. R. (2020). Spider mites cause more damage to tomato in the dark when induced defenses are lower. Journal of chemical ecology, 46, pp.631-641. https://doi.org/10.1007/s10886-020-01195-1
23. Liu, J., and Wang, X. (2021). Plant diseases and pest detection based on deep learning: a review. Plant Methods, 17, pp. 1-18. https://doi.org/10.1186/s13007-021-00722-9
24. Liang, Jingsai. "Confusion Matrix: Machine Learning." POGIL Activity Clearinghouse 3, no. 4 (2022). https://pac.pogil.org/index.php/pac/article/view/304
25. Mariyappan. D, Ganapathy. T, and Ramalingam. R. (2013). Survey and incidence of tomato leaf curl virus in Tamil Nadu.Pestology, vol. 37, no. 7, pp. 10-15.
26. Mazumdar, P., Singh, P., Kethiravan, D., Ramathani, I. and Ramakrishnan, N. (2021). Late blight in tomato: insights into the pathogenesis of the aggressive pathogen Phytophthora infestans and future research priorities. Planta, 253(6), pp.1-24. https://doi.org/10.1007/s00425-021-03636-x
27. Moriones, E. & Navas-Castillo, J. (2000). Tomato yellow leaf curl virus, an emerging virus complex causing epidemics worldwide. Virus research, 71(1-2), pp.123-134. https://doi.org/10.1016/S0168-1702(00)00193-3
28. Nguyen, L.D., Lin, D., Lin, Z. and Cao, J. (2018). Deep CNNs for microscopic image classification by exploiting transfer learning and feature concatenation. In 2018 IEEE International Symposium on Circuits and Systems (ISCAS). pp. 1-5. doi: 10.1109/ISCAS.2018.8351550
29. Oquab, M., Bottou, L., Laptev, I. and Sivic, J. (2014). Learning and transferring mid-level image representations using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1717-1724.
30. Osdaghi, E., Jones, J.B., Sharma, A., Goss, E.M., Abrahamian, P., Newberry, E.A., Potnis, N., Carvalho, R., Choudhary, M., Paret, M.L. and Timilsina, S. (2021). A centenary for bacterial spot of tomato and pepper. Molecular Plant Pathology, 22(12), p.1500-1519. d oi: 10.1111/mpp.13125
31. Pan, S.J. & Yang, Q. (2009). A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10), pp.1345-1359. doi: 10.1109/TKDE.2009.191
32. Potnis, N., Timilsina, S., Strayer, A., Shantharaj, D., Barak, J.D., Paret, M.L., Vallad, G.E. and Jones, J.B. (2015). Bacterial spot of tomato and pepper: diverse X anthomonas species with a wide variety of virulence factors posing a worldwide challenge. Molecular plant pathology, 16(9), pp.907-920. https://doi.org/10.1111/mpp.12244
33. Qasim Khan (2022), Tomato Disease Multiple Sources, CC0: Public Domain https://www.kaggle.com/datasets/cookiefinder/tomato-disease-multiple-sources
34. Rangarajan, Aravind Krishnaswamy, Raja Purushothaman, and Aniirudh Ramesh. "Tomato crop disease classification using pre-trained deep learning algorithm." Procedia computer science 133 (2018): 1040-1047. https://doi.org/10.1016/j.procs.2018.07.070
35. Saito, T. & Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PloS one, 10(3), p.e0118432. https://doi.org/10.1371/journal.pone.0118432
36. Saleem, M. H., Potgieter, J., and Arif, K. M. (2020). Plant disease classification: A comparative evaluation of convolutional neural networks and deep learning optimizers. Plants, 9(10), 1319. https://doi.org/10.3390/plants9101319
37. Saleem, M. H., Khanchi, S., Potgieter, J., and Arif, K. M. (2020). Image-based plant disease identification by deep learning meta-architectures. Plants, 9(11), 1451. https://doi.org/10.3390/plants9111451
38. Shorten, C. & Khoshgoftaar, T.M. (2019). A survey on image data augmentation for deep learning. Journal of Big Data, 6(1), pp.1-48. https://doi.org/10.1186/s40537-019-0197-0
39. Sukhija, S., Krishnan, N.C. and Singh, G. (2016). Supervised Heterogeneous Domain Adaptation via Random Forests. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, pp. 2039-2045. http://localhost:8080/xmlui/handle/123456789/2879
40. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. and Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826.
41. Wang, C., Chen, D., Hao, L., Liu, X., Zeng, Y., Chen, J. and Zhang, G. (2019). Pulmonary image classification based on inception-v3 transfer learning model. IEEE Access, 7, pp.146533-146541. DOI: 10.1109/ACCESS.2019.2946000
42. Wang, K., Gao, X., Zhao, Y., Li, X., Dou, D. and Xu, C.Z. (2019). Pay attention to features, transfer learn faster CNNs. In International Conference on Learning Representations, pp. 1-14.
43. Weeraratne, W.A.P.G., Wijerathne, W.M.S.D.K. and Dissanayake, D.M.K.K., 2020. Occurrence of target spot of tomato caused by Corynespora cassiicola in Sri Lanka. Ceylon Journal of Science, 49(5), pp.397-400. http://doi.org/10.4038/cjs.v49i5.7807
44. Wiatowski, T.& Bölcskei, H. (2017). A mathematical theory of deep convolutional neural networks for feature extraction. IEEE Transactions on Information Theory, vol. 64, no. 3. pp.1845-1866. DOI: 10.1109/ACCESS.2020.3024111
45. Yang, Y., Liu, T., Shen, D., Wang, J., Ling, X., Hu, Z., Chen, T., Hu, J., Huang, J., Yu, W. and Dou, D. (2019). Tomato yellow leaf curl virus intergenic siRNAs target a host long noncoding RNA to modulate disease symptoms. PLoS pathogens, 15(1), pp. 1-22. https://doi.org/10.1371/journal.ppat.1007534
46. Yoshida, K., Asano, S., Sushida, H. and Iida, Y. (2021). Occurrence of tomato leaf mold caused by novel race 2.4. 9 of Cladosporium fulvum in Japan. Journal of General Plant Pathology, 87(1), pp.35- 38. https://doi.org/10.1007/s10327-020-00963-x
47. Zagoruyko, S. & Komodakis, N. (2016). Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. conference paper at ICLR, arXiv preprint arXiv:1612.03928. https://doi.org/10.48550/arXiv.1612.03928
48. Zhang, H., Zhang, L. and Jiang, Y. (2019). Overfitting and underfitting analysis for deep learning based end-to-end communication systems. In 2019 11th International Conference on Wireless Communications and Signal Processing (WCSP), pp. 1-6. DOI: 10.1109/WCSP.2019.8927876
49. Zhang, K., Wu, Q., Liu, A. and Meng, X. (2018). Can deep learning identify tomato leaf disease? Advances in Multimedia, Special issue, pp. 1-10. https://doi.org/10.1155/2018/6710865
50. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A. and Torralba, A . (2016). Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929.
51. Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H. and He, Q. (2020). A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1), pp.43-76. DOI: 10.1109/JPROC.2020.3004555