VR-Assisted Comparative Analysis of HEVC Intra Prediction Modes Using Deep Learning and Rule-Based Systems

Abstract

This paper proposes an artificial intelligence-based solution to classify the three basic intrinsic prediction modes (Planar, DC and Angular) defined in the High Efficiency Video Coding (HEVC) standard. A convolutional neural network (CNN) based deep learning model trained with 32×32 blocks obtained from 30+ classical gray level test images is developed. As a result of the training, the model demonstrated a successful classification performance with an overall accuracy of over 89% and a macro F1 score of approximately 88%. The model was converted into ONNX format and integrated into a Unity-based virtual reality (VR) environment, thus creating an interactive analysis platform where users can observe the predictions of both artificial intelligence and rule-based systems at the block level comparatively. In this environment, users can also examine the reasoning of the predictions. The proposed system provides a holistic solution in terms of classification performance, interpretability and user experience, and makes innovative contributions to the analysis and visualization of video coding processes for educational purposes.

Keywords

• HEVC
• Intra prediction
• Convolutional neural network
• Rule-based expert system
• VR visulation
• Video coding

References

1. Adeniji I, Casarona M, Bielory L, Bancairen L, Menzel M, Perigo N, Blackmon C, Niepielko MG, Insley J, Joiner D. 2024. Using Unity for Scientific Visualization as a Coursebased Undergraduate Research Experience. J Comput Sci Educ, 15(1): 35-40. https://doi.org/10.22369/issn.2153-4136/15/1/7
2. Askin MB, Celikcan U. 2022. Learning based versus heuristic based: A comparative analysis of visual saliency prediction in immersive virtual reality. Comput Animat Virtual Worlds, 33(6): e2106. https://doi.org/10.1002/cav.2106
3. Cucchiara R, Piccardi M, Mello P. 2000. Image Analysis and rule-based reasoning for a traffic monitoring system. IEEE Transact Intel Transport Syst, 1(2): 119-130.
4. Cui W, Zhang T, Zhang S, Jiang F, Zuo W, Zhao D. 2018. Convolutional neural networks based intra prediction for HEVC. http://arxiv.org/abs/1808.05734
5. Feng A, Gao C, Li L, Liu D, Wu F. 2021. Cnn-based depth map prediction for fast block partitioning in hevc intra coding. IEEE International Conference on Multimedia and Expo (ICME), Jult 05-09, Shenzhen, China, pp: 1-6. https://doi.org/10.1109/ICME51207.2021.9428069
6. Galpin F, Racape F, Jaiswal S, Bordes P, Leannec F, Francois E. 2019. CNN-based driving of block partitioning for intra slices encoding. Data Compression Conference, March 26-29, Snowbird, UT, US, pp: 162-171. https://doi.org/10.1109/DCC.2019.00024
7. Imen W, Amna M, Fatma B, Ezahra SF, Masmoudi N. 2022. Fast HEVC intra-CU decision partition algorithm with modified LeNet-5 and AlexNet. Signal Image Video Proces, 16(7): 1811-1819. https://doi.org/10.1007/s11760-022-02139-w
8. Ishikawa S, Todo M, Taki M, Uchiyama Y, Matsunaga K, Lin P, Ogihara T, Yasui M. 2023. Example-based explainable AI and its application for remote sensing image classification. Int J Appl Earth Obser Geoinfo, 118: 103215. https://doi.org/10.1016/j.jag.2023.103215

9. Islam RU, Hossain MS, Andersson K. 2020. A deep learning inspired belief rule-based expert system. IEEE Access, 8: 190637-190651. https://doi.org/10.1109/ACCESS.2020.3031438
10. Kaplan M, Akman A. 2024. Parallel implementation of discrete cosine transform (DCT) methods on GPU for HEVC, 2nd International Congress of Electrical and Computer Engineering. pp. 281–293. https://doi.org/10.1007/978-3-031-52760-9_20
11. Kuanar S, Rao KR, Bilas M, Bredow J. 2019. Adaptive CU mode selection in HEVC intra prediction: A deep learning approach. Circuits Syst Signal Proces, 38(11): 5081-5102. https://doi.org/10.1007/s00034-019-01110-4
12. Laude T, Ostermann J. 2016. Deep learning-based intra prediction mode decision for HEVC. 2016 Picture Coding Symposium (PCS), December 04-07, Nuremberg, Germany, pp: 1-5. https://doi.org/10.1109/PCS.2016.7906399
13. Li N, Wang Z, Zhang Q, He L, Zhang W. 2025. Fast intra-prediction mode decision algorithm for versatile video coding based on gradient and convolutional neural network. Electronics, 14(10): 2031. https://doi.org/10.3390/electronics14102031
14. Liu X, Li Y, Liu D, Wang P, Yang LT. 2019. An adaptive CU size decision algorithm for HEVC intra prediction based on complexity classification using machine learning. IEEE Transact Circuits Syst Video Technol, 29(1): 144-155. https://doi.org/10.1109/TCSVT.2017.2777903
15. Moran A, Gadepally V, Hubbell M, Kepner J. 2015. Improving big data visual analytics with interactive virtual reality. 2015 IEEE High Performance Extreme Computing Conference (HPEC), September 15-17, Waltham, MA, US, pp: 1-6. https://doi.org/10.1109/HPEC.2015.7322473
16. Pattimi H, Srinivasarao BKN. 2024. High-speed coding unit depth identifications using CU-VGG deep learning architectures. Arabian J Sci Eng, 49: 16287-16298. https://doi.org/10.1007/s13369-024-08928-4
17. Ren W, Su J, Sun C, Shi Z. 2019. An IBP-CNN Based fast block partition for intra prediction. 2019 Picture Coding Symposium (PCS), November 12-15, Ningbo, China, pp: 1-5. https://doi.org/10.1109/PCS48520.2019.8954522
18. Schorr C, Goodarzi P, Chen F, Dahmen T. 2021. Neuroscope: An explainable AI toolbox for semantic segmentation and image classification of convolutional neural nets. Appl Sci, 11(5): 2199. https://doi.org/10.3390/app11052199
19. Sullivan GJ, Ohm JR, Han WJ, Wiegand T. 2012. Overview of the high efficiency video coding (HEVC) standard. IEEE Transact Circuits Syst Video Technol, 22(12): 1649-1668. https://doi.org/10.1109/TCSVT.2012.2221191
20. Swamy TN, Ramesha K. 2022. Improved intra prediction algorithm for HEVC with conventional and convolutional neural network approach. Math Stat Eng Appl, 71(2): 272-280.
21. Zaki F, Mohamed AE, Sayed SG. 2021. CtuNet: A deep learning-based Framework for Fast CTU Partitioning of H265/HEVC Intra- coding. Ain Shams Eng J, 12(2): 1859-1866. https://doi.org/10.1016/j.asej.2021.01.001
22. Zhao L, Zhang L, Ma S, Zhao D. 2011. Fast mode decision algorithm for intra prediction in HEVC. 2011 Visual Communications and Image Processing (VCIP), November 06-09, Tainan, Taiwan, pp: 1-4. https://doi.org/10.1109/VCIP.2011.6115979
23. Zisad SN, Chowdhury E, Hossain MS, Islam RU, Andersson K. 2021. An integrated deep learning and belief rule-based expert system for visual sentiment analysis under uncertainty. Algorithms, 14(7): 213. https://doi.org/10.3390/a14070213