ADAPTIVE WEIGHTED IOU-FOCAL (AWIF) LOSS FUNCTION FOR ENHANCED 3D OBJECT GENERATION: A COMPARATIVE STUDY USING 3D-VAE-GAN ARCHITECTURE

Abstract

This study evaluates the efficacy of a three-dimensional (3D) Variational Autoencoder-Generative Adversarial Network (3D-VAE-GAN) architecture in generating objects from images, with a focus on reconstruction loss functions. The analysis employs several loss functions, including Mean Squared Error (MSE), Binary Cross-Entropy (BCE), Focal, Dice, Intersection Over Union (IoU), and the newly developed Adaptive Weighted IoU-Focal (AWIF), to compare the effect of the metrics. In experiments conducted with 13 different objects from the ShapeNet dataset, both quantitative and qualitative assessments indicated that the AWIF loss function outperformed other loss functions. While volumetric overlap metrics (IoU) showed saturation due to the inherent sparsity of voxel grids, the proposed AWIF loss significantly improved surface geometric fidelity. Specifically, it reduced the Chamfer Distance (CD) error by 7.3% compared to MSE and 10.2% compared to standard IoU loss, proving its superior capability in reconstructing fine surface details. Furthermore, time analysis of the AWIF loss function was conducted, demonstrating its potential competitiveness in this domain.

Keywords

• 3D Object Generation
• 3D-VAE-GAN
• Loss Function
• Reconstruction

References

1. J. McCarthy, M. L. Minsky, N. Rochester, and C. E. Shannon, "A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence, August 31, 1955," AI Mag., vol. 27, no. 4, pp. 12-12, 2006.
2. S. J. Russell and P. Norvig, Artificial Intelligence a Modern Approach, 3rd ed. Pearson, 2016.
3. M. De Berg, O. Cheong, M. Van Kreveld, and M. Overmars, "Linear Programming: Manufacturing with Molds," in Computational Geometry. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008, pp. 63-93.
4. R. B. Rusu and S. Cousins, "3D Is Here: Point Cloud Library (pcl)," in Proc. 2011 IEEE International Conference on Robotics and Automation, IEEE, 2011, pp. 1-4.
5. W. E. Lorensen and H. E. Cline, "Marching Cubes: A High Resolution 3D Surface Construction Algorithm," in Seminal Graphics. New York, NY, USA: ACM, 1998, pp. 347-353.
6. Y. LeCun, Y. Bengio, and G. Hinton, "Deep Learning," Nature, vol. 521, no. 7553, pp. 436-444, 2015.
7. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet Classification with Deep Convolutional Neural Networks," Adv. Neural Inf. Process. Syst., vol. 25, 2012.
8. D. Maturana and S. Scherer, "VoxNet: A 3D Convolutional Neural Network for Real-Time Object Recognition," in Proc. 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2015, pp. 922-928.

9. Z. Wu et al., "3D ShapeNets: A Deep Representation for Volumetric Shapes," in Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 1912-1920.
10. D. P. Kingma and M. Welling, "Auto-Encoding Variational Bayes," arXiv Prepr. arXiv:1312.6114, 2013.
11. I. J. Goodfellow et al., "Generative Adversarial Nets," Adv. Neural Inf. Process. Syst., vol. 27, 2014.
12. J. Wu, C. Zhang, T. Xue, B. Freeman, and J. Tenenbaum, "Learning a Probabilistic Latent Space of Object Shapes Via 3D Generative-Adversarial Modeling," Adv. Neural Inf. Process. Syst., vol. 29, 2016.
13. T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, "Focal Loss for Dense Object Detection," in Proc. IEEE International Conference on Computer Vision, 2017, pp. 2980-2988.
14. F. Milletari, N. Navab, and S.-A. Ahmadi, "V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation," in Proc. 2016 Fourth International Conference on 3D Vision (3DV), 2016, pp. 565-571.
15. M. A. Rahman and Y. Wang, "Optimizing Intersection-Over-Union in Deep Neural Networks for Image Segmentation," in Advances in Visual Computing, vol. 10072,
16. G. Bebis, R. Boyle, B. Parvin, D. Koracin, F. Porikli, S. Skaff, A. Entezari, J. Min, D. Iwai, A. Sadagic, C. Scheidegger, and T. Isenberg, Eds. Cham: Springer International Publishing, 2016, pp. 234-244.
17. A. X. Chang et al., "ShapeNet: An Information-Rich 3D Model Repository," arXiv Prepr. arXiv:1512.03012, 2015.
18. C. B. Choy, D. Xu, J. Gwak, K. Chen, and S. Savarese, "3D-R2N2: A Unified Approach for Single and Multi-View 3D Object Reconstruction," in Computer Vision – ECCV 2016, vol. 9912, B. Leibe, J. Matas, N. Sebe, and M. Welling, Eds. Cham: Springer International Publishing, 2016, pp. 628-644.
19. P. Jaccard, "Étude Comparative De La Distribution Florale Dans Une Portion Des Alpes Et Des Jura," Bull Soc Vaudoise Sci Nat, vol. 37, pp. 547-579, 1901.
20. L. R. Dice, "Measures of the Amount of Ecologic Association Between Species," Ecology, vol. 26, no. 3, pp. 297-302, 1945.
21. N. Wang, Y. Zhang, Z. Li, Y. Fu, W. Liu, and Y.-G. Jiang, "Pixel2mesh: Generating 3d Mesh Models from Single Rgb Images," in Proc. European Conference on Computer Vision (ECCV), 2018, pp. 52-67.
22. H. Xie, H. Yao, X. Sun, S. Zhou, and S. Zhang, "Pix2vox: Context-Aware 3D Reconstruction from Single and Multi-View Images," in Proc. IEEE/CVF International Conference on Computer Vision, 2019, pp. 2690-2698.
23. Z. Chen, A. Tagliasacchi, and H. Zhang, "Bsp-Net: Generating Compact Meshes Via Binary Space Partitioning," in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 45-54.
24. X. Yan, L. Lin, N. J. Mitra, D. Lischinski, D. Cohen-Or, and H. Huang, "Shapeformer: Transformer-Based Shape Completion Via Sparse Representation," in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 6239-6249.
25. J. Gao et al., "Get3d: A Generative Model of High Quality 3d Textured Shapes Learned from Images," Adv. Neural Inf. Process. Syst., vol. 35, pp. 31841-31854, 2022.
26. H. Jun and A. Nichol, "Shap-E: Generating Conditional 3D Implicit Functions," arXiv Prepr. arXiv:2305.02463, 2023.
27. X. Ren, J. Huang, X. Zeng, K. Museth, S. Fidler, and F. Williams, "XCube: Large-Scale 3D Generative Modeling Using Sparse Voxel Hierarchies," in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 4209-4219.
28. M.-S. Yu et al., "A Variational Autoencoder Cascade Generative Adversarial Network for Scalable 3D Object Generation and Reconstruction," Sensors, vol. 24, no. 3, p. 751, 2024.
29. M. Liu et al., "One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion," in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 10072-10083.
30. Y. Siddiqui et al., "MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers," in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 19615-19625.
31. Y. Hong et al., "LRM: Large Reconstruction Model for Single Image to 3D," arXiv Prepr. arXiv:2311.04400, 2024.
32. M. Savva et al., "Shrec16 Track: Largescale 3D Shape Retrieval from ShapeNet Core55," in Proc. Eurographics Workshop on 3D Object Retrieval, 2016, p. 13.
33. G. E. Hinton and R. R. Salakhutdinov, "Reducing the Dimensionality of Data with Neural Networks," Science, vol. 313, no. 5786, pp. 504-507, Jul. 2006.
34. L. Metz, B. Poole, D. Pfau, and J. Sohl-Dickstein, "Unrolled Generative Adversarial Networks," arXiv Prepr. arXiv:1611.02163, 2017.
35. A. Radford, L. Metz, and S. Chintala, "Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks," arXiv Prepr. arXiv:1511.06434, 2016.
36. K. Janocha and W. M. Czarnecki, "On Loss Functions for Deep Neural Networks in Classification," arXiv Prepr. arXiv:1702.05659, 2017.
37. C. M. Bishop and N. M. Nasrabadi, Pattern Recognition and Machine Learning, vol. 4. Springer, 2006. T. Hastie, R. Tibshirani, J. Friedman, and J. Franklin, "The Elements of Statistical Learning: Data Mining, Inference and Prediction," Math. Intell., vol. 27, no. 2, pp. 83-85, 2005.
38. K. G. Kim, "Book Review: Deep Learning," Healthc. Inform. Res., vol. 22, no. 4, p. 351, 2016.
39. H. He and E. A. Garcia, "Learning from Imbalanced Data," IEEE Trans. Knowl. Data Eng., vol. 21, no. 9, pp. 1263-1284, 2009.
40. C. H. Sudre, W. Li, T. Vercauteren, S. Ourselin, and M. Jorge Cardoso, "Generalised Dice Overlap as a Deep Learning Loss Function for Highly Unbalanced Segmentations," in Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, vol. 10553,
41. M. J. Cardoso, T. Arbel, G. Carneiro, T. Syeda-Mahmood, J. M. R. S. Tavares, M. Moradi, A. Bradley, H. Greenspan, J. P. Papa, A. Madabhushi, J. C. Nascimento, J. S. Cardoso, V. Belagiannis, and Z. Lu, Eds. Cham: Springer International Publishing, 2017, pp. 240-248.
42. Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, "3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation," in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016, vol. 9901, S. Ourselin, L. Joskowicz, M. R. Sabuncu, G. Unal, and W. Wells, Eds. Cham: Springer International Publishing, 2016, pp. 424-432.
43. A. Sharma, O. Grau, and M. Fritz, "VConv-DAE: Deep Volumetric Shape Learning Without Object Labels," in Computer Vision – ECCV 2016 Workshops, vol. 9915, G. Hua and H. Jégou, Eds. Cham: Springer International Publishing, 2016, pp. 236-250.
44. S. Nowozin, "Optimal Decisions from Probabilistic Models: The Intersection-Over-Union Case," in Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 548-555.
45. H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese, "Generalized Intersection Over Union: A Metric and a Loss for Bounding Box Regression," in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 658-666.