3D reconstruction of coronary arteries using deep networks from synthetic X-ray angiogram data

Abstract

Cardiovascular disease (CVD) is one of the most common health problems that are responsible for one-third of all deaths around the globe. Although X-Ray angiography has deficiencies such as two-dimensional (2D) representation of three dimensional (3D) structures, vessel overlapping, noisy background, the existence of other tissues/organs in images, etc., it is used as the gold standard technique for the diagnosis and in some cases treatment of CVDs. To overcome the deficiencies, great efforts have been drawn on retrieval of actual 3D representation of coronary arterial tree from 2D X-ray angiograms. However, the proposed algorithms are based on analytical methods and enforce some constraints. With the evolution of deep neural networks, 3D reconstruction from images can be achieved effectively. In this study, we propose a new data structure for the representation of objects in a tubular shape for 3D reconstruction of arteries using deep learning. Moreover, we propose a method to generate synthetic coronaries from data of real subjects. Then, we validate tubular shape representation using 3 typical deep learning architectures with synthetic X-ray data we produced. The input to deep learning architectures is multi-view segmented X-Ray images and the output is the structured tubular representation. We compare results qualitatively in terms of visual appearance and quantitatively in terms of Chamfer Distance and Mean Squared Error. The results demonstrate that tubular representation has promising performance in 3D reconstruction of coronaries. We observe that convolutional neural network (CNN) based architectures yield better 3D reconstruction performance with 9.9e-3 on Chamfer Distance. On the other hand, LSTM-based network fails to learn the coronary tree structure and we conclude that LSTMs are not appropriate for auto-regression problems as depicted in this study.

Keywords

• 3D Reconstruction
• coronary artery tree
• deep learning
• synthetic coronary data-set

References

1. World health organization, https://www.who.int/news-room/fact-sheets/detail/cardiovascular-diseases-(cvds), Accessed: 2020-07-22.
2. Xiao, R., Yang, J., Fan, J., Ai, D., Wang, G., Wang, Y., Shape context and projection geometry constrained vasculature matching for 3d reconstruction of coronary artery, Neuro computing, 195 (2016), 65–73, https://dx.doi.org/10.1016/j.neucom.2015.08.110.
3. Zheng, S., Meiying, T., Jian, S., Sequential reconstruction of vessel skeletons from X-ray coronary angiographic sequences, Comput. Med. Imaging Graph., 34 (5) (2010), 333–345, https://dx.doi.org/10.1016/j.compmedimag.2009.12.004.
4. Fazlali, H. R., Karimi, N., Soroushmehr, S. M. R., Sinha, S., Samavi, S., Nalamothu, B., Najarian, K., Vessel region detection in coronary X-ray angiograms, In Proc. - Int. Conf. Image Process. ICIP (ICIP) (2015), IEEE, pp. 1493–1497, https://dx.doi.org/0.1109/ICIP.2015.7351049.
5. Medical radiation, https://www.medicalradiation.com/types-of-medical-imaging/imaging-using-x-rays/angiography/, Accessed: 2020-07-23.
6. Gers, F. A., Eck, D., Schmidhuber, J., Applying lstm to time series predictable through time-window approaches, In Neural Nets WIRN Vietri-01, Springer, 2002, pp. 193–200, https://dx.doi.org/10.1007/978-1-4471-0219-9 20.
7. Cong, W., Yang, J., Ai, D., Chen, Y., Liu, Y., Wang, Y., Quantitative analysis of deformable model-based 3-d reconstruction of coronary artery from multiple angiograms, IEEE Trans. Biomed. Eng., 62 (8) (2015), 2079–2090, https://dx.doi.org/10.1109/TBME.2015.2408633.
8. Yang, J., Cong, W., Chen, Y., Fan, J., Liu, Y., Wang, Y., External force back-projective composition and globally deformable optimization for 3-d coronary artery reconstruction, Phys. Med. Biol., 59 (4) (2014), 975, https://dx.doi.org/10.1088/0031-9155/59/4/975.

9. Chen, S. J., Carroll, J. D., 3-d reconstruction of coronary arterial tree to optimize angiographic visualization, IEEE Trans. Med. Imag., 19 (4) (2000), 318–336, https://dx.doi.org/10.1109/42.848183.
10. Chen, S.-Y., Carroll, J. D., Kinematic and deformation analysis of 4-d coronary arterial trees reconstructed from cine angiograms, IEEE Trans. Med. Imag., 22 (6) (2003), 710–721, https://dx.doi.org/10.1109/TMI.2003.814788.
11. Andriotis, A., Zifan, A., Gavaises, M., Liatsis, P., Pantos, I., Theodorakakos, A., Efstathopoulos, E. P., Katritsis, D., A new method of three-dimensional coronary artery reconstruction from x-ray angiography: Validation against a virtual phantom and multislice computed tomography, Catheter. Cardiovasc. Interv., 71 (1) (2008), 28–43, https://dx.doi.org/10.1002/ccd.21414.
12. Yang, J., Wang, Y., Liu, Y., Tang, S., Chen, W., Novel approach for 3-d reconstruction of coronary arteries from two uncalibrated angiographic images, IEEE Trans. Image Process., 18 (7) (2009), 1563–1572, https://dx.doi.org/10.1109/TIP.2009.2017363.
13. Liu, X., Hou, F., Hao, A., Qin, H., A parallelized 4d reconstruction algorithm for vascular structures and motions based on energy optimization, Vis. Comput., 31 (11) (2015), 1431–1446, https://dx.doi.org/10.1007/s00371-014-1024-4.
14. Sarry, L., Boire, J.-Y., Three-dimensional tracking of coronary arteries from biplane angiographic sequences using parametrically deformable models, IEEE Trans. Med. Imag., 20 (12) (2001), 1341–1351, https://dx.doi.org/10.1109/42.974929.
15. Canero, C., Vilari˜no, F., Mauri, J., Radeva, P., Predictive (un) distortion model and 3d reconstruction by biplane snakes, IEEE Trans. Med. Imag., 21 (9) (2002), 1188–1201, https://dx.doi.org/10.1109/TMI.2002.804421.
16. Hoffmann, K. R., Sen, A., Lan, L., Chua, K.-G., Esthappan, J., Mazzucco, M., A system for determination of 3d vessel tree centerlines from biplane images, Int. J. Card. Imag., 16 (5) (2000), 315–330, https://dx.doi.org/10.1023/A:1026528209003.
17. Shechter, G., Devernay, F., Coste-Maniere, E., Quyyumi, A., McVeigh, E. R., Three dimensional motion tracking of coronary arteries in biplane cineangiograms, IEEE Trans. Med. Imag., 22 (4) (2003), 493–503, https://dx.doi.org/10.1109/TMI.2003.809090.
18. Fallavollita, P., Cheriet, F., Optimal 3d reconstruction of coronary arteries for 3d clinical assessment, Comput. Med. Imaging Graph., 32 (6) (2008), 476–487, https://dx.doi.org/10.1016/j.compmedimag.2008.05.001.
19. Wiesent, K., Barth, K., Navab, N., Durlak, P., Brunner, T., Schuetz, O., Seissler, W., Enhanced 3-d-reconstruction algorithm for c-arm systems suitable for interventional procedures, IEEE Trans. Med. Imag., 19 (5) (2000), 391–403, https://dx.doi.org/10.1109/42.870250.
20. Liao, R., Luc, D., Sun, Y., Kirchberg, K., 3-d reconstruction of the coronary artery tree from multiple views of a rotational X-ray angiography, Int. J. Card. Imag., 26 (7) (2010), 733–749, https://dx.doi.org/10.1007/s10554-009-9528-0.
21. Torr, P. H., Murray, D. W., The development and comparison of robust methods for estimating the fundamental matrix, Int. J. Comput. Vis., 24 (3) (1997), 271–300, https://dx.doi.org/10.1023/A:1007927408552.
22. Navaneet, K., Mandikal, P., Agarwal, M., Babu, R. V., Capnet: Continuous approximation projection for 3-d point cloud reconstruction using 2d supervision, In Proceedings of the AAAI Conference on Artificial Intelligence (2019), vol. 33, pp. 8819–8826, https://dx.doi.org/10.1609/aaai.v33i01.33018819.
23. Fan, H., Su, H., Guibas, L. J., A point set generation network for 3d object reconstruction from a single image, In CVPR (2017), pp. 605–613.
24. Zamorski, M., Zieba, M., Klukowski, P., Nowak, R., Kurach, K., Stokowiec, W., Trzcinski, T., Adversarial autoencoders for compact representations of 3d point clouds, Comput. Vis. Image Underst., 193 (2020), 102921, https://dx.doi.org/10.1016/j.cviu.2020.102921.
25. Lun, Z., Gadelha, M., Kalogerakis, E., Maji, S., Wang, R., 3d shape reconstruction from sketches via multi-view convolutional networks, In Proc. - 2017 Int. Conf. 3D Vis. (3DV) (2017), IEEE, pp. 67–77, https://dx.doi.org/10.1109/3DV.2017.00018.
26. Arsalan Soltani, A., Huang, H., Wu, J., Kulkarni, T. D., Tenenbaum, J. B., Synthesizing 3d shapes via modeling multi-view depth maps and silhouettes with deep generative networks, In CVPR (2017), pp. 1511–1519.
27. Choy, C. B., Xu, D., Gwak, J., Chen, K., Savarese, S., 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction, In ECCV (2016), Springer, pp. 628–644, https://dx.doi.org/10.1007/978-3-319-46484-8 38.
28. Riegler, G., Osman Ulusoy, A., Geiger, A., Octnet: Learning deep 3d representations at high resolutions, In CVPR (2017), pp. 3577–3586.
29. Tatarchenko, M., Dosovitskiy, A., Brox, T., Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs, In Proc. IEEE Int. Conf. Comput. Vis. (2017), pp. 2088–2096.
30. Yan, X., Yang, J., Yumer, E., Guo, Y., Lee, H., Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision, In Adv. Neural Inf. Process. Syst. (2016), pp. 1696–1704.
31. Wang, P.-S., Liu, Y., Guo, Y.-X., Sun, C.-Y., Tong, X., O-cnn: Octree-based convolutional neural networks for 3d shape analysis, ACM Trans. Graph., 36 (4) (2017), 1–11, https://dx.doi.org/10.1145/3072959.3073608.
32. Xie, H., Yao, H., Sun, X., Zhou, S., Tong, X., Weighted voxel: a novel voxel representation for 3d reconstruction, In Proceedings of the 10th International Conference on Internet Multimedia Computing and Service (2018), pp. 1–4, https://dx.doi.org/10.1145/3240876.3240888.
33. Hane, C., Tulsiani, S., Malik, J., Hierarchical surface prediction for 3d object reconstruction, In Proc. - 2017 Int. Conf. 3D Vis. (3DV) (2017), IEEE, pp. 412–420, https://dx.doi.org/10.1109/3DV.2017.00054.
34. Paschalidou, D., Ulusoy, O., Schmitt, C., Van Gool, L., Geiger, A., Raynet: Learning volumetric 3d reconstruction with ray potentials, In CVPR (2018), pp. 3897–3906.
35. Wang, N., Zhang, Y., Li, Z., Fu, Y., Liu, W., Jiang, Y.-G., Pixel2mesh: Generating 3d mesh models from single rgb images, In ECCV (2018), pp. 52–67.
36. Pontes, J. K., Kong, C., Sridharan, S., Lucey, S., Eriksson, A., Fookes, C., Image2mesh: A learning framework for single image 3d reconstruction, In Asian Conference on Computer Vision (2018), Springer, pp. 365–381, https://dx.doi.org/10.1007/978-3-030-20887-5 23.
37. Tatarchenko, M., Dosovitskiy, A., Brox, T., Multi-view 3d models from single images with a convolutional network, In ECCV (2016), Springer, pp. 322–337, https://dx.doi.org/10.1007/978-3-319-46478-7 20.
38. Girdhar, R., Fouhey, D. F., Rodriguez, M., Gupta, A., Learning a predictable and generative vector representation for objects, In ECCV (2016), Springer, pp. 484–499, https://dx.doi.org/10.1007/978-3-319-46466-4 29.
39. Chang, A. X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al., Shapenet: An information-rich 3d model repository, arXiv:1512.03012 (2015).
40. Everingham, M., Van Gool, L., Williams, C., Winn, J., Zisserman, A., The pascal visual object classes challenge 2012 (voc2012), Results (2012).
41. Blanco, P. J., Bulant, C. A., Muller, L. O., Talou, G. M., Bezerra, C. G., Lemos, P., Feijo, R. A., Comparison of 1d and 3d models for the estimation of fractional flow reserve, Sci. Rep., 8 (1) (2018), 1–12, https://dx.doi.org/10.1038/s41598-018-35344-0.
42. Horn, F., Leghissa, M., Kaeppler, S., Pelzer, G., Rieger, J., Seifert, M., Wandner, J., Weber, T., Michel, T., Riess, C., et al., Implementation of a talbot-lau interferometer in a clinical-like c-arm setup: A feasibility study, Sci. Rep., 8 (1) (2018), 1–11, https://dx.doi.org/10.1038/s41598-018-19482-z.