Semantic Segmentation to Extract Coronary Arteries in Fluoroscopy Angiograms ============================================================================ * Chen Zhao * Haipeng Tang * Jinshan Tang * Chaoyang Zhang * Zhuo He * Yu-Ping Wang * Hong-Wen Deng * Robert Bober * Weihua Zhou ## Abstract Coronary artery disease (CAD) is the leading cause of death worldwide, constituting more than one-fourth of global mortalities every year. Accurate semantic segmentation of each artery in fluoroscopy angiograms is important for assessment of the stenosis and CAD diagnosis and treatment. However, due to the morphological similarity among different types of arteries, it is hard for deep-learning-based models to generate semantic segmentation with an end-to-end approach. In this paper, we propose a multi-step semantic segmentation algorithm based on the analysis of graphs extracted from fluoroscopy angiograms. The proposed algorithm firstly extracts the entire arterial binary mask (binary vascular tree) by Feature Pyramid U-Net++. Then we extract the centerline of the binary vascular tree and separate it into different vessel segments. Finally, by extracting the underlying arterial topology, position and pixel features, we construct a powerful coronary artery classifier based on random forest. Each vessel segment is classified into left coronary artery (LCA), left anterior descending (LAD) and other types of arterial segments. We tested the proposed method on a dataset with 69 LAO and 103 RAO fluoroscopic angiograms and achieved classification accuracies of 66.4% and 61.49% respectively. The experimental results show the effectiveness of the proposed algorithm, which can be used to analyze the individual arteries in fluoroscopy angiograms. Keywords * Fluoroscopy Angiogram * Deep Learning * Semantic Segmentation * U-Net ## 1 Introduction Coronary artery disease (CAD) is the leading cause of morbidity and mortality in the United States and costs about 350 billion dollars annually [1]. The gold standard for diagnosis and therapeutic decisions for CAD is invasive coronary angiography (ICA). ICA involves an injection of contrast into the epicardial arteries with acquisition of continuous fluoroscopy. Detection of CAD is performed by Doctors through the comparison of the visualization of the diseased vessel segments within arteries and the normal vessel segments. X-ray fluoroscopy imaging is the most commonly used imaging modality to visualize blood vessels in interventional cardiology [2]. Assessment of vessel structures and territories is essential for the diagnosis and treatment. The semantic segmentation of coronary vessels is extremely important in clinical practice of interventional cardiology: for CAD clinical decisions, successfully detecting the percent stenosis of a coronary artery would entail real-time guidance [3]. The left coronary artery (LCA) is the artery that arises from the aorta above the left cusp of the aortic valve and feeds blood to the left side of the heart. It is the left main stem coronary artery. The LCA branches into the left anterior descending (LAD) artery, which courses between the left and right ventricles towards the LV apex on the anterior wall of the heart, and the left circumflex artery (LCX) courses towards the left inferior direction along the atrioventricular groove. Typically, radiologists analyze the whole vascular tree according to the position and morphological tendency of LAD and LCX. Hence, the first step is to extract LAD and LCX. However, the extraction of LAD and LCX depends on manual segmentation, which is time-consuming and tedious. ![Fig. 1.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/05/27/2020.05.26.20103440/F1.medium.gif) [Fig. 1.](http://medrxiv.org/content/early/2020/05/27/2020.05.26.20103440/F1) Fig. 1. Illustration of semantic segmentation of vessels in (a) Left Anterior Oblique (LAO) view and (b) Right Anterior Oblique (RAO) view. For each view, the raw image and semantic maps are juxtaposed vertically. Recently, deep learning techniques, especially variants based on convolutional neural networks (CNN), have been employed on vessel segmentation in coronary angiography. In [4], the authors implemented a CNN network which contains 2 convolutional layers to predict the patched vessel binary mask. In [5], a CNN was used to classify the central pixel within the cropped patches into vessel pixel or background. Even though the proposed approaches have achieved a state-of-the-art performance on vessel segmentation in fluoroscopic images, the methods are not suitable for semantic segmentation to extract individual vessel segments. The existing approaches extracted the entire vascular tree in X-ray angiography and did not take the position and topology of vessel segments into consideration. Hence, current approaches have a major limitation: it is difficult to assess vessel anatomy in fluoroscopy images due to the very low contrast, moving objects, inconstant contrast agent, and limited view angles [6]. The morphological similarity among different types of arteries brings challenges to learning-based models for semantic segmentation. In this paper, we propose a new hybrid deep-learning and graph-based approach to extract individual coronary arteries in fluoroscopy angiograms by incorporating global position, topology and pixel information. The focus of this work is on the classification of vessel segments into LAD, LCX, D1, OM1 and other vessels. Initially we focus on the segmentation of the entire vascular tree and then we extract the LAD and LCX (major vessels [7]]) from the X-ray fluoroscopy images by using the designed *Feature Pyramid U-Net++* (FP-U-Net++). After that, we extract the centerline of the whole vascular tree, and find the key points to generate the vessel graph. Finally, we extract positional and topological features from vessel segments and graph edge and perform the semantic segmentation. The workflow of the proposed approach is shown in Fig. 2. ![Fig. 2.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/05/27/2020.05.26.20103440/F2.medium.gif) [Fig. 2.](http://medrxiv.org/content/early/2020/05/27/2020.05.26.20103440/F2) Fig. 2. Workflow of coronary arteries semantic segmentation. ## 2 Methods The proposed method contains five steps: (1) Generation of binary vascular tree using multi-output U-Net++; (2) Extraction of LAD and LCX; (3) Generation of vessel graph; (4) Feature extraction by considering position, topology and pixels of vessel segments; (5) Vessel segment labeling. Each step is depicted in the Fig. 2 from (a) to (e). ### 2.1 Vascular Tree and Major Vessels Segmentation In general, current models for medical image segmentation are variants of the encoder-decoder based architecture, such as U-Net [8]. Many recent networks employed the classification network with the pre-trained weights in ImageNet [9] as the backbone of the encoder. In U-Net++, the skip-connections are modified by using nested and dense connections [10]. Accordingly, an improved U-Net++ is used for segmentation. In the proposed approach, the multi-scale technique will be improved by feature pyramids, which are built upon image pyramids and will form a fundamental solution for utilizing features in different scales. The detailed design of the proposed improved vessel extraction network is shown in Fig. 3. To leverage the pyramid features of the hierarchy decoder in U-Net++, we resize the feature maps extracted from different layers and integrate them. to generate the final feature map. By using pyramid features, the multi-scale problems would be significantly resolved. ![Fig. 3.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/05/27/2020.05.26.20103440/F3.medium.gif) [Fig. 3.](http://medrxiv.org/content/early/2020/05/27/2020.05.26.20103440/F3) Fig. 3. Architecture of multi-output FP-U-Net++. Both vascular tree segmentation task and major vessels segmentation task are performed by applying the designed FP-U-Net++ network. For vascular tree segmentation, the input of the network is the raw fluoroscopic angiograms and the output is the binary mask of the whole vascular tree. For major vessel segmentation, the input is the masked fluoroscopic angiograms by vascular tree binary mask and the output is the binary mask of LAD or LCX arteries. We separately train the FP-U-Net++ to generate the LAD and LCX binary masks. To optimize the parameters, we adopt the dice coefficient function as the loss function. ### 2.2 Vessel Graph Generation The centerline of vascular tree is generated by using erosion and dilatation operations. The extracted centerline is thus a representation of the vascular tree which contains three types of the nodes [12]: the degree-one nodes are the end of vessels and the degree-two nodes are connecting point; if the degree is greater or equal to 3, the node can be a joint point. We iterate each pixel in centerline then the vessel nodes are extracted by finding the joint points (degree greater or equal to 3) and the end points (degree equals to 1). To find the links between the nodes, we add each pair of points into the graph if they are connected adjacently. Then we will remove the degree-two points and corresponding edges from the centerline to generate the final undirect graph. Each vessel segment is represented by a link between two nodes in the generated graph. Semantic segmentation will label the vessels by determining the type of each graph edge and assigning the vessel type to the vessel segment based on the vessel pixels between any of the two adjacent nodes [11], the topology information, and the pixels within the vessel segments. ### 2.3 Vessel Segments Labelling In section 2.1, we have extracted the two major vessels and label them as LAD and LCX. For other three classes of vessels, we will label them based on the generated graph. The basic idea is to classify the segments into different classes of vessels. After that, we will assign the arteries with different class labels to achieve semantic segmentation. As demonstrated in section 2.2, each vessel segment is represented by a link between two nodes. When we classify a specific segment, we use the features extracted from this segment and its corresponding edge. For feature extraction from the edge, topology information will be used: the degree of the nodes connected by this edge will be used as the feature. We also extract position and pixel features from the segment (See table 1). By using the extracted features of each vessel segment, we will use an off-the-shell machine learning classifier to perform vessel segment classification and a grid search will be used to find the best classifier (see section 3). By classifying the vessel segments into different classes, the semantic segmentation results are generated, and each segment is labeled using the label corresponding to that class. View this table: [Table 1.](http://medrxiv.org/content/early/2020/05/27/2020.05.26.20103440/T1) Table 1. List of features measured for each vessel segment. ## 3 Experimental Results ### Dataset This retrospective study enrolled 73 patients who received fluoroscopy angiography between September 2010 and November 2017. Fluoroscopy angiography was performed using a Toshiba Infinix angiography system and acquired at 15 frames/sec and with a tube voltage of 80kV. Each image has a size of 512×512 with a pixel size of 0.342mm. The dataset consists of 172 fluoroscopy angiograms with 69 LAO images and 103 RAO images and all the images derived from LCA images. The vessel contours were manually drawn by well-trained operators, confirmed by an experienced interventional cardiologist and then provided to this study as the ground truth. For each angiogram, we annotated LAD, LCX, D1 and OM1 arteries, which are important for clinical analysis. The study was approved by the Oshsner Medical Center Institutional Review Board. ### Results on Vascular Tree and Major Vessels Segmentation We implemented our deep learning network with Keras, and we trained our designed FP-U-Net++ neural network on LAO and RAO dataset separately on a workstation with a Titan V GPU. On each subset, we randomly selected 80% samples for training and the rest for testing. We trained our model with 1000 iterations using the RMSprop optimizer with an initial learning rate to 0.001. For the backbone of FP-U-Net++, we adopted the Inception ResnetV2 [13] with the pretrained weight on ImageNet dataset and fine-tuned the parameters during the training period. To evaluate the model performance on vascular tree and major vessels segmentation, we adopted the Dice coefficient, specificity (SP) and sensitivity (SN). The model performance is shown in Table 2. View this table: [Table 2.](http://medrxiv.org/content/early/2020/05/27/2020.05.26.20103440/T2) Table 2. Performance evaluation on vascular tree and major vessel segmentation. ### Results on Vessel Graph Generation After generating vessel binary masks and extracting major vessels, we extracted the centerlines and generated a graph for each fluoroscopic angiogram. In Fig. 4, the joint points and end points are annotated by red star and green plus respectively. ![Fig. 4.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/05/27/2020.05.26.20103440/F4.medium.gif) [Fig. 4.](http://medrxiv.org/content/early/2020/05/27/2020.05.26.20103440/F4) Fig. 4. Experimental results on LAO (top) and RAO (bottom) subsets of (a) original images, (b) vascular binary segmentation results, (c) generated vascular centerlines, (d) generated vessel graph, (e) semantic segmentation predictions, and (f) semantic segmentation ground truths. ### Results on Vessel Semantic Segmentation For each generated vascular graph, we separated each vessel segment based on the centerline and extracted the corresponding features listed in Table 1. For LAO subset, 1152 segments were generated; for RAO subset, 4094 segments were generated. We performed a grid search on the classifiers for vessel segment classification, including support vector machine, gaussian process, stochastic gradient descent, random forest and ensemble learning with several base classifiers. According to experimental results, random forest classifier achieved the best performance on both LAO and RAO subsets. As demonstrated in section 2.3, the pixels within the vessel segments will be classified into one class; however, we will evaluate our classification model on pixel accuracy rather than segment accuracy, which is defined as: ![Formula][1] The pixel accuracy on fluoroscopic images is shown in Table 3. View this table: [Table 3.](http://medrxiv.org/content/early/2020/05/27/2020.05.26.20103440/T3) Table 3. Accuracy of pixel classification by using random forest. According to Table 3, our approach achieved an accuracy score at 0.6640 on LAO subset and 0.6149 on RAO subset. In Fig. 4, we visualized the results generated on each step for several examples on LAO and RAO subsets. ## 4 Conclusion We propose a new hybrid deep-learning and graph-based approach to extract individual coronary arteries in fluoroscopy angiograms by incorporating global position and topology information. To precisely extract the entire vascular tree and major vessels, we modify a U-Net++ network by concatenating hierarchical feature maps to generate the final prediction. By incorporating the clinical knowledge, we firstly extract the LAD and LCX vessel segments by FP-U-Net++ to further determine other vessel segments. By generating the centerline of vascular tree, we find the key points and convert the centerline into vessel graph. Each vessel segment is represented by the edge in the graph. Finally, we extract 56 handcraft features of the vessel segment and employ a random forest classifier to classify the vessel segments and further generate the semantic segmentation results. The proposed approach achieves an accuracy score at 66.4% and 61.49% on LAO and RAO subsets respectively. To our best knowledge, this is the first learning-based method to extract individual arteries in fluoroscopy angiograms and it has great promise for clinical uses. ## Data Availability The data is currently unavailable to the public. ## Acknowledgments This research was supported by a grant from the American Heart Association under award number 17AIREA33700016, a grant from Ochsner Hospital Foundation (PI: Weihua Zhou), and a new faculty startup grant from Michigan Technological University Institute of Computing and Cybersystems (PI: Weihua Zhou). This research was also supported in part by the National Institutes of Health under award numbers U19AG055373, R01GM109068, R01MH104680, R01MH107354 and by the National Science Foundation NSF under award number 1539067. * Received May 26, 2020. * Revision received May 26, 2020. * Accepted May 27, 2020. * © 2020, Posted by Cold Spring Harbor Laboratory The copyright holder for this pre-print is the author. All rights reserved. The material may not be redistributed, re-used or adapted without the author's permission. ## References 1. 1.Benjamin, E. J., Muntner, P., & Bittencourt, M. S. (2019). Heart disease and stroke statistics-2019 update: a report From the American Heart Association. Circulation, 139(10), e56-e528. doi:10.1161/CIR.0000000000000659 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1161/CIR.0000000000000659&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=30700139&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F05%2F27%2F2020.05.26.20103440.atom) 2. 2.Boden WE, O’Rourke RA, Teo KK, Hartigan PM, Maron DJ, Kostuk WJ, et al. COURAGE: optimal medical therapy with or without PCI for stable coronary disease. N Engl J Med. 2007;356: 1503–16. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJMoa070829&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=17387127&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F05%2F27%2F2020.05.26.20103440.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000245624400004&link_type=ISI) 3. 3.Tang, H., Tang, S. and Zhou, W., 2017. A review of image-guided approaches for cardiac resynchronisation therapy. Arrhythmia & electrophysiology review, 6(2), p.69. 4. 4.Nasr-Esfahani E, Karimi N, Jafari M, et al. Segmentation of vessels in angiograms using convolutional neural networks. Biomedical Signal Processing and Control. 2018;40:240–251. 5. 5.Nasr-Esfahani E, Samavi S, Karimi N, et al. Vessel extraction in X-ray angiograms using deep learning. 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). 2016. doi:10.1109/embc.2016.7590784. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1109/embc.2016.7590784&link_type=DOI) 6. 6.Petkov S, Radeva P, Carrillo X, Gatta C (2017) Automatic Segmentation and Tracking of Interventional Tools in Coronary Angiographies. J Case Repo Imag 1: 004. 7. 7.Yang, S., Kweon, J., Roh, J.H., Lee, J.H., Kang, H., Park, L.J., Kim, D.J., Yang, H., Hur, J., Kang, D.Y. and Lee, P.H., 2019. Deep learning segmentation of major vessels in X-ray coronary angiography. Scientific reports, 9(1), pp.1–11. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/s41598-019-43087-9&link_type=DOI) 8. 8.Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation. Lecture Notes in Computer Science Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015. 2015:234–241. doi:10.1007/978-3-319-24574-4_28. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/978-3-319-24574-4_28&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=WOS:00036596&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F05%2F27%2F2020.05.26.20103440.atom) 9. 9.Deng, J., Dong, W., Socher, R., Li, L.J., Li, K. and Fei-Fei, L., 2009, June. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition (pp. 248–255). IEEE. 10. 10.Zhou Z, Siddiquee MMR, Tajbakhsh N, Liang J. Unet++: A nested u-net architecture for medical image segmentation. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. 2018: 3–11. 11. 11.Dashtbozorg, B., Mendonça, A.M. and Campilho, A., 2013. An automatic graph-based approach for artery/vein classification in retinal images. IEEE Transactions on Image Processing, 23(3), pp.1073–1083. 12. 12.Xie, J., Zhao, Y., Liu, Y., Su, P., Zhao, Y., Cheng, J., Zheng, Y. and Liu, J., 2019. Topology reconstruction of tree-like structure in images via structural similarity measure and dominant set clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 8505-8513). 13. 13.Szegedy, C., Ioffe, S., Vanhoucke, V. and Alemi, A.A., 2017, February. Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-first AAAI conference on artificial intelligence. [1]: /embed/graphic-7.gif