Application of artificial intelligence using a novel EUS-based convolutional neural network model to identify and distinguish benign and malignant hepatic masses

Published:August 27, 2020DOI:

      Background and Aims

      Detection and characterization of focal liver lesions (FLLs) is key for optimizing treatment for patients who may have a primary hepatic cancer or metastatic disease to the liver. This is the first study to develop an EUS-based convolutional neural network (CNN) model for the purpose of identifying and classifying FLLs.


      A prospective EUS database comprising cases of FLLs visualized and sampled via EUS was reviewed. Relevant still images and videos of liver parenchyma and FLLs were extracted. Patient data were then randomly distributed for the purpose of CNN model training and testing. Once a final model was created, occlusion heatmap analysis was performed to assess the ability of the EUS-CNN model to autonomously identify FLLs. The performance of the EUS-CNN for differentiating benign and malignant FLLs was also analyzed.


      A total of 210,685 unique EUS images from 256 patients were used to train, validate, and test the CNN model. Occlusion heatmap analyses demonstrated that the EUS-CNN model was successful in autonomously locating FLLs in 92.0% of EUS video assets. When evaluating any random still image extracted from videos or physician-captured images, the CNN model was 90% sensitive and 71% specific (area under the receiver operating characteristic [AUROC], 0.861) for classifying malignant FLLs. When evaluating full-length video assets, the EUS-CNN model was 100% sensitive and 80% specific (AUROC, 0.904) for classifying malignant FLLs.


      This study demonstrated the capability of an EUS-CNN model to autonomously identify FLLs and to accurately classify them as either malignant or benign lesions.

      Graphical abstract


      AUROC (area under the receiver operating characteristic), CNN (convolutional neural network), FLL (focal liver lesion), FNB (fine-needle biopsy), PCS (physician-captured still), TUS (transabdominal ultrasound)
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Gastrointestinal Endoscopy
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Schwartz L.H.
        • Gandras E.J.
        • Colangelo S.M.
        • et al.
        Prevalence and importance of small hepatic lesions found at CT in patients with cancer.
        Radiology. 1999; 210: 71-74
        • Khalil H.I.
        • Patterson S.A.
        • Panicek D.M.
        Hepatic lesions deemed too small to characterize at CT: prevalence and importance in women with breast cancer.
        Radiology. 2005; 235: 872-878
        • Bhalla M.
        • Aldakkak M.
        • Kulkarni N.M.
        • et al.
        Characterizing indeterminate liver lesions in patients with localized pancreatic cancer at the time of diagnosis.
        Abdom Radiol (NY). 2018; 43: 351-363
        • Mohamed E.
        • Adiamah A.
        • Dunn W.K.
        • et al.
        Outcome of indeterminate liver lesions on computed tomography in patients with colorectal cancer.
        Ann R Coll Surg Engl. 2018; 100: 382-387
        • Fujii-Lau L.L.
        • Abu Dayyeh B.K.
        • Bruno M.J.
        • et al.
        EUS-derived criteria for distinguishing benign from malignant metastatic solid hepatic masses.
        Gastrointest Endosc. 2015; 81 (e1181-7): 1188-1196
        • Esteva A.
        • Kuprel B.
        • Novoa R.A.
        • et al.
        Dermatologist-level classification of skin cancer with deep neural networks.
        Nature. 2017; 542: 115-118
        • Jadoon M.M.
        • Zhang Q.
        • Haq I.U.
        • et al.
        Three-class mammogram classification based on descriptive CNN features.
        Biomed Res Int. 2017; 2017: 3640901
        • Zachariah R.
        • Samarasena J.
        • Luba D.
        • et al.
        Prediction of polyp pathology using convolutional neural networks achieves "resect and discard" thresholds.
        Am J Gastroenterol. 2020; 115: 138-144
        • Gargeya R.
        • Leng T.
        Automated identification of diabetic retinopathy using deep learning.
        Ophthalmology. 2017; 124: 962-969
        • Norton I.D.
        • Zheng Y.
        • Wiersema M.S.
        • et al.
        Neural network analysis of EUS images to differentiate between pancreatic malignancy and pancreatitis.
        Gastrointest Endosc. 2001; 54: 625-629
        • Saftoiu A.
        • Vilmann P.
        • Gorunescu F.
        • et al.
        Neural network analysis of dynamic sequences of EUS elastography used for the differential diagnosis of chronic pancreatitis and pancreatic cancer.
        Gastrointest Endosc. 2008; 68: 1086-1094
        • Schmauch B.
        • Herent P.
        • Jehanno P.
        • et al.
        Diagnosis of focal liver lesions from ultrasound using deep learning.
        Diagn Interv Imaging. 2019; 100: 227-233
        • He K.
        • Zhang X.
        • Ren S.
        • et al.
        Identity mappings in deep residual networks.
        arXiv. 2016; 160305027
        • Pedregosa F.
        • Varoquaux G.
        • Gramfort A.
        • et al.
        Scikit-learn: machine learning in Python.
        J Mach Learn. 2011; 12: 2825-2830
        • Zhu Y.
        • Wang Q.C.
        • Xu M.D.
        • et al.
        Application of convolutional neural network in the diagnosis of the invasion depth of gastric cancer based on conventional endoscopy.
        Gastrointest Endosc. 2019; 89: 806-15 e801
        • Leenhardt R.
        • Vasseur P.
        • Li C.
        • et al.
        A neural network algorithm for detection of GI angiectasia during small-bowel capsule endoscopy.
        Gastrointest Endosc. 2019; 89: 189-194
        • Urban G.
        • Tripathi P.
        • Alkayali T.
        • et al.
        deep learning localizes and identifies polyps in real time with 96% accuracy in screening colonoscopy.
        Gastroenterology. 2018; 155: 1069-1078.e1068
        • Zhu M.
        • Xu C.
        • Yu J.
        • et al.
        Differentiation of pancreatic cancer and chronic pancreatitis using computer-aided diagnosis of endoscopic ultrasound (EUS) images: a diagnostic test.
        PLoS One. 2013; 8e63820
        • Das A.
        • Nguyen C.C.
        • Li F.
        • et al.
        Digital image analysis of EUS images accurately differentiates pancreatic cancer from chronic pancreatitis and normal tissue.
        Gastrointest Endosc. 2008; 67: 861-867
        • Hassan T.M.
        • Elmogy M.
        • Sallam E.
        Diagnosis of focal liver diseases based on deep learning technique for ultrasound images.
        Arab J Sci Eng. 2017; 42: 3127-3140
        • Sujana H.
        • Swarnamani S.
        • Suresh S.
        Application of artificial neural networks for the classification of liver lesions by image texture parameters.
        Ultrasound Med Biol. 1996; 22: 1177-1181
        • Yoshida H.
        • Casalino D.D.
        • Keserci B.
        • et al.
        Wavelet-packet-based texture analysis for differentiation between benign and malignant liver tumours in ultrasound images.
        Phys Med Biol. 2003; 48: 3735-3753
        • Balasubramanian D.
        • Srinivasan P.
        • Gurupatham R.
        Automatic classification of focal lesions in ultrasound liver images using principal component analysis and neural networks.
        in: 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Lyon. 2007: 2134-2137
        • Mittal D.
        • Kumar V.
        • Saxena S.C.
        • et al.
        Neural network based focal liver lesion diagnosis using ultrasound images.
        Comput Med Imaging Graph. 2011; 35: 315-323
        • Hwang Y.N.
        • Lee J.H.
        • Kim G.Y.
        • et al.
        Classification of focal liver lesions on ultrasound images by extracting hybrid textural features and using an artificial neural network.
        Biomed Mater Eng. 2015; 26: S1599-S1611
        • Kermany D.S.
        • Goldbaum M.
        • Cai W.
        • et al.
        Identifying medical diagnoses and treatable diseases by image-based deep learning.
        Cell. 2018; 172: 1122-31 e1129
        • Madani A.
        • Arnaout R.
        • Mofrad M.
        • et al.
        Fast and accurate view classification of echocardiograms using deep learning.
        NPJ Digital Med. 2018; 1: 6
        • Campos S.
        • Poley J.W.
        • van Driel L.
        • et al.
        The role of EUS in diagnosis and treatment of liver disorders.
        Endosc Int Open. 2019; 7: E1262-E1275

      Linked Article

      • Artificial intelligence applications in EUS: the journey of a thousand miles begins with a single step
        Gastrointestinal EndoscopyVol. 93Issue 5
        • Preview
          Artificial intelligence (AI) analysis of medical images is a burgeoning field of active research and industry investment. The ability for AI technology to analyze millions of pixels of imaging data for thousands of patients and to extract information beyond what is possible with the human eye and brain holds great promise.1 AI technologies are already seeing expanding use in radiology. It has been said that “AI won’t replace radiologists, but radiologists who use AI will replace radiologists who don’t.”2
        • Full-Text
        • PDF