Artificial intelligence−enhanced white-light colonoscopy with attention guidance predicts colorectal cancer invasion depth

Published:April 10, 2021DOI:

      Background and Aims

      Endoscopic submucosal dissection (ESD) and EMR are applied in treating superficial colorectal neoplasms but are contraindicated by deeply invasive colorectal cancer (CRC). The invasion depth of neoplasms can be examined by an automated artificial intelligence (AI) system to determine the applicability of ESD and EMR.


      A deep convolutional neural network with a tumor localization branch to guide invasion depth classification was constructed on the GoogLeNet architecture. The model was trained using 7734 nonmagnified white-light colonoscopy (WLC) images supplemented by image augmentation from 657 lesions labeled with histopathologic analysis of invasion depth. An independent testing dataset consisting of 1634 WLC images from 156 lesions was used to validate the model.


      For predicting noninvasive and superficially invasive neoplasms, the model achieved an overall accuracy of 91.1% (95% confidence interval [CI], 89.6%-92.4%), with 91.2% sensitivity (95% CI, 88.8%-93.3%) and 91.0% specificity (95% CI, 89.0%-92.7%) at an optimal cutoff of .41 and the area under the receiver operating characteristic (AUROC) curve of .970 (95% CI, .962-.978). Inclusion of the advanced CRC data significantly increased the sensitivity in differentiating superficial neoplasms from deeply invasive early CRC to 65.3% (95% CI, 61.9%-68.8%) with an AUROC curve of .729 (95% CI, .699-.759), similar to experienced endoscopists (.691; 95% CI, .624-.758).


      We have developed an AI-enhanced attention-guided WLC system that differentiates noninvasive or superficially submucosal invasive neoplasms from deeply invasive CRC with high accuracy, sensitivity, and specificity.

      Graphical abstract


      AEWL (artificial intelligence–enhanced attention-guided white light colonoscopy), AI (artificial intelligence), AUROC (area under the receiver operating characteristic curve), CAD (computer-aided diagnosis), CNN (convolutional neural network), CRC (colorectal cancer), ER (endoscopic resection), ESD (endoscopic submucosal dissection), IEE (image-enhanced endoscopy), ME (magnifying endoscopy), NPV (negative predictive value), PPV (positive predictive value), ROC (receiver operating characteristic), WLC (white-light colonoscopy)
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Gastrointestinal Endoscopy
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Kobayashi N.
        • Saito Y.
        • Sano Y.
        • et al.
        Determining the treatment strategy for colorectal neoplastic lesions: endoscopic assessment or the non-lifting sign for diagnosing invasion depth?.
        Endoscopy. 2007; 39: 701-705
        • Tanaka S.
        • Kashida H.
        • Saito Y.
        • et al.
        JGES guidelines for colorectal endoscopic submucosal dissection/endoscopic mucosal resection.
        Dig Endosc. 2015; 27: 417-434
        • Pimentel-Nunes P.
        • Dinis-Ribeiro M.
        • Ponchon T.
        • et al.
        Endoscopic submucosal dissection: European Society of Gastrointestinal Endoscopy (ESGE) guideline.
        Endoscopy. 2015; 47: 829-854
        • Niimi K.
        • Fujishiro M.
        • Kodashima S.
        • et al.
        Long-term outcomes of endoscopic submucosal dissection for colorectal epithelial neoplasms.
        Endoscopy. 2010; 42: 723-729
        • Saito Y.
        • Uraoka T.
        • Yamaguchi Y.
        • et al.
        A prospective, multicenter study of 1111 colorectal endoscopic submucosal dissections (with video).
        Gastrointest Endosc. 2010; 72: 1217-1225
        • Puli S.R.
        • Kakugawa Y.
        • Gotoda T.
        • et al.
        Meta-analysis and systematic review of colorectal endoscopic mucosal resection.
        World J Gastroenterol. 2009; 15: 4273-4277
        • Tateishi Y.
        • Nakanishi Y.
        • Taniguchi H.
        • et al.
        Pathological prognostic factors predicting lymph node metastasis in submucosal invasive (T1) colorectal carcinoma.
        Mod Pathol. 2010; 23: 1068-1072
        • Yoshida N.
        • Naito Y.
        • Yagi N.
        • et al.
        Importance of histological evaluation in endoscopic resection of early colorectal cancer.
        World J Gastrointest Pathophysiol. 2012; 3: 51-59
        • Hashiguchi Y.
        • Muro K.
        • Saito Y.
        • et al.
        Japanese Society for Cancer of the Colon and Rectum (JSCCR) guidelines 2019 for the treatment of colorectal cancer.
        Int J Clin Oncol. 2020; 25: 1-42
        • Hassan C.
        • Pickhardt P.J.
        • Rex D.K.
        A resect and discard strategy would improve cost-effectiveness of colorectal cancer screening.
        Clin Gastroenterol Hepatol. 2010; 8: 865-869
        • Li M.
        • Ali S.M.
        • Umm-a-OmarahGilani S.
        • et al.
        Kudo's pit pattern classification for colorectal neoplasms: a meta-analysis.
        World J Gastroenterol. 2014; 20: 12649-12656
        • Ikehara H.
        • Saito Y.
        • Matsuda T.
        • et al.
        Diagnosis of depth of invasion for early colorectal cancer using magnifying colonoscopy.
        J Gastroenterol Hepatol. 2010; 25: 905-912
        • Tanaka S.
        • Hayashi N.
        • Oka S.
        • et al.
        Endoscopic assessment of colorectal cancer with superficial or deep submucosal invasion using magnifying colonoscopy.
        Clin Endosc. 2013; 46: 138-146
        • Kakushima N.
        Endoscopic diagnosis for the depth of early colorectal cancer.
        J Gastroenterol Hepatol. 2010; 25: 850-851
        • Matsuda T.
        • Fujii T.
        • Saito Y.
        • et al.
        Efficacy of the invasive/non-invasive pattern by magnifying chromoendoscopy to estimate the depth of invasion of early colorectal neoplasms.
        Am J Gastroenterol. 2008; 103: 2700-2706
        • Sano Y.
        • Hirata D.
        • Saito Y.
        Japan NBI Expert Team classification: Narrow-band imaging magnifying endoscopic classification of colorectal tumors.
        Dig Endosc. 2018; 30: 543-545
        • Yoshida N.
        • Yagi N.
        • Inada Y.
        • et al.
        Ability of a novel blue laser imaging system for the diagnosis of colorectal polyps.
        Dig Endosc. 2014; 26: 250-258
        • Sano Y.
        • Tanaka S.
        • Kudo S.E.
        • et al.
        Narrow-band imaging (NBI) magnifying endoscopic classification of colorectal tumors proposed by the Japan NBI Expert Team.
        Dig Endosc. 2016; 28: 526-533
        • Zhang S.
        • Bamakan S.M.H.
        • Qu Q.
        • et al.
        Learning for personalized medicine: a comprehensive review from a deep learning perspective.
        IEEE Rev Biomed Eng. 2019; 12: 194-208
        • Wang P.
        • Berzin T.M.
        • Glissen Brown J.R.
        • et al.
        Real-time automatic detection system increases colonoscopic polyp and adenoma detection rates: a prospective randomised controlled study.
        Gut. 2019; 68: 1813-1819
        • Byrne M.F.
        • Chapados N.
        • Soudan F.
        • et al.
        Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model.
        Gut. 2019; 68: 94-100
        • Lui T.K.L.
        • Wong K.K.Y.
        • Mak L.L.Y.
        • et al.
        Endoscopic prediction of deeply submucosal invasive carcinoma with use of artificial intelligence.
        Endosc Int Open. 2019; 7: E514-E520
        • Takeda K.
        • Kudo S.E.
        • Mori Y.
        • et al.
        Accuracy of diagnosing invasive colorectal cancer using computer-aided endocytoscopy.
        Endoscopy. 2017; 49: 798-802
        • Tokunaga M.
        • Matsumura T.
        • Nankinzan R.
        • et al.
        Computer-aided diagnosis system using only white-light endoscopy for the prediction of invasion depth in colorectal cancer.
        Gastrointest Endosc. 2021; 93: 647-653
        • Zhou B.
        • Khosla A.
        • Lapedriza A.
        • et al.
        Learning deep features for discriminative localization.
        (Available at:)
        • Russakovsky O.
        • Deng J.
        • Su H.
        • et al.
        ImageNet large scale visual recognition challenge.
        (Available at:)
        • Szegedy C.
        • Liu W.
        • Jia Y.
        • et al.
        Going deeper with convolutions.
        (Available at:)
        • Liu L.
        • Tan R.T.
        • You S.
        Loss guided activation for action recognition in still images.
        (Available at:)
        • Ribeiro E.
        • Uhl A.
        • Wimmer G.
        • et al.
        Exploring deep learning and transfer learning for colonic polyp classification.
        Comput Math Methods Med. 2016; 20166584725
        • Wong S.C.
        • Gatt A.
        • Stamatescu V.
        • et al.
        Understanding data augmentation for classification: when to warp?.
        (Available at:)
        • Zhu Y.
        • Wang Q.C.
        • Xu M.D.
        • et al.
        Application of convolutional neural network in the diagnosis of the invasion depth of gastric cancer based on conventional endoscopy.
        Gastrointest Endosc. 2019; 89: 806-815
        • Hewett D.G.
        • Kaltenbach T.
        • Sano Y.
        • et al.
        Validation of a simple classification system for endoscopic diagnosis of small colorectal polyps using narrow-band imaging.
        Gastroenterology. 2012; 143: 599-607
        • Mukae M.
        • Kobayashi K.
        • Sada M.
        • et al.
        Diagnostic performance of EUS for evaluating the invasion depth of early colorectal cancers.
        Gastrointest Endosc. 2015; 81: 682-690
        • Yu Y.
        • Wang J.
        • Ng C.W.
        • et al.
        Deep learning enables automated scoring of liver fibrosis stages.
        Sci Rep. 2018; 8: 16016
        • Ito N.
        • Kawahira H.
        • Nakashima H.
        • et al.
        Endoscopic diagnostic support system for cT1b colorectal cancer using deep learning.
        Oncology. 2019; 96: 44-50
        • Liu W.
        • Anguelov D.
        • Erhan D.
        • et al.
        SSD: single shot multibox detector.
        (Available at:)
        • Pogorelov K.
        • Randel K.R.
        • Griwodz C.
        • et al.
        Kvasir: a multi-class image dataset for computer aided gastrointestinal disease detection.
        (Proceedings of the 8th ACM on Multimedia Systems Conference. Taipei, Taiwan: Association for Computing Machinery)2017: 164-169
        • Hayashi N.
        • Tanaka S.
        • Hewett D.G.
        • et al.
        Endoscopic prediction of deep submucosal invasive carcinoma: validation of the narrow-band imaging international colorectal endoscopic (NICE) classification.
        Gastrointest Endosc. 2013; 78: 625-632

      Linked Article

      • Challenge to the “impossible”
        Gastrointestinal EndoscopyVol. 94Issue 3
        • Preview
          A Dutch research team engaged in the national colorectal cancer screening program published an “amazing” result in 2020.1 As many as 60% of T1 (submucosally invasive) colorectal cancers detected during the program were misdiagnosed as adenomas by the on-site endoscopists and thus could be susceptible to inappropriate treatment intervention. Is this surprisingly low sensitivity for cancer recognition (40% in this case) reality? We would say yes, although many retrospective studies assessing advanced endoscopic modalities have suggested >90% sensitivities in predicting T1 cancers.
        • Full-Text
        • PDF