Classification for invasion depth of esophageal squamous cell carcinoma using a deep neural network compared with experienced endoscopists

      Background and Aims

      Cancer invasion depth is a critical factor affecting the choice of treatment in patients with superficial squamous cell carcinoma (SCC). However, the diagnosis of invasion depth is currently subjective and liable to interobserver variability.


      We developed a deep learning-based artificial intelligence (AI) system based on Single Shot MultiBox Detector architecture for the assessment of superficial esophageal SCC. We obtained endoscopic images from patients with superficial esophageal SCC at our facility between December 2005 and December 2016.


      After excluding poor-quality images, 8660 non-magnified endoscopic (non-ME) and 5678 ME images from 804 superficial esophageal SCCs with pathologic proof of cancer invasion depth were used as the training dataset, and 405 non-ME images and 509 ME images from 155 patients were selected for the validation set. Our system showed a sensitivity of 90.1%, specificity of 95.8%, positive predictive value of 99.2%, negative predictive value of 63.9%, and an accuracy of 91.0% for differentiating pathologic mucosal and submucosal microinvasive (SM1) cancers from submucosal deep invasive (SM2/3) cancers. Cancer invasion depth was diagnosed by 16 experienced endoscopists using the same validation set, with an overall sensitivity of 89.8%, specificity of 88.3%, positive predictive value of 97.9%, negative predictive value of 65.5%, and an accuracy of 89.6%.


      This newly developed AI system showed favorable performance for diagnosing invasion depth in patients with superficial esophageal SCC, with comparable performance to experienced endoscopists.


      AI (artificial intelligence), BLI (blue-laser imaging), CNN (convolutional neural network), EP (epithelium), ER (endoscopic resection), ESD (endoscopic submucosal dissection), LPM (lamina propria mucosa), M (mucosal), ME (magnified endoscopy), MM (muscularis mucosa), NBI (narrow-band imaging), p (pathologic), SCC (squamous cell carcinoma), SM1 (submucosal microinvasive), SM2/3 (submucosal deep invasive), VGG (visual geometry group), WLI (white-light imaging)
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Gastrointestinal Endoscopy
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Ferlay J.
        • Soerjomataram I.
        • Dikshit R.
        • et al.
        Cancer incidence and mortality worldwide: sources, methods and major patterns in GLOBOCAN 2012.
        Int J Cancer. 2015; 136: E359-E386
        • Kodama M.
        • Kakegawa T.
        Treatment of superficial cancer of the esophagus: a summary of responses to a questionnaire on superficial cancer of the esophagus in Japan.
        Surgery. 1998; 123: 432-439
        • Yamashina T.
        • Ishihara R.
        • Nagai K.
        • et al.
        Long-term outcome and metastatic risk after endoscopic resection of superficial esophageal squamous cell carcinoma.
        Am J Gastroenterol. 2013; 108: 544-551
        • Katada C.
        • Muto M.
        • Momma K.
        • et al.
        Clinical outcome after endoscopic mucosal resection for esophageal squamous cell carcinoma invading the muscularis mucosae--a multicenter retrospective cohort study.
        Endoscopy. 2007; 39: 779-783
        • Shimizu Y.
        • Tsukagoshi H.
        • Fujita M.
        • et al.
        Long-term outcome after endoscopic mucosal resection in patients with esophageal squamous cell carcinoma invading the muscularis mucosae or deeper.
        Gastrointest Endosc. 2002; 56: 387-390
        • Igaki H.
        • Kato H.
        • Tachimori Y.
        • et al.
        Clinicopathologic characteristics and survival of patients with clinical Stage I squamous cell carcinomas of the thoracic esophagus treated with three-field lymph node dissection.
        Eur J Cardiothorac Surg. 2001; 20: 1089-1094
        • Yamamoto S.
        • Ishihara R.
        • Motoori M.
        • et al.
        Comparison between definitive chemoradiotherapy and esophagectomy in patients with clinical stage I esophageal squamous cell carcinoma.
        Am J Gastroenterol. 2011; 106: 1048-1054
        • Tachimori Y.
        • Ozawa S.
        • Numasaki H.
        • et al.
        Comprehensive Registry of Esophageal Cancer in Japan, 2011.
        Esophagus. 2018; 15: 127-152
        • Birkmeyer J.D.
        • Siewers A.E.
        • Finlayson E.V.
        • et al.
        Hospital volume and surgical mortality in the United States.
        N Engl J Med. 2002; 346: 1128-1137
        • Chang A.C.
        • Ji H.
        • Birkmeyer N.J.
        • et al.
        Outcomes after transhiatal and transthoracic esophagectomy for cancer.
        Ann Thorac Surg. 2008; 85: 424-429
        • Kitagawa Y.
        • Uno T.
        • Oyama T.
        • et al.
        Esophageal cancer practice guidelines 2017 edited by the Japan Esophageal Society.
        Esophagus. 2018; 16: 1-24
        • Pimentel-Nunes P.
        • Dinis-Ribeiro M.
        • Ponchon T.
        • et al.
        Endoscopic submucosal dissection: European Society of Gastrointestinal Endoscopy (ESGE) Guideline.
        Endoscopy. 2015; 47: 829-854
        • Akutsu Y.
        • Uesato M.
        • Shuto K.
        • et al.
        The overall prevalence of metastasis in T1 esophageal squamous cell carcinoma: a retrospective analysis of 295 patients.
        Ann Surg. 2013; 257: 1032-1038
        • Yoshida T.
        • Inoue H.
        • Usui S.
        • et al.
        Narrow-band imaging system with magnifying endoscopy for superficial esophageal lesions.
        Gastrointest Endosc. 2004; 59: 288-295
        • Arima M.
        • Arima H.
        • Tada M.
        Evaluation of microvascular pattern classification of superficial esophageal lesions by magnifying endoscopy [Japanese].
        Stomach Intest. 2007; 42: 589-595
        • Ebi M.
        • Shimura T.
        • Yamada T.
        • et al.
        Multicenter, prospective trial of white-light imaging alone versus white-light imaging followed by magnifying endoscopy with narrow-band imaging for the real-time imaging and diagnosis of invasion depth in superficial esophageal squamous cell carcinoma.
        Gastrointest Endosc. 2015; 81: 1355-1361.e2
        • Lee M.W.
        • Kim G.H.
        • I H
        • et al.
        Predicting the invasion depth of esophageal squamous cell carcinoma: comparison of endoscopic ultrasonography and magnifying endoscopy.
        Scand J Gastroenterol. 2014; 49: 853-861
        • Pouw R.E.
        • Heldoorn N.
        • Herrero L.A.
        • et al.
        Do we still need EUS in the workup of patients with early esophageal neoplasia? A retrospective analysis of 131 cases.
        Gastrointest Endosc. 2011; 73: 662-668
        • Thosani N.
        • Singh H.
        • Kapadia A.
        • et al.
        Diagnostic accuracy of EUS in differentiating mucosal versus submucosal invasion of superficial esophageal cancers: a systematic review and meta-analysis.
        Gastrointest Endosc. 2012; 75: 242-253
        • Silver D.
        • Huang A.
        • Maddison C.J.
        • et al.
        Mastering the game of Go with deep neural networks and tree search.
        Nature. 2016; 529: 484-489
        • Russakovsky O.
        • Deng J.
        • Su H.
        • et al.
        Imagenet large scale visual recognition challenge.
        Int J Comput Vis. 2015; 115: 211-252
        • Esteva A.
        • Kuprel B.
        • Novoa R.A.
        • et al.
        Dermatologist-level classification of skin cancer with deep neural networks.
        Nature. 2017; 542: 115-118
        • Gulshan V.
        • Peng L.
        • Coram M.
        • et al.
        Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs.
        JAMA. 2016; 316: 2402-2410
        • Byrne M.F.
        • Chapados N.
        • Soudan F.
        • et al.
        Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model.
        Gut. 2019; 68: 94-100
        • Shichijo S.
        • Nomura S.
        • Aoyama K.
        • et al.
        Application of convolutional neural networks in the diagnosis of Helicobacter pylori infection based on endoscopic images.
        EBioMedicine. 2017; 25: 106-111
        • Hirasawa T.
        • Aoyama K.
        • Tanimoto T.
        • et al.
        Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images.
        Gastric Cancer. 2018; 21: 653-660
        • Horie Y.
        • Yoshio T.
        • Aoyama K.
        • et al.
        Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
        Gastrointest Endosc. 2019; 89: 25-32
        • Goodfellow I.
        • Bengio Y.
        • Courville A.
        • et al.
        Deep learning.
        MIT Press, Cambridge, MA2016
        • Litjens G.
        • Kooi T.
        • Bejnordi B.E.
        • et al.
        A survey on deep learning in medical image analysis.
        Med Image Anal. 2017; 42: 60-68
        • Landis J.R.
        • Koch G.G.
        The measurement of observer agreement for categorical data.
        Biometrics. 1977; 33: 159-174
        • Oyama T.
        • Inoue H.
        • Arima M.
        • et al.
        Prediction of the invasion depth of superficial squamous cell carcinoma based on microvessel morphology: magnifying endoscopic classification of the Japan Esophageal Society.
        Esophagus. 2017; 14: 105-112
        • Ishihara R.
        • Matsuura N.
        • Hanaoka N.
        • et al.
        Endoscopic imaging modalities for diagnosing invasion depth of superficial esophageal squamous cell carcinoma: a systematic review and meta-analysis.
        BMC Gastroenterol. 2017; 17: 24
        • Yu T.
        • Geng J.
        • Song W.
        • et al.
        Diagnostic accuracy of magnifying endoscopy with narrow band imaging and its diagnostic value for invasion depth staging in esophageal squamous cell carcinoma: a systematic review and meta-analysis.
        Biomed Res Int. 2018; 2018: 8591387
        • Shelhamer E.
        • Long J.
        • Darrell T.
        Fully convolutional networks for semantic segmentation.
        IEEE Trans Pattern Anal Mach Intell. 2017; 39: 640-651