Development of a deep learning model for the histologic diagnosis of dysplasia in Barrett’s esophagus

      Background and Aims

      The risk of progression in Barrett’s esophagus (BE) increases with development of dysplasia. There is a critical need to improve the diagnosis of BE dysplasia, given substantial interobserver disagreement among expert pathologists and overdiagnosis of dysplasia by community pathologists. We developed a deep learning model to predict dysplasia grade on whole-slide imaging.


      We digitized nondysplastic BE (NDBE), low-grade dysplasia (LGD), and high-grade dysplasia (HGD) histology slides. Two expert pathologists confirmed all histology and digitally annotated areas of dysplasia. Training, validation, and test sets were created (by a random 70/20/10 split). We used an ensemble approach combining a “you only look once” model to identify regions of interest and histology class (NDBE, LGD, or HGD) followed by a ResNet101 model pretrained on ImageNet applied to the regions of interest. Diagnostic performance was determined for the whole slide.


      We included slides from 542 patients (164 NDBE, 226 LGD, and 152 HGD) yielding 8596 bounding boxes in the training set, 1946 bounding boxes in the validation set, and 840 boxes in the test set. When the ensemble model was used, sensitivity and specificity for LGD was 81.3% and 100%, respectively, and >90% for NDBE and HGD. The overall positive predictive value and sensitivity metric (calculated as F1 score) was .91 for NDBE, .90 for LGD, and 1.0 for HGD.


      We successfully trained and validated a deep learning model to accurately identify dysplasia on whole-slide images. This model can potentially help improve the histologic diagnosis of BE dysplasia and the appropriate application of endoscopic therapy.


      BE (Barrett’s esophagus), EAC (esophageal adenocarcinoma), EET (endoscopic eradication therapy), NDBE (nondysplastic Barrett’s esophagus), HGD (high-grade dysplasia), LGD (low-grade dysplasia), WSI (whole-slide image), YOLO (you only look once)
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Gastrointestinal Endoscopy
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Siegel R.L.
        • Miller K.D.
        • Jemal A.
        Cancer statistics, 2020.
        CA Cancer J Clin. 2020; 70: 7-30
        • Rastogi A.
        • Puli S.
        • El-Serag H.B.
        • et al.
        Incidence of esophageal adenocarcinoma in patients with Barrett's esophagus and high-grade dysplasia: a meta-analysis.
        Gastrointest Endosc. 2008; 67: 394-398
        • Shaheen N.J.
        • Falk G.W.
        • Iyer P.G.
        • et al.
        ACG clinical guideline: diagnosis and management of Barrett's esophagus.
        Am J Gastroenterol. 2016; 111 (quiz 51): 30-50
        • Qumseya B.
        • Sultan S.
        • Bain P.
        • et al.
        ASGE guideline on screening and surveillance of Barrett's esophagus.
        Gastrointest Endosc. 2019; 90: 335-359
        • Odze R.D.
        Diagnosis and grading of dysplasia in Barrett's oesophagus.
        J Clin Pathol. 2006; 59: 1029-1038
        • Curvers W.L.
        • ten Kate F.J.
        • Krishnadath K.K.
        • et al.
        Low-grade dysplasia in Barrett's esophagus: overdiagnosed and underestimated.
        Am J Gastroenterol. 2010; 105: 1523-1530
      1. Huang G, Liu Z, Maaten LVD, et al. Densely connected convolutional networks. Presented at the 2017 IEEE Conference on Computer Vision and Pattern Recognition, July 21-26, 2017. Available at: Accessed July 21, 2022.

        • Anuse A.
        • Vyas V.
        A novel training algorithm for convolutional neural network.
        Complex Intell Syst. 2016; 2: 221-234
        • Aoki T.
        • Yamada A.
        • Aoyama K.
        • et al.
        Automatic detection of erosions and ulcerations in wireless capsule endoscopy images based on a deep convolutional neural network.
        Gastrointest Endosc. 2019; 89: 357-363
        • Marya N.B.
        • Powers P.D.
        • Chari S.T.
        • et al.
        Utilisation of artificial intelligence for the development of an EUS-convolutional neural network model trained to enhance the diagnosis of autoimmune pancreatitis.
        Gut. 2021; 70: 1335-1344
        • Tomita N.
        • Abdollahi B.
        • Wei J.
        • et al.
        Attention-based deep neural networks for detection of cancerous and precancerous esophagus tissue on histopathological slides.
        JAMA Netw Open. 2019; 2e1914645
        • Beuque M.
        • Martin-Lorenzo M.
        • Balluff B.
        • et al.
        Machine learning for grading and prognosis of esophageal dysplasia using mass spectrometry and histological imaging.
        Comput Biol Med. 2021; 138104918
        • Hansen L.K.
        • Salamon P.
        Neural network ensembles.
        IEEE Trans Pattern Anal Machine Intell. 1990; 12: 993-1001
        • Goldblum J.R.
        Current issues in Barrett's esophagus and Barrett's-related dysplasia.
        Mod Pathol. 2015; 28: S1-S6
        • Naini B.V.
        • Souza R.F.
        • Odze R.D.
        Barrett's esophagus: a comprehensive and contemporary review for pathologists.
        Am J Surg Pathol. 2016; 40: e45-e66
        • Pedregosa F.
        • Varoquaux G.
        • Gramfort A.
        • et al.
        Scikit-learn: machine learning in Python.
        J Machine Learn Res. 2011; 12: 2825-2830
        • Ganaie M.A.
        • Hu M.
        • Tanveer M.
        • et al.
        Ensemble deep learning: a review.
        ArXiv. 2021; (abs/2104.02395)
      2. Girshick R, Donahue J, Darrell T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition: IEEE Computer Society; 2014. p. 580-7.

        • Redmon J.
        • Divvala S.
        • Girshick R.
        • et al.
        You only look once: unified, real-time object detection.
        ArXiv. 2015; 150602640
        • Szegedy C.
        • Liu W.
        • Jia Y.
        • et al.
        Going deeper with convolutions.
        ArXiv. 2014; : 1409-4842
        • Krizhevsky A.
        • Sutskever I.
        • Hinton G.E.
        ImageNet classification with deep convolutional neural networks.
        in: Proceedings of the 25th International Conference on Neural Information Processing Systems. 1. Curran Associates Inc, Lake Tahoe, NV2012: 1097-1105
        • Lecun Y.
        • Bottou L.
        • Bengio Y.
        • et al.
        Gradient-based learning applied to document recognition.
        Proc IEEE. 1998; 86: 2278-2324
        • Montgomery E.
        • Bronner M.P.
        • Goldblum J.R.
        • et al.
        Reproducibility of the diagnosis of dysplasia in Barrett esophagus: a reaffirmation.
        Hum Pathol. 2001; 32: 368-378
        • Vennalaganti P.
        • Kanakadandi V.
        • Goldblum J.R.
        • et al.
        Discordance among pathologists in the United States and Europe in diagnosis of low-grade dysplasia for patients with Barrett's esophagus.
        Gastroenterology. 2017; 152: 564-570
        • Alikhan M.
        • Rex D.
        • Khan A.
        • et al.
        Variable pathologic interpretation of columnar lined esophagus by general pathologists in community practice.
        Gastrointest Endosc. 1999; 50: 23-26
        • Krishnamoorthi R.
        • Lewis J.T.
        • Krishna M.
        • et al.
        Predictors of progression in Barrett's esophagus with low-grade dysplasia: results from a multicenter prospective BE registry.
        Am J Gastroenterol. 2017; 112: 867-873
        • Duits L.C.
        • van der Wel M.J.
        • Cotton C.C.
        • et al.
        Patients with Barrett’s esophagus and confirmed persistent low-grade dysplasia are at increased risk for progression to neoplasia.
        Gastroenterology. 2017; 152: 993-1001
        • Singh S.
        • Manickam P.
        • Amin A.V.
        • et al.
        Incidence of esophageal adenocarcinoma in Barrett's esophagus with low-grade dysplasia: a systematic review and meta-analysis.
        Gastrointest Endosc. 2014; 79: 897-909.e4
        • Krishnamoorthi R.
        • Hargraves I.
        • Gopalakrishnan N.
        • et al.
        Development and pilot testing of decision aid for shared decision making in Barrett's esophagus with low-grade dysplasia.
        J Clin Gastroenterol. 2021; 55: 36-42


        • Mmdetection
        OpenMMLab detection toolbox and benchmark.
        (Available at:) (Accessed December 21, 2021)
        • Chen K.
        • Wang J.
        • Pang J.
        • et al.
        Open mmlab detection toolbox and benchmark.
        (Available at:) (Accessed July 21, 2022)
        • Liu X.
        • Li M.
        • Hao F.
        • et al.
        GLO-YOLO: a dynamic glomerular detecting and slicing model in whole slide images.
        in: Proceedings of the 2020 Conference on Artificial Intelligence and Healthcare. Association for Computing Machinery, Taiyuan, China2020: 229-233
        • Swiderska-Chadaj Z.
        • Pinckaers H.
        • van Rijthoven M.
        • et al.
        Learning to detect lymphocytes in immunohistochemistry with deep learning.
        Med Image Anal. 2019; 58101547
        • Lin T.-Y.
        • Maire M.
        • Belongie S.
        • et al.
        Microsoft COCO: common objects in context.
        in: Computer Vision—ECCV 2014. Springer International Publishing, Cham, Switzerland2014
        • He K.
        • Zhang X.
        • Ren S.
        • et al.
        Deep residual learning for image recognition.
        ArXiv. 2015; 151203385
      1. Deng J, Dong W, Socher R, et al. ImageNet: a large-scale hierarchical image database. Available at: Accessed July 21, 2022.

        • Howard J.
        • Gugger S.
        Fastai. A layered API for deep learning.
        Information. 2020; 11
        • Huang G.
        • Liu Z.
        • van der Maaten L.
        • et al.
        Densely connected convolutional networks.
        ArXiv. 2016; 160806993

      Linked Article

      • Artificial intelligence for dysplasia grading in Barrett’s esophagus: hematoxylin and eosin is here to stay
        Gastrointestinal EndoscopyVol. 96Issue 6
        • Preview
          Artificial intelligence (AI) is defined by the Oxford English Dictionary as “the theory and development of computer systems able to perform tasks that normally require human intelligence, such as visual perception, speech recognition, decision-making, and translation between languages.” Although still in its infancy, AI has already globally transformed several aspects of 21st century technology, influencing aviation, advertising, law enforcement, and warfare.1-3 It is only natural that AI will also transform medicine and improve patient care.
        • Full-Text
        • PDF