Deep learning algorithms for automated detection of Crohn’s disease ulcers by video capsule endoscopy

Published:November 16, 2019DOI:

      Background and Aims

      The aim of our study was to develop and evaluate a deep learning algorithm for the automated detection of small-bowel ulcers in Crohn’s disease (CD) on capsule endoscopy (CE) images of individual patients.


      We retrospectively collected CE images of known CD patients and control subjects. Each image was labeled by an expert gastroenterologist as either normal mucosa or containing mucosal ulcers. A convolutional neural network was trained to classify images into either normal mucosa or mucosal ulcers. First, we trained the network on 5-fold randomly split images (each fold with 80% training images and 20% images testing). We then conducted 10 experiments in which images from n – 1 patients were used to train a network and images from a different individual patient were used to test the network. Results of the networks were compared for randomly split images and for individual patients. Area under the curves (AUCs) and accuracies were computed for each individual network.


      Overall, our dataset included 17,640 CE images from 49 patients: 7391 images with mucosal ulcers and 10,249 images of normal mucosa. For randomly split images results were excellent, with AUCs of .99 and accuracies ranging from 95.4% to 96.7%. For individual patient-level experiments, the AUCs were also excellent (.94-.99).


      Deep learning technology provides accurate and fast automated detection of mucosal ulcers on CE images. Individual patient-level analysis provided high and consistent diagnostic accuracy with shortened reading time; in the future, deep learning algorithms may augment and facilitate CE reading.

      Graphical abstract


      AI (artificial intelligence), AUC (area under the curve), CD (Crohn’s disease), CE (capsule endoscopy), CNN (convolutional neural network), t-SNE (t-distributed stochastic neighbor embedding algorithm)
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Gastrointestinal Endoscopy
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Pennazio M.
        • Spada C.
        • Eliakim R.
        • et al.
        Small-bowel capsule endoscopy and device-assisted enteroscopy for diagnosis and treatment of small-bowel disorders: European Society of Gastrointestinal Endoscopy (ESGE) clinical guideline.
        Endoscopy. 2015; 47: 352-386
        • Kopylov U.
        • Koulaouzidis A.
        • Klang E.
        • et al.
        Monitoring of small bowel Crohn's disease.
        Exp Rev Gastroenterol Hepatol. 2017; 11: 1047-1058
        • Eliakim R.
        Video capsule endoscopy of the small bowel.
        Curr Opin Gastroenterol. 2010; 26: 129-133
        • Waterman M.
        • Eliakim R.
        Capsule enteroscopy of the small intestine.
        Abdom Imag. 2009; 34: 452-458
        • Eliakim R.
        Video capsule endoscopy of the small bowel.
        Curr Opin Gastroenterol. 2008; 24: 159-163
        • Melmed G.Y.
        • Dubinsky M.C.
        • Rubin D.T.
        • et al.
        Utility of video capsule endoscopy for longitudinal monitoring of Crohn’s disease activity in the small bowel: a prospective study.
        Gastrointest Endosc. 2018; 88: 947-955
        • Sturm A.
        • Maaser C.
        • Calabrese E.
        • et al.
        ECCO-ESGAR guideline for diagnostic assessment in IBD part 2: IBD scores and general principles and technical aspects.
        J Crohn's Colitis. 2018; 13: 273-284
        • Maaser C.
        • Sturm A.
        • Vavricka S.R.
        • et al.
        ECCO-ESGAR guideline for diagnostic assessment in IBD part 1: initial diagnosis, monitoring of known IBD, detection of complications.
        J Crohn's Colitis. 2018; 13: 144-164
        • Kopylov U.
        • Yablecovitch D.
        • Lahat A.
        • et al.
        Detection of small bowel mucosal healing and deep remission in patients with known small bowel Crohn’s disease using biomarkers, capsule endoscopy, and imaging.
        Am J Gastroenterol. 2015; 110: 1316
        • Yung D.E.
        • Har-Noy O.
        • Tham Y.S.
        • et al.
        Capsule endoscopy, magnetic resonance enterography, and small bowel ultrasound for evaluation of postoperative recurrence in crohn’s disease: systematic review and meta-analysis.
        Inflamm Bowel Dis. 2017; 24: 93-100
        • Kopylov U.
        • Yung D.E.
        • Engel T.
        • et al.
        Diagnostic yield of capsule endoscopy versus magnetic resonance enterography and small bowel contrast ultrasound in the evaluation of small bowel Crohn’s disease: systematic review and meta-analysis.
        Dig Liver Dis. 2017; 49: 854-863
        • Gralnek I.
        • Defranchis R.
        • Seidman E.
        • et al.
        Development of a capsule endoscopy scoring index for small bowel mucosal inflammatory change.
        Aliment Pharmacol Therap. 2008; 27: 146-154
        • De Vos M.
        • Cuvelier C.
        • Mielants H.
        • et al.
        Ileocolonoscopy in seronegative spondylarthropathy.
        Gastroenterology. 1989; 96: 339-344
        • Niv Y.
        • Ilani S.
        • Levi Z.
        • et al.
        Validation of the Capsule Endoscopy Crohn’s Disease Activity Index (CECDAI or Niv score): a multicenter prospective study.
        Endoscopy. 2012; 44: 21-26
        • Yablecovitch D.
        • Lahat A.
        • Neuman S.
        • et al.
        The Lewis score or the Capsule Endoscopy Crohn’s Disease Activity Index: Which one is better for the assessment of small bowel inflammation in established Crohn’s disease?.
        Therap Adv Gastroenterol. 2018; 11 (1756283X17747780)
        • Cotter J.
        • de Castro F.D.
        • Magalhães J.
        • et al.
        Validation of the Lewis score for the evaluation of small-bowel Crohn’s disease activity.
        Endoscopy. 2015; 47: 330-335
        • Klang E.
        Deep learning and medical imaging.
        J Thorac Dis. 2018; 10: 1325-1328
        • Soffer S.
        • Ben-Cohen A.
        • Shimon O.
        • et al.
        Convolutional neural networks for radiologic images: a radiologist’s guide.
        Radiology. 2019; 290: 590-606
        • Vieira P.M.
        • Silva C.P.
        • Costa D.
        • et al.
        Automatic segmentation and detection of small bowel angioectasias in WCE images.
        Ann Biomed Eng. 2019; 47: 1446-1462
        • Blanes-Vidal V.
        • Baatrup G.
        • Nadimi E.S.
        Addressing priority challenges in the detection and assessment of colorectal polyps from capsule endoscopy and colonoscopy in colorectal cancer screening using machine learning.
        Acta Oncol. 2019; 58: S29-S36
        • Hwang Y.
        • Park J.
        • Lim Y.J.
        • et al.
        Application of artificial intelligence in capsule endoscopy: Where are we now?.
        Clin Endosc. 2018; 51: 547
        • Zhou T.
        • Han G.
        • Li B.N.
        • et al.
        Quantitative analysis of patients with celiac disease by video capsule endoscopy: a deep learning method.
        Comput Biol Med. 2017; 85: 1-6
        • Yuan Y.
        • Meng M.Q.H.
        Deep learning for polyp recognition in wireless capsule endoscopy images.
        Med Phys. 2017; 44: 1379-1389
        • Jia X.
        • Meng M.Q.
        Gastrointestinal bleeding detection in wireless capsule endoscopy images using handcrafted and CNN features.
        Conf Proc IEEE Eng Med Biol Soc. 2017; 2017: 3154-3157
        • Min J.K.
        • Kwak M.S.
        • Cha J.M.
        Overview of deep learning in gastrointestinal endoscopy.
        Gut Liver. 2019; 13: 388
        • Hosoe N.
        • Takabayashi K.
        • Ogata H.
        • et al.
        Capsule endoscopy for small-intestinal disorders: current status.
        Dig Endosc. 2019; 31: 498-507
        • Jolliffe I.T.
        • Cadima J.
        Principal component analysis: a review and recent developments.
        Phil Trans R Soc A Math Phys Eng Sci. 2016; 374 (20150202)
        • Arora S.
        • Hu W.
        • Kothari P.K.
        An analysis of the t-SNE algorithm for data visualization. arXiv e-prints.
        (Available at:) (Accessed March 5, 2018)
        • Chollet F.
        Xception: deep learning with depthwise separable convolutions. arXiv e-prints.
        (Available at:) (Accessed October 7, 2016)
        • Russakovsky O.
        • Deng J.
        • Su H.
        • et al.
        Imagenet large scale visual recognition challenge.
        Int J Comput Vis. 2015; 115: 211-252
        • Cao C.
        • Liu F.
        • Tan H.
        • et al.
        Deep learning and its applications in biomedicine.
        Genom Proteom Bioinform. 2018; 16: 17-32
        • Leenhardt R.
        • Vasseur P.
        • Li C.
        • et al.
        A neural network algorithm for detection of GI angiectasia during small-bowel capsule endoscopy.
        Gastrointest Endosc. 2019; 89: 189-194
        • Iakovidis D.K.
        • Georgakopoulos S.V.
        • Vasilakakis M.
        • et al.
        Detecting and locating gastrointestinal anomalies using deep learning and iterative cluster unification.
        IEEE Trans Med Imag. 2018; 37: 2196-2210
        • Kopylov U.
        • Starr M.
        • Watts C.
        • et al.
        Detection of Crohn disease in patients with spondyloarthropathy: the SpACE capsule study.
        J Rheumatol. 2018; 45: 498-505
        • Kopylov U.
        • Ben-Horin S.
        • Seidman E.G.
        • et al.
        Video capsule endoscopy of the small bowel for monitoring of Crohn's disease.
        Inflamm Bowel Dis. 2015; 21: 2726-2735
        • Shah S.C.
        • Colombel J.F.
        • Sands B.E.
        • et al.
        Systematic review with meta-analysis: mucosal healing is associated with improved long-term outcomes in Crohn's disease.
        Aliment Pharmacol Ther. 2016; 43: 317-333
        • Greener T.
        • Klang E.
        • Yablecovitch D.
        • et al.
        The impact of magnetic resonance enterography and capsule endoscopy on the re-classification of disease in patients with known Crohn’s disease: a prospective Israeli IBD Research Nucleus (IIRN) study.
        J Crohn's Colitis. 2016; 10: 525-531
        • Flamant M.
        • Trang C.
        • Maillard O.
        • et al.
        The prevalence and outcome of jejunal lesions visualized by small bowel capsule endoscopy in Crohn’s disease.
        Inflamm Bowel Dis. 2013; 19: 1390-1396
        • Carvalho P.B.
        • Rosa B.
        • Cotter J.
        Mucosal healing in Crohn’s disease—Are we reaching as far as possible with capsule endoscopy?.
        J Crohn's Colitis. 2014; 8: 1566-1567
        • Ben-Horin S.
        • Lahat A.
        • Amitai M.M.
        • et al.
        Assessment of small bowel mucosal healing by video capsule endoscopy for the prediction of short-term and long-term risk of Crohn's disease flare: a prospective cohort study.
        Lancet Gastroenterol Hepatol. 2019; 4: 519-528
        • Kopylov U.
        • Klang E.
        • Yablecovitch D.
        • et al.
        Magnetic resonance enterography versus capsule endoscopy activity indices for quantification of small bowel inflammation in Crohn’s disease.
        Therap Adv Gastroenterol. 2016; 9: 655-663
        • Tham Y.S.
        • Yung D.E.
        • Fay S.
        • et al.
        Fecal calprotectin for detection of postoperative endoscopic recurrence in Crohn’s disease: systematic review and meta-analysis.
        Therap Adv Gastroenterol. 2018; 11 (1756284818785571)
        • Koulaouzidis A.
        • Sipponen T.
        • Nemeth A.
        • et al.
        Association between fecal calprotectin levels and small-bowel inflammation score in capsule endoscopy: a multicenter retrospective study.
        Dig Dis Sci. 2016; 61: 2033-2040
        • Khanna R.
        • Nelson S.A.
        • Feagan B.G.
        • et al.
        Endoscopic scoring indices for evaluation of disease activity in Crohn's disease.
        Cochrane Database System Rev. 2016; : Cd010642
        • Eliakim R.
        • Spada C.
        • Lapidus A.
        • et al.
        Evaluation of a new pan-enteric video capsule endoscopy system in patients with suspected or established inflammatory bowel disease—feasibility study.
        Endosc Int Open. 2018; 6: E1235-E1246
        • Aoki T.
        • Yamada A.
        • Aoyama K.
        • et al.
        Automatic detection of erosions and ulcerations in wireless capsule endoscopy images based on a deep convolutional neural network.
        Gastrointest Endosc. 2019; 89: 357-363
        • Sidhu R.
        • Sanders D.
        • Morris A.
        • et al.
        Guidelines on small bowel enteroscopy and capsule endoscopy in adults.
        Gut. 2008; 57: 125-136
        • Maaser C.
        • Sturm A.
        • Vavricka S.R.
        • et al.
        ECCO-ESGAR guideline for diagnostic assessment in IBD Part 1: Initial diagnosis, monitoring of known IBD, detection of complications.
        J Crohn's Colitis. 2019; 13: 144-164
        • Dionisio P.M.
        • Gurudu S.R.
        • Leighton J.A.
        • et al.
        Capsule endoscopy has a significantly higher diagnostic yield in patients with suspected and established small-bowel Crohn's disease: a meta-analysis.
        Am J Gastroenterol. 2010; 105 (quiz 9): 1240-1248
        • Price A.B.
        Pathology of drug-associated gastrointestinal disease.
        Br J Clin Pharmacol. 2003; 56: 477-482