Automatic detection of various abnormalities in capsule endoscopy videos by a deep learning-based system: a multicenter study

      Background and Aims

      A deep convolutional neural network (CNN) system could be a high-level screening tool for capsule endoscopy (CE) reading but has not been established for targeting various abnormalities. We aimed to develop a CNN-based system and compare it with the existing QuickView mode in terms of their ability to detect various abnormalities.


      We trained a CNN system using 66,028 CE images (44,684 images of abnormalities and 21,344 normal images). The detection rate of the CNN for various abnormalities was assessed per patient, using an independent test set of 379 consecutive small-bowel CE videos from 3 institutions. Mucosal breaks, angioectasia, protruding lesions, and blood content were present in 94, 29, 81, and 23 patients, respectively. The detection capability of the CNN was compared with that of QuickView mode.


      The CNN picked up 1,135,104 images (22.5%) from the 5,050,226 test images, and thus, the sampling rate of QuickView mode was set to 23% in this study. In total, the detection rate of the CNN for abnormalities per patient was significantly higher than that of QuickView mode (99% vs 89%, P < .001). The detection rates of the CNN for mucosal breaks, angioectasia, protruding lesions, and blood content were 100% (94 of 94), 97% (28 of 29), 99% (80 of 81), and 100% (23 of 23), respectively, and those of QuickView mode were 91%, 97%, 80%, and 96%, respectively.


      We developed and tested a CNN-based detection system for various abnormalities using multicenter CE videos. This system could serve as an alternative high-level screening tool to QuickView mode.


      CE (capsule endoscopy), CNN (convolutional neural network), SSD (Single Shot MultiBox Detector), CEST (capsule endoscopy structured terminology)
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Gastrointestinal Endoscopy
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Iddan G.
        • Meron G.
        • Glukhovsky A.
        • et al.
        Wireless capsule endoscopy.
        Nature. 2000; 405: 417
        • Korman L.Y.
        • Delvaux M.
        • Gay G.
        • et al.
        Capsule endoscopy structured terminology (CEST): proposal of a standardized and structured terminology for reporting capsule endoscopy procedures.
        Endoscopy. 2005; 37: 951-959
        • Kyriakos N.
        • Karagiannis S.
        • Galanis P.
        • et al.
        Evaluation of four time-saving methods of reading capsule endoscopy videos.
        Eur J Gastroenterol Hepatol. 2012; 24: 1276-1280
        • Koulaouzidis A.
        • Smirnidis A.
        • Douglas S.
        • et al.
        QuickView in small-bowel capsule endoscopy is useful in certain clinical settings, but QuickView with blue mode is of no additional benefit.
        Eur J Gastroenterol Hepatol. 2012; 24: 1099-1104
        • Shiotani A.
        • Honda K.
        • Kawakami M.
        • et al.
        Analysis of small-bowel capsule endoscopy reading by using quickview mode: training assistants for reading may produce a high diagnostic yield and save time for physicians.
        J Clin Gastroenterol. 2012; 46: e92-e95
        • Hosoe N.
        • Rey J.F.
        • Imaeda H.
        • et al.
        Evaluations of capsule endoscopy software in reducing the reading time and the rate of false negatives by inexperienced endoscopists.
        Clin Res Hepatol Gastroenterol. 2012; 36: 66-71
        • Yuan Y.
        • Meng M.Q.
        Deep learning for polyp recognition in wireless capsule endoscopy images.
        Med Phys. 2017; 44: 1379-1389
        • Xiao J.
        • Meng M.Q.
        A deep convolutional neural network for bleeding detection in wireless capsule endoscopy images.
        Conf Proc IEEE Eng Med Biol Soc. 2016; 2016: 639-642
        • Zhou T.
        • Han G.
        • Li B.N.
        • et al.
        Quantitative analysis of patients with celiac disease by video capsule endoscopy: a deep learning method.
        Comput Biol Med. 2017; 85: 1-6
        • Alaskar H.
        • Hussain A.
        • Al-Aseem N.
        • et al.
        Application of convolutional neural networks for automated ulcer detection in wireless capsule endoscopy images.
        Sensors (Basel). 2019; 19: 1265
        • Aoki T.
        • Yamada A.
        • Aoyama K.
        • et al.
        Automatic detection of erosions and ulcerations in wireless capsule endoscopy images based on a deep convolutional neural network.
        Gastrointest Endosc. 2019; 89: 357-363.e2
        • Leenhardt R.
        • Vasseur P.
        • Li C.
        • et al.
        A neural network algorithm for detection of GI angiectasia during small-bowel capsule endoscopy.
        Gastrointest Endosc. 2019; 89: 189-194
        • Tsuboi A.
        • Oka S.
        • Aoyama K.
        • et al.
        Artificial intelligence using a convolutional neural network for automatic detection of small-bowel angioectasia in capsule endoscopy images.
        Dig Endosc. 2020; 32: 382-390
        • Ding Z.
        • Shi H.
        • Zhang H.
        • et al.
        Gastroenterologist-level identification of small bowel diseases and normal variants by capsule endoscopy using a deep-learning model.
        Gastroenterology. 2019; 157: 1044-1054.e5
        • Aoki T.
        • Yamada A.
        • Aoyama K.
        • et al.
        Clinical usefulness of a deep learning-based system as the first screening on small-bowel capsule endoscopy reading.
        Dig Endosc. 2020; 32: 585-591
        • Klang E.
        • Barash Y.
        • Yehuda Margalit R.
        • et al.
        Deep learning algorithms for automated detection of Crohn's disease ulcers by video capsule endoscopy.
        Gastrointest Endosc. 2020; 91: 606-613.e2
        • Aoki T.
        • Yamada A.
        • Kato Y.
        • et al.
        Automatic detection of blood content in capsule endoscopy images based on a deep convolutional neural network.
        J Gastroenterol Hepatol. 2019 Nov 23;
        • Saito H.
        • Aoki T.
        • Aoyama K.
        • et al.
        Automatic detection and classification of protruding lesions in wireless capsule endoscopy images based on a deep convolutional neural network.
        Gastrointest Endosc. 2020; 92: 944-951
        • Aoki T.
        • Yamada A.
        • Hirata Y.
        • et al.
        Etiology and long-term rebleeding of endoscopic ulcerative lesions in the small bowel in patients with obscure gastrointestinal bleeding: a multicenter cohort study.
        J Gastroenterol Hepatol. 2018; 33: 1327-1334
        • Graham D.Y.
        • Opekun A.R.
        • Willingham F.F.
        • et al.
        Visible small-intestinal mucosal injury in chronic NSAID users.
        Clin Gastroenterol Hepatol. 2005; 3: 55-59
        • Yano T.
        • Yamamoto H.
        • Sunada K.
        • et al.
        Endoscopic classification of vascular lesions of the small intestine (with videos).
        Gastrointest Endosc. 2008; 67: 169-172