Advertisement

A primer on artificial intelligence and its application to endoscopy

      Artificial intelligence (AI) has emerged as a powerful and exciting new technology poised to impact many aspects of health care. In endoscopy, AI is now being used to detect and characterize benign and malignant GI lesions and assess malignant lesion depth of invasion. It will undoubtedly also find use in capsule endoscopy and inflammatory bowel disease. Herein, we provide the general endoscopist with a brief overview of AI and its emerging uses in our field. We also touch on the challenges of incorporating AI into clinical practice, such as workflow integration, data storage, and data privacy.

      Graphical abstract

      Abbreviations:

      AI (artificial intelligence), CE (capsule endoscopy), IBD (inflammatory bowel disease), NBI (narrow-band imaging), PIVI (Preservation and Incorporation of Valuable Endoscopic Innovations), WL (white light)
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Gastrointestinal Endoscopy
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Engelbrecht A.P.
        Computational intelligence: an introduction.
        John Wiley & Sons, Chichester2007
        • Neumann H.
        • Bisschops R.
        Artificial intelligence and the future of endoscopy.
        Dig Endosc. 2019; 31: 389-390
        • Turing A.M.
        The essential turing.
        Oxford University Press, New York2004
        • LeCun Y.
        • Bengio Y.
        • Hinton G.
        Deep learning.
        Nature. 2015; 521: 436-444
        • Mitchell T.
        • Buchanan B.
        • DeJong G.
        • et al.
        Machine learning.
        Annu Rev Computer Sci. 1990; 4: 417-433
        • Michie D.
        • Spiegelhalter D.J.
        • Taylor C.
        Machine learning.
        Neural Stat Class. 1994; 13: 1-298
        • Sathya R.
        • Abraham A.
        Comparison of supervised and unsupervised learning algorithms for pattern classification.
        Int J Adv Res Artific Intell. 2013; 2: 34-38
        • Goodfellow I.
        • Bengio Y.
        • Courville A.
        Deep learning.
        MIT press, 2016
      1. Bergstra J.S. Bardenet R. Bengio Y. Algorithms for hyper-parameter optimization. Advances in neural information processing systems. 2011: 2546-2554
        • Lever J.
        • Krzywinski M.
        • Altman N.
        Points of significance: model selection and overfitting.
        Nature Publishing Group, 2016
      2. Bareinboim E. Pearl J. Controlling selection bias in causal inference. Artificial Intelligence and Statistics, 2012 Mar 21: 100-108
        • Byrne M.F.
        • Chapados N.
        • Soudan F.
        • et al.
        Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model.
        Gut. 2019; 68: 94-100
      3. Deng J. Dong W. Socher R. Imagenet: a large-scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition. 2009: 248-255
        • Han D.
        • Liu Q.
        • Fan W.
        A new image classification method using CNN transfer learning and web data augmentation.
        Exp Syst Appl. 2018; 95: 43-56
        • U.S. Food and Drug Administration
        Clinical Decision software: guidance for industry and Food and Drug Administration staff.
        2019 (Available at: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/clinical-decision-support-software. Accessed July 1, 2020)
        • Challen R.
        • Denny J.
        • Pitt M.
        • et al.
        Artificial intelligence, bias and clinical safety.
        BMJ Qual Saf. 2019; 28: 231-237
        • Abouelmehdi K.
        • Beni-Hessane A.
        • Khaloufi H.
        Big healthcare data: preserving security and privacy.
        J Big Data. 2018; 5: 1
        • Burke W.
        • Beskow L.M.
        • Trinidad S.B.
        • et al.
        Informed consent in translational genomics: insufficient without trustworthy governance.
        J Law Med Ethics. 2018; 46: 79-86
        • Tang A.
        • Tam R.
        • Cadrin-Chênevert A.
        • et al.
        Canadian Association of Radiologists white paper on artificial intelligence in radiology.
        Can Assoc Radiol J. 2018; 69: 120-135
        • Topol E.J.
        High-performance medicine: the convergence of human and artificial intelligence.
        Nat Med. 2019; 25: 44
        • Min J.K.
        • Kwak M.S.
        • Cha J.M.
        Overview of deep learning in gastrointestinal endoscopy.
        Gut Liver. 2019; 13: 388
        • Picardo S.
        • Ragunath K.
        Artificial intelligence in endoscopy: The guardian angel is around the corner.
        Gastrointest Endosc. 2020; 91: 340-341
        • Sharma P.
        • Hawes R.H.
        • Bansal A.
        • et al.
        Standard endoscopy with random biopsies versus narrow band imaging targeted biopsies in Barrett's oesophagus: a prospective, international, randomised controlled trial.
        Gut. 2013; 62: 15-21
        • Sharma P.
        • Savides T.J.
        • Canto M.I.
        • et al.
        The American Society for Gastrointestinal Endoscopy PIVI (preservation and incorporation of valuable endoscopic innovations) on imaging in Barrett's esophagus.
        Gastrointest Endosc. 2012; 76: 252-254
        • Fonollà R.
        • Scheeve T.
        • Struyvenberg M.R.
        • et al.
        Ensemble of deep convolutional neural networks for classification of early Barrett’s neoplasia using volumetric laser endomicroscopy.
        Appl Sci. 2019; 9: 2183
      4. Hong J, Park B-Y, Park H, editors. Convolutional neural network classifier for distinguishing Barrett's esophagus and neoplasia endomicroscopy images. 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society Conf Proc IEEE Eng Med Biol Soc 2017;2017:2892-5.

        • Ebigbo A.
        • Mendel R.
        • Probst A.
        • et al.
        Computer-aided diagnosis using deep learning in the evaluation of early oesophageal adenocarcinoma.
        Gut. 2019; 68: 1143-1145
        • Muto M.
        • Minashi K.
        • Yano T.
        • et al.
        Early detection of superficial squamous cell carcinoma in the head and neck region and esophagus by narrow band imaging: a multicenter randomized controlled trial.
        J Clin Oncol. 2010; 28: 1566
        • Shimizu Y.
        • Omori T.
        • Yokoyama A.
        • et al.
        Endoscopic diagnosis of early squamous neoplasia of the esophagus with iodine staining: high-grade intra-epithelial neoplasia turns pink within a few minutes.
        J Gastroenterol Hepatol. 2008; 23: 546-550
        • Horie Y.
        • Yoshio T.
        • Aoyama K.
        • et al.
        Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
        Gastrointest Endosc. 2019; 89: 25-32
        • Everson M.
        • Herrera L.
        • Li W.
        • et al.
        Artificial intelligence for the real-time classification of intrapapillary capillary loop patterns in the endoscopic diagnosis of early oesophageal squamous cell carcinoma: a proof-of-concept study.
        United Eur Gastroenterol J. 2019; 7: 297-306
        • Kumagai Y.
        • Takubo K.
        • Kawada K.
        • et al.
        Diagnosis using deep-learning artificial intelligence based on the endocytoscopic observation of the esophagus.
        Esophagus. 2019; 16: 180-187
        • Nakagawa K.
        • Ishihara R.
        • Aoyama K.
        • et al.
        Classification for invasion depth of esophageal squamous cell carcinoma using a deep neural network compared with experienced endoscopists.
        Gastrointest Endosc. 2019; 90: 407-414
        • Zhao Y.-Y.
        • Xue D.-X.
        • Wang Y.-L.
        • et al.
        Computer-assisted diagnosis of early esophageal squamous cell carcinoma using narrow-band imaging magnifying endoscopy.
        Endoscopy. 2019; 51: 333-341
        • Van Cutsem E.
        • Sagaert X.
        • Topal B.
        • et al.
        Gastric cancer.
        Lancet. 2016; 388: 2654-2664
        • Menon S.
        • Trudgill N.
        How commonly is upper gastrointestinal cancer missed at endoscopy? A meta-analysis.
        Endosc Int Open. 2014; 2: E46-E50
        • Hirasawa T.
        • Aoyama K.
        • Tanimoto T.
        • et al.
        Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images.
        Gastric Cancer. 2018; 21: 653-660
        • Zhu Y.
        • Wang Q.-C.
        • Xu M.-D.
        • et al.
        Application of convolutional neural network in the diagnosis of the invasion depth of gastric cancer based on conventional endoscopy.
        Gastrointest Endosc. 2019; 89: 806-815
        • Ishioka M.
        • Hirasawa T.
        • Tada T.
        Detecting gastric cancer from video images using convolutional neural networks.
        Dig Endosc. 2019; 31: e34-e35
        • Gado A.S.
        • Ebeid B.A.
        • Axon A.T.
        Quality assurance in gastrointestinal endoscopy: An Egyptian experience.
        Arab J Gastroenterol. 2016; 17: 153-158
        • Wu L.
        • Zhang J.
        • Zhou W.
        • et al.
        Randomised controlled trial of WISENSE, a real-time quality improving system for monitoring blind spots during esophagogastroduodenoscopy.
        Gut. 2019; 68: 2161-2169
        • Zhao S.
        • Wang S.
        • Pan P.
        • et al.
        Magnitude, risk factors, and factors associated with adenoma miss rate of tandem colonoscopy: a systematic review and meta-analysis.
        Gastroenterology. 2019; 156: 1661-1674
        • Misawa M.
        • Kudo S.-E.
        • Mori Y.
        • et al.
        Characterization of colorectal lesions using a computer-aided diagnostic system for narrow-band imaging endocytoscopy.
        Gastroenterology. 2016; 150: 1531-1532
        • Urban G.
        • Tripathi P.
        • Alkayali T.
        • et al.
        Deep learning localizes and identifies polyps in real time with 96% accuracy in screening colonoscopy.
        Gastroenterology. 2018; 155: 1069-1078
        • Wang P.
        • Berzin T.M.
        • Brown J.R.G.
        • et al.
        Real-time automatic detection system increases colonoscopic polyp and adenoma detection rates: a prospective randomised controlled study.
        Gut. 2019; 68: 1813-1819
        • Dayyeh B.K.A.
        • Thosani N.
        • Konda V.
        • et al.
        ASGE Technology Committee systematic review and meta-analysis assessing the ASGE PIVI thresholds for adopting real-time endoscopic assessment of the histology of diminutive colorectal polyps.
        Gastrointest Endosc. 2015; 81: 502.e1-502.e16
        • Bisschops R.
        • East J.E.
        • Hassan C.
        • et al.
        Advanced imaging for detection and differentiation of colorectal neoplasia: European Society of Gastrointestinal Endoscopy (ESGE) guideline–update 2019.
        Endoscopy. 2019; 51: 1155-1179
        • East J.E.
        • Rees C.J.
        Making optical biopsy a clinical reality in colonoscopy.
        Lancet Gastroenterol Hepatol. 2018; 3: 10-12
        • Chen P.-J.
        • Lin M.-C.
        • Lai M.-J.
        • et al.
        Accurate classification of diminutive colorectal polyps using computer-aided analysis.
        Gastroenterology. 2018; 154: 568-575
        • Mori Y.
        • Kudo S.-E.
        • Misawa M.
        • et al.
        Real-time use of artificial intelligence in identification of diminutive polyps during colonoscopy: a prospective study.
        Ann Intern Med. 2018; 169: 357-366
        • Vinsard D.G.
        • Mori Y.
        • Misawa M.
        • et al.
        Quality assurance of computer-aided detection and diagnosis in colonoscopy.
        Gastrointest Endosc. 2019; 90: 55-63
        • Mori Y.
        • Kudo S.-E.
        • Misawa M.
        • Mori K.
        Simultaneous detection and characterization of diminutive polyps with the use of artificial intelligence during colonoscopy.
        VideoGIE. 2019; 4: 7-10
        • Guizard N.
        • Ghalehjegh S.H.
        • Henkel M.
        • et al.
        Artificial intelligence for real-time multiple polyp detection with identification, tracking, and optical biopsy during colonoscopy.
        Gastroenterology. 2019; 156: S-48-49
        • De Lange T.
        • Larsen S.
        • Aabakken L.
        Inter-observer agreement in the assessment of endoscopic findings in ulcerative colitis.
        BMC Gastroenterol. 2004; 4: 9
        • Ozawa T.
        • Ishihara S.
        • Fujishiro M.
        • et al.
        Novel computer-assisted diagnosis system for endoscopic disease activity in patients with ulcerative colitis.
        Gastrointest Endosc. 2019; 89: 416-421
        • Stidham R.W.
        • Liu W.
        • Bishu S.
        • et al.
        Performance of a Deep learning model vs human reviewers in grading endoscopic disease severity of patients with ulcerative colitis.
        JAMA Netw Ope. 2019; 2 (e193963-e)
        • Maeda Y.
        • Kudo S.-E.
        • Mori Y.
        • et al.
        Fully automated diagnostic system with artificial intelligence using endocytoscopy to identify the presence of histologic inflammation associated with ulcerative colitis (with video).
        Gastrointest Endosc. 2019; 89: 408-415
        • Gottlieb K.
        • Travis S.
        • Feagan B.
        • et al.
        Central reading of endoscopy endpoints in inflammatory bowel disease trials.
        Inflamm Bowel Dis. 2015; 21: 2475-2482
        • Spada C.
        • Hassan C.
        • Galmiche J.-P.
        • et al.
        Colon capsule endoscopy: European Society of Gastrointestinal Endoscopy (ESGE) guideline.
        Endoscopy. 2012; 44: 527-536
        • Zou Y.
        • Li L.
        • Wang Y.
        • et al.
        Classifying digestive organs in wireless capsule endoscopy images based on deep convolutional neural network.
        2015 IEEE International Conference on Digital Signal Processing (DSP). 2015; : 1274-1278
        • Leenhardt R.
        • Vasseur P.
        • Li C.
        • et al.
        A neural network algorithm for detection of GI angiectasia during small-bowel capsule endoscopy.
        Gastrointest Endosc. 2019; 89: 189-194
        • Aoki T.
        • Yamada A.
        • Aoyama K.
        • et al.
        Automatic detection of erosions and ulcerations in wireless capsule endoscopy images based on a deep convolutional neural network.
        Gastrointest Endosc. 2019; 89: 357-363
        • He J.-Y.
        • Wu X.
        • Jiang Y.-G.
        • et al.
        Hookworm detection in wireless capsule endoscopy images with deep learning.
        IEEE Trans Image Process. 2018; 27: 2379-2392
        • Saito H.
        • Aoki T.
        • Aoyama K.
        • et al.
        Automatic detection and classification of protruding lesions in wireless capsule endoscopy images based on a deep convolutional neural network.
        Gastrointest Endosc. 2020; 92: 144-151
        • Yuan Y.
        • Meng M.Q.H.
        Deep learning for polyp recognition in wireless capsule endoscopy images.
        Med Phys. 2017; 44: 1379-1389
        • Byrne M.F.
        • Donnellan F.
        Artificial intelligence and capsule endoscopy: Is the truly “smart” capsule nearly here?.
        Gastrointest Endosc. 2019; 89: 195-197

      References

        • Fonollà R.
        • Scheeve T.
        • Struyvenberg M.R.
        • et al.
        Ensemble of deep convolutional neural networks for classification of early Barrett’s neoplasia using volumetric laser endomicroscopy.
        Appl Sci. 2019; 9: 2183
        • Hong J.
        • Park B.-Y.
        • Park H.
        Convolutional neural network classifier for distinguishing Barrett's esophagus and neoplasia endomicroscopy images.
        Conf Proc IEEE Eng Med Biol Soc. 2017; 2017: 2892-2895
        • Ebigbo A.
        • Mendel R.
        • Probst A.
        • et al.
        Computer-aided diagnosis using deep learning in the evaluation of early oesophageal adenocarcinoma.
        Gut. 2019; 68: 1143-1145
        • Horie Y.
        • Yoshio T.
        • Aoyama K.
        • et al.
        Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
        Gastrointest Endosc. 2019; 89: 25-32
        • Everson M.
        • Herrera L.
        • Li W.
        • et al.
        Artificial intelligence for the real-time classification of intrapapillary capillary loop patterns in the endoscopic diagnosis of early oesophageal squamous cell carcinoma: a proof-of-concept study.
        United European Gastroenterol J. 2019; 7: 297-306
        • Kumagai Y.
        • Takubo K.
        • Kawada K.
        • et al.
        Diagnosis using deep-learning artificial intelligence based on the endocytoscopic observation of the esophagus.
        Esophagus. 2019; 16: 180-187
        • Nakagawa K.
        • Ishihara R.
        • Aoyama K.
        • et al.
        Classification for invasion depth of esophageal squamous cell carcinoma using a deep neural network compared with experienced endoscopists.
        Gastrointest Endosc. 2019; 90: 407-414
        • Zhao Y.-Y.
        • Xue D.-X.
        • Wang Y.-L.
        • et al.
        Computer-assisted diagnosis of early esophageal squamous cell carcinoma using narrow-band imaging magnifying endoscopy.
        Endoscopy. 2019; 51: 333-341
        • Hirasawa T.
        • Aoyama K.
        • Tanimoto T.
        • et al.
        Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images.
        Gastric Cancer. 2018; 21: 653-660
        • Zhu Y.
        • Wang Q.-C.
        • Xu M.-D.
        • et al.
        Application of convolutional neural network in the diagnosis of the invasion depth of gastric cancer based on conventional endoscopy.
        Gastrointest Endosc. 2019; 89: 806-815.e1
        • Ishioka M.
        • Hirasawa T.
        • Tada T.
        Detecting gastric cancer from video images using convolutional neural networks.
        Dig Endosc. 2019; 31: e34-e35
        • Wu L.
        • Zhang J.
        • Zhou W.
        • et al.
        Randomised controlled trial of WISENSE, a real-time quality improving system for monitoring blind spots during esophagogastroduodenoscopy.
        Gut. Epub. 2019 Mar 11;
        • Urban G.
        • Tripathi P.
        • Alkayali T.
        • et al.
        Deep learning localizes and identifies polyps in real time with 96% accuracy in screening colonoscopy.
        Gastroenterology. 2018; 155: 1069-1078
        • Wang P.
        • Berzin T.M.
        • Brown J.R.G.
        • et al.
        Real-time automatic detection system increases colonoscopic polyp and adenoma detection rates: a prospective randomised controlled study.
        Gut. 2019; 68: 1813-1819
        • Misawa M.
        • Kudo S.-E.
        • Mori Y.
        • et al.
        Characterization of colorectal lesions using a computer-aided diagnostic system for narrow-band imaging endocytoscopy.
        Gastroenterology. 2016; 150: 1531-1532
        • Chen P.-J.
        • Lin M.-C.
        • Lai M.-J.
        • et al.
        Accurate classification of diminutive colorectal polyps using computer-aided analysis.
        Gastroenterology. 2018; 154: 568-575
        • Mori Y.
        • Kudo S.-E.
        • Misawa M.
        • et al.
        Real-time use of artificial intelligence in identification of diminutive polyps during colonoscopy: a prospective study.
        Ann Intern Med. 2018; 169: 357-366
        • Byrne M.F.
        • Chapados N.
        • Soudan F.
        • et al.
        Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model.
        Gut. 2019; 68: 94-100
        • Mori Y.
        • Kudo S.-E.
        • Misawa M.
        • et al.
        Simultaneous detection and characterization of diminutive polyps with the use of artificial intelligence during colonoscopy.
        VideoGIE. 2019; 4: 7-10
        • Guizard N.
        • Ghalehjegh S.H.
        • Henkel
        • et al.
        Artificial intelligence for real-time multiple polyp detection with identification, tracking, and optical biopsy during colonoscopy.
        Gastroenterology. 2019; 156: S48-S49
        • Ozawa T.
        • Ishihara S.
        • Fujishiro M.
        • et al.
        Novel computer-assisted diagnosis system for endoscopic disease activity in patients with ulcerative colitis.
        Gastrointest Endosc. 2019; 89: 416-421
        • Stidham R.W.
        • Liu W.
        • Bishu H.
        • et al.
        Performance of a deep learning model vs human reviewers in grading endoscopic disease severity of patients with ulcerative colitis.
        JAMA Netw Open. 2019; 2e193963
        • Maeda Y.
        • Kudo S.-E.
        • Mori Y.
        • et al.
        Fully automated diagnostic system with artificial intelligence using endocytoscopy to identify the presence of histologic inflammation associated with ulcerative colitis (with video).
        Gastrointest Endosc. 2019; 89: 408-415
        • Zou Y.
        • Li L.
        • Wang Y.
        • et al.
        Classifying digestive organs in wireless capsule endoscopy images based on deep convolutional neural network.
        in: Presented at 2015 IEEE International Conference on Digital Signal Processing. IEEE, Piscataway, NJ2015: 1274-1278
        • Leenhardt R.
        • et al.
        A neural network algorithm for detection of GI angiectasia during small-bowel capsule endoscopy.
        Gastrointest Endosc. 2019; 89: 189-194
        • Aoki T.
        • Yamada A.
        • Aoyama K.
        • et al.
        Automatic detection of erosions and ulcerations in wireless capsule endoscopy images based on a deep convolutional neural network.
        Gastrointest Endosc. 2019; 89: 357-363
        • He J.-Y.
        • Wu X.
        • Jiang Y.-G.
        • et al.
        Hookworm detection in wireless capsule endoscopy images with deep learning.
        IEEE Trans Image Process. 2018; 27: 2379-2392
        • Saito H.
        • Aoki T.
        • Aoyama K.
        • et al.
        Automatic detection and classification of protruding lesions in wireless capsule endoscopy images based on a deep convolutional neural network.
        Gastrointest Endosc. 2020; 92: 144-151
        • Yuan Y.
        • Meng M.Q.H.
        Deep learning for polyp recognition in wireless capsule endoscopy images.
        Med Phys. 2017; 44: 1379-1389