Real-time artificial intelligence–based histologic classification of colorectal polyps with augmented visualization

  • Eladio Rodriguez-Diaz
    Affiliations
    Research Service, VA Boston Healthcare System, Boston, MA

    Department of Biomedical Engineering, Boston University College of Engineering, Boston, MA
    Search for articles by this author
  • György Baffy
    Affiliations
    Department of Medicine, Section of Gastroenterology, VA Boston Healthcare System, Boston, MA

    Department of Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA
    Search for articles by this author
  • Wai-Kit Lo
    Affiliations
    Department of Medicine, Section of Gastroenterology, VA Boston Healthcare System, Boston, MA

    Department of Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA
    Search for articles by this author
  • Hiroshi Mashimo
    Affiliations
    Department of Medicine, Section of Gastroenterology, VA Boston Healthcare System, Boston, MA

    Department of Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA
    Search for articles by this author
  • Gitanjali Vidyarthi
    Affiliations
    Section of Gastroenterology, James A. Haley Veterans Hospital, Tampa, FL

    Department of Medicine, Morsani College of Medicine, University of South Florida, Tampa, FL
    Search for articles by this author
  • Shyam S. Mohapatra
    Affiliations
    Research Service, James A. Haley Veterans Hospital, Tampa, FL

    Department of Medicine, Morsani College of Medicine, University of South Florida, Tampa, FL
    Search for articles by this author
  • Satish K. Singh
    Correspondence
    Reprint requests: Satish K. Singh, MD, Department of Medicine, Section of Gastroenterology, VA Boston Healthcare System, 150 South Huntington Ave, GI-111, Boston, MA 02130.
    Affiliations
    Research Service, VA Boston Healthcare System, Boston, MA

    Department of Biomedical Engineering, Boston University College of Engineering, Boston, MA

    Department of Medicine, Section of Gastroenterology, VA Boston Healthcare System, Boston, MA

    Department of Medicine, Boston University School of Medicine, Boston, MA

    Department of Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA
    Search for articles by this author
Published:September 16, 2020DOI:https://doi.org/10.1016/j.gie.2020.09.018

      Background and Aims

      Artificial intelligence (AI)–based computer-aided diagnostic (CADx) algorithms are a promising approach for real-time histology (RTH) of colonic polyps. Our aim is to present a novel in situ CADx approach that seeks to increase transparency and interpretability of results by generating an intuitive augmented visualization of the model’s predicted histology over the polyp surface.

      Methods

      We developed a deep learning model using semantic segmentation to delineate polyp boundaries and a deep learning model to classify subregions within the segmented polyp. These subregions were classified independently and were subsequently aggregated to generate a histology map of the polyp’s surface. We used 740 high-magnification narrow-band images from 607 polyps in 286 patients and over 65,000 subregions to train and validate the model.

      Results

      The model achieved a sensitivity of .96, specificity of .84, negative predictive value (NPV) of .91, and high-confidence rate (HCR) of .88, distinguishing 171 neoplastic polyps from 83 non-neoplastic polyps of all sizes. Among 93 neoplastic and 75 non-neoplastic polyps ≤5 mm, the model achieved a sensitivity of .95, specificity of .84, NPV of .91, and HCR of .86.

      Conclusions

      The CADx model is capable of accurately distinguishing neoplastic from non-neoplastic polyps and provides a histology map of the spatial distribution of localized histologic predictions along the delineated polyp surface. This capability may improve interpretability and transparency of AI-based RTH and offer intuitive, accurate, and user-friendly guidance in real time for the clinical management and documentation of optical histology results.

      Abbreviations:

      AI (artificial intelligence), CADx (computer-aided diagnosis), HCR (high-confidence rate), NBI (narrow-band imaging), NICE (NBI International Colorectal Endoscopic), NPV (negative predictive value), PIVI (Preservation and Incorporation of Valuable Endoscopic Innovations), RTH (real-time histology), SSAP (sessile serrated adenoma/polyp)
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Gastrointestinal Endoscopy
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Rex D.K.
        • Kahi C.
        • O’Brien M.
        • et al.
        The American Society for Gastrointestinal Endoscopy PIVI (Preservation and Incorporation of Valuable Endoscopic Innovations) on real-time endoscopic assessment of the histology of diminutive colorectal polyps.
        Gastrointest Endosc. 2011; 73: 419-422
        • Hassan C.
        • Pickhardt P.J.
        • Rex D.K.
        A resect and discard strategy would improve cost-effectiveness of colorectal cancer screening.
        Clin Gastroenterol Hepatol. 2010; 8: 865-869
        • Kessler W.R.
        • Imperiale T.F.
        • Klein R.W.
        • et al.
        A quantitative assessment of the risks and cost savings of forgoing histologic examination of diminutive polyps.
        Endoscopy. 2011; 43: 683-691
        • Gupta N.
        • Bansal A.
        • Rao D.
        • et al.
        Accuracy of in vivo optical diagnosis of colon polyp histology by narrow-band imaging in predicting colonoscopy surveillance intervals.
        Gastrointest Endosc. 2012; 75: 494-502
        • Mori Y.
        • Kudo S.
        • East J.E.
        • et al.
        Cost savings in colonoscopy with artificial intelligence-aided polyp diagnosis: an add-on analysis of a clinical trial (with video).
        Gastrointest Endosc. 2020; 92: 905-911
        • Sikka S.
        • Ringold D.
        • Jonnalagadda S.
        • et al.
        Comparison of white light and narrow band high definition images in predicting colon polyp histology, using standard colonoscopes without optical magnification.
        Endoscopy. 2008; 40: 818-822
        • Wada Y.
        • Kudo S.
        • Kashida H.
        • et al.
        Diagnosis of colorectal lesions with the magnifying narrow-band imaging system.
        Gastrointest Endosc. 2009; 70: 522-531
        • Rastogi A.
        • Pondugula K.
        • Bansal A.
        • et al.
        Recognition of surface mucosal and vascular patterns of colon polyps by using narrow-band imaging: interobserver and intraobserver agreement and prediction of polyp histology.
        Gastrointest Endosc. 2009; 69: 716-722
        • Rex D.K.
        Narrow-band imaging without optical magnification for histologic analysis of colorectal polyps.
        Gastroenterology. 2009; 136: 1174-1181
        • Hewett D.G.
        • Kaltenbach T.
        • Sano Y.
        • et al.
        Validation of a simple classification system for endoscopic diagnosis of small colorectal polyps using narrow-band imaging.
        Gastroenterol Endosc. 2012; 54: 3642
        • IJspeert J.E.G.
        • Bastiaansen B.A.J.
        • Van Leerdam M.E.
        • et al.
        Development and validation of the WASP classification system for optical diagnosis of adenomas, hyperplastic polyps and sessile serrated adenomas/polyps.
        Gut. 2016; 65: 963-970
        • Ladabaum U.
        • Fioritto A.
        • Mitani A.
        • et al.
        Real-time optical biopsy of colon polyps with narrow band imaging in community practice does not yet meet key thresholds for clinical decisions.
        Gastroenterology. 2013; 144: 81-91
        • Rastogi A.
        • Rao D.S.
        • Gupta N.
        • et al.
        Impact of a computer-based teaching module on characterization of diminutive colon polyps by using narrow-band imaging by non-experts in academic and community practice: a video-based study.
        Gastrointest Endosc. 2014; 79: 390-398
        • Abu Dayyeh B.K.
        • Thosani N.
        • Konda V.
        • et al.
        ASGE Technology Committee systematic review and meta-analysis assessing the ASGE PIVI thresholds for adopting real-time endoscopic assessment of the histology of diminutive colorectal polyps.
        Gastrointest Endosc. 2015; 81: 502-516
        • Mori Y.
        • Kudo S.
        • Chiu P.W.Y.
        • et al.
        Impact of an automated system for endocytoscopic diagnosis of small colorectal lesions: an international web-based study.
        Endoscopy. 2016; 48: 1110-1118
        • Kominami Y.
        • Yoshida S.
        • Tanaka S.
        • et al.
        Computer-aided diagnosis of colorectal polyp histology by using a real-time image recognition system and narrow-band imaging magnifying colonoscopy.
        Gastrointest Endosc. 2016; 83: 643-649
        • Gross S.
        • Trautwein C.
        • Behrens A.
        • et al.
        Computer-based classification of small colorectal polyps by using narrow-band imaging with optical magnification.
        Gastrointest Endosc. 2011; 74: 1354-1359
      1. Tamaki T, Yoshimuta J, Takeda T, et al. A system for colorectal tumor classification in magnifying endoscopic NBI images. In: Kimmel R, Klette R, Sugimoto A (eds). Computer Vision–ACCV 2010, Lecture Notes in Computer Science, 2011;6493:452-63.

        • Tischendorf J.J.W.
        • Gross S.
        • Winograd R.
        • et al.
        Computer-aided classification of colorectal polyps based on vascular patterns: a pilot study.
        Endoscopy. 2010; 42: 203-207
        • Komeda Y.
        • Handa H.
        • Watanabe T.
        • et al.
        Computer-aided diagnosis based on convolutional neural network system for colorectal polyp classification: preliminary experience.
        Oncology. 2017; 93: 30-34
        • Byrne M.F.
        • Chapados N.
        • Soudan F.
        • et al.
        Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model.
        Gut. 2019; 68: 94-100
        • Zachariah R.
        • Samarasena J.
        • Luba D.
        • et al.
        Prediction of polyp pathology using convolutional neural networks achieves “resect and discard” thresholds.
        Am J Gastroenterol. 2020; 115: 138-144
        • Chen P.J.
        • Lin M.C.
        • Lai M.J.
        • et al.
        Accurate classification of diminutive colorectal polyps using computer-aided analysis.
        Gastroenterology. 2018; 154: 568-575
        • Zorron Cheng Tao Pu L.
        • Maicas G.
        • Tian Y.
        • et al.
        Computer-aided diagnosis for characterization of colorectal lesions: comprehensive software that includes differentiation of serrated lesions.
        Gastrointest Endosc. 2020; 92: 891-899
        • Yang Y.J.
        • Bang C.S.
        Application of artificial intelligence in gastroenterology.
        World J Gastroenterol. 2019; 25: 1666-1683
        • Cho B.-J.
        • Bang C.S.
        Artificial intelligence for the determination of a management strategy for diminutive colorectal polyps.
        Am J Gastroenterol. 2020; 115: 70-72
      2. Chen LC, Zhu Y, Papandreou G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Ferrari V, Hebert M, Sminchisescu C, Weiss Y (eds). Computer Vision–ECCV 2018 Lecture Notes in Computer Science, 2018;11211:833-51.

      3. He K, Zhang X, Ren S, et al. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV; 2016:770-8.

        • Fan C.
        • Younis A.
        • Bookhout C.E.
        • et al.
        Management of serrated polyps of the colon.
        Curr Treat Options Gastroenterol. 2018; 16: 182-202
        • Gupta S.
        • Lieberman D.
        • Anderson J.C.
        • et al.
        Recommendations for follow-up after colonoscopy and polypectomy: a consensus update by the US Multi-Society Task Force on Colorectal Cancer.
        Gastroenterology. 2020; 158: 1131-1153
        • Brenner H.
        • Stock C.
        • Hoffmeister M.
        Effect of screening sigmoidoscopy and screening colonoscopy on colorectal cancer incidence and mortality: Systematic review and meta-analysis of randomised controlled trials and observational studies.
        BMJ. 2014; 348
        • Rex D.K.
        • Johnson D.A.
        • Anderson J.C.
        • et al.
        American College of Gastroenterology guidelines for colorectal cancer screening 2008.
        Am J Gastroenterol. 2009; 104: 739-750
        • Rastogi A.
        • Keighley J.
        • Singh V.
        • et al.
        High accuracy of narrow band imaging without magnification for the real-time characterization of polyp histology and its comparison with high-definition white light colonoscopy: a prospective study.
        Am J Gastroenterol. 2009; 104: 2422-2430
        • Mori Y.
        • Kudo S.
        • Misawa M.
        • et al.
        Real-time use of artificial intelligence in identification of diminutive polyps during colonoscopy.
        Ann Intern Med. 2018; 169: 357
        • Berzin T.M.
        • Parasa S.
        • Wallace M.B.
        • et al.
        Position statement on priorities for artificial intelligence in GI endoscopy: a report by the ASGE Task Force.
        Gastrointest Endosc. 2020; 92: 951-959
        • East J.E.
        • Vleugels J.L.
        • Roelandt P.
        • et al.
        Advanced endoscopic imaging: European Society of Gastrointestinal Endoscopy (ESGE) technology review.
        Endoscopy. 2016; 48: 1029-1045