Real-time automated diagnosis of colorectal cancer invasion depth using a deep learning model with multimodal data (with video)

  • Author Footnotes
    ∗ Drs Lu, Xu, and Yao contributed equally to this article.
    Zihua Lu
    Footnotes
    ∗ Drs Lu, Xu, and Yao contributed equally to this article.
    Affiliations
    Department of Gastroenterology, Renmin Hospital of Wuhan University, Wuhan, China

    Hubei Provincial Clinical Research Center for Digestive Disease Minimally Invasive Incision, Renmin Hospital of Wuhan University, Wuhan, China

    Key Laboratory of Hubei Province for Digestive System Disease, Renmin Hospital of Wuhan University, Wuhan, China
    Search for articles by this author
  • Author Footnotes
    ∗ Drs Lu, Xu, and Yao contributed equally to this article.
    Youming Xu
    Footnotes
    ∗ Drs Lu, Xu, and Yao contributed equally to this article.
    Affiliations
    Department of Gastroenterology, Renmin Hospital of Wuhan University, Wuhan, China

    Hubei Provincial Clinical Research Center for Digestive Disease Minimally Invasive Incision, Renmin Hospital of Wuhan University, Wuhan, China

    Key Laboratory of Hubei Province for Digestive System Disease, Renmin Hospital of Wuhan University, Wuhan, China
    Search for articles by this author
  • Author Footnotes
    ∗ Drs Lu, Xu, and Yao contributed equally to this article.
    Liwen Yao
    Footnotes
    ∗ Drs Lu, Xu, and Yao contributed equally to this article.
    Affiliations
    Department of Gastroenterology, Renmin Hospital of Wuhan University, Wuhan, China

    Hubei Provincial Clinical Research Center for Digestive Disease Minimally Invasive Incision, Renmin Hospital of Wuhan University, Wuhan, China

    Key Laboratory of Hubei Province for Digestive System Disease, Renmin Hospital of Wuhan University, Wuhan, China
    Search for articles by this author
  • Wei Zhou
    Affiliations
    Department of Gastroenterology, Renmin Hospital of Wuhan University, Wuhan, China

    Hubei Provincial Clinical Research Center for Digestive Disease Minimally Invasive Incision, Renmin Hospital of Wuhan University, Wuhan, China

    Key Laboratory of Hubei Province for Digestive System Disease, Renmin Hospital of Wuhan University, Wuhan, China
    Search for articles by this author
  • Wei Gong
    Affiliations
    Department of Gastroenterology, Shenzhen Hospital of Southern Medical University, Shenzhen, China
    Search for articles by this author
  • Genhua Yang
    Affiliations
    Department of Gastroenterology, Shenzhen Hospital of Southern Medical University, Shenzhen, China
    Search for articles by this author
  • Mingwen Guo
    Affiliations
    Department of Gastroenterology, The First Hospital of Yichang, Yichang, China
    Search for articles by this author
  • Beiping Zhang
    Affiliations
    Department of Gastroenterology, Guangdong Province Traditional Chinese Medical Hospital, Guangzhou, China
    Search for articles by this author
  • Xu Huang
    Affiliations
    Department of Gastroenterology, Renmin Hospital of Wuhan University, Wuhan, China

    Hubei Provincial Clinical Research Center for Digestive Disease Minimally Invasive Incision, Renmin Hospital of Wuhan University, Wuhan, China

    Key Laboratory of Hubei Province for Digestive System Disease, Renmin Hospital of Wuhan University, Wuhan, China
    Search for articles by this author
  • Chunping He
    Affiliations
    Department of Gastroenterology, Renmin Hospital of Wuhan University, Wuhan, China

    Hubei Provincial Clinical Research Center for Digestive Disease Minimally Invasive Incision, Renmin Hospital of Wuhan University, Wuhan, China

    Key Laboratory of Hubei Province for Digestive System Disease, Renmin Hospital of Wuhan University, Wuhan, China
    Search for articles by this author
  • Rui Zhou
    Affiliations
    Department of Gastroenterology, Renmin Hospital of Wuhan University, Wuhan, China

    Hubei Provincial Clinical Research Center for Digestive Disease Minimally Invasive Incision, Renmin Hospital of Wuhan University, Wuhan, China

    Key Laboratory of Hubei Province for Digestive System Disease, Renmin Hospital of Wuhan University, Wuhan, China
    Search for articles by this author
  • Yunchao Deng
    Affiliations
    Department of Gastroenterology, Renmin Hospital of Wuhan University, Wuhan, China

    Hubei Provincial Clinical Research Center for Digestive Disease Minimally Invasive Incision, Renmin Hospital of Wuhan University, Wuhan, China

    Key Laboratory of Hubei Province for Digestive System Disease, Renmin Hospital of Wuhan University, Wuhan, China
    Search for articles by this author
  • Honggang Yu
    Correspondence
    Reprint requests: Honggang Yu, MD, Department of Gastroenterology, Renmin Hospital of Wuhan University, Jiefang Road 238, Hubei Province, Wuhan 430060, China.
    Affiliations
    Department of Gastroenterology, Renmin Hospital of Wuhan University, Wuhan, China

    Hubei Provincial Clinical Research Center for Digestive Disease Minimally Invasive Incision, Renmin Hospital of Wuhan University, Wuhan, China

    Key Laboratory of Hubei Province for Digestive System Disease, Renmin Hospital of Wuhan University, Wuhan, China
    Search for articles by this author
  • Author Footnotes
    ∗ Drs Lu, Xu, and Yao contributed equally to this article.
Published:December 14, 2021DOI:https://doi.org/10.1016/j.gie.2021.11.049

      Background and Aims

      The optical diagnosis of colorectal cancer (CRC) invasion depth with white light (WL) and image-enhanced endoscopy (IEE) remains challenging. We aimed to construct and validate a 2-modal deep learning–based system, incorporated with both WL and IEE images (named Endo-CRC) in estimating the invasion depth of CRC.

      Methods

      Samples were retrospectively obtained from 3 hospitals in China. We combined WL and IEE images into image pairs. Altogether, 337,278 image pairs from 268 noninvasive and superficial CRC and 181,934 image pairs from 82 deep CRC were used for training. A total of 296,644 and 4528 image pairs were used for internal and external tests and for comparison with endoscopists. Thirty-five videos were used for evaluating the real-time performance of the Endo-CRC system. Two deep learning models, solely using either WL (model W) or IEE images (model I), were constructed to compare with Endo-CRC.

      Results

      The accuracies of Endo-CRC in internal image tests with and without advanced CRC were 91.61% and 93.78%, respectively, and 88.65% in the external test, which did not include advanced CRC. In an endoscopist–machine competition, Endo-CRC achieved an expert comparable accuracy of 88.11% and the highest sensitivity compared with all endoscopists. In a video test, Endo-CRC achieved an accuracy of 100.00%. Compared with model W and model I, Endo-CRC had a higher accuracy (per image pair: 91.61% vs 88.27% compared with model I and 91.61% vs 81.32% compared with model W).

      Conclusions

      The Endo-CRC system has great potential for assisting in CRC invasion depth diagnosis and may be well applied in clinical practice.

      Abbreviations:

      AI (artificial intelligence), BLI (blue laser imaging), CNN (convolutional neural network), CRC (colorectal cancer), ER (endoscopic resection), IEE (image-enhanced endoscopy), NBI (narrow-band imaging), suit-ER (lesions suitable for endoscopic resection), unsuit-ER (lesions unsuitable for endoscopic resection), WL (white light)
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Gastrointestinal Endoscopy
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Bray F.
        • Ferlay J.
        • Soerjomataram I.
        • et al.
        Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries.
        CA Cancer J Clin. 2018; 68: 394-424
        • Tanaka S.
        • Kashida H.
        • Saito Y.
        • et al.
        JGES guidelines for colorectal endoscopic submucosal dissection/endoscopic mucosal resection.
        Dig Endosc. 2015; 27: 417-434
        • Yoda Y.
        • Ikematsu H.
        • Matsuda T.
        • et al.
        A large-scale multicenter study of long-term outcomes after endoscopic resection for submucosal invasive colorectal cancer.
        Endoscopy. 2013; 45: 718-724
        • Shaukat A.
        • Kaltenbach T.
        • Dominitz J.A.
        • et al.
        Endoscopic recognition and management strategies for malignant colorectal polyps: recommendations of the US Multi-Society Task Force on Colorectal Cancer.
        Gastrointest Endosc. 2020; 92: 997-1015
        • Bisschops R.
        • East J.E.
        • Hassan C.
        • et al.
        Advanced imaging for detection and differentiation of colorectal neoplasia: European Society of Gastrointestinal Endoscopy (ESGE) guideline—update 2019.
        Endoscopy. 2019; 51: 1155-1179
        • Rutter M.D.
        • Chattree A.
        • Barbour J.A.
        • et al.
        British Society of Gastroenterology/Association of Coloproctologists of Great Britain and Ireland guidelines for the management of large non-pedunculated colorectal polyps.
        Gut. 2015; 64: 1847-1873
        • Backes Y.
        • Schwartz M.P.
        • Ter Borg F.
        • et al.
        Multicentre prospective evaluation of real-time optical diagnosis of T1 colorectal cancer in large non-pedunculated colorectal polyps using narrow band imaging (the OPTICAL study).
        Gut. 2019; 68: 271-279
        • Peery A.F.
        • Cools K.S.
        • Strassle P.D.
        • et al.
        Increasing rates of surgery for patients with nonmalignant colorectal polyps in the United States.
        Gastroenterology. 2018; 154: 1352-1360
        • Le Berre C.
        • Sandborn W.J.
        • Aridhi S.
        • et al.
        Application of artificial intelligence to gastroenterology and hepatology.
        Gastroenterology. 2020; 158: 76-94
        • Cao J.S.
        • Lu Z.Y.
        • Chen M.Y.
        • et al.
        Artificial intelligence in gastroenterology and hepatology: status and challenges.
        World J Gastroenterol. 2021; 27: 1664
        • Wang K.W.
        • Dong M.
        Potential applications of artificial intelligence in colorectal polyps and cancer: recent advances and prospects.
        World J Gastroenterol. 2020; 26: 5090
        • Tokunaga M.
        • Matsumura T.
        • Nankinzan R.
        • et al.
        Computer-aided diagnosis system using only white-light endoscopy for the prediction of invasion depth in colorectal cancer.
        Gastrointest Endosc. 2021; 93: 647-653
        • Xiaobei L.
        • Jiahao W.
        • Zelong H.
        • et al.
        Artificial intelligence−enhanced white-light colonoscopy with attention guidance predicts colorectal cancer invasion depth.
        Gastrointest Endosc. 2021; 94: 627-638
        • Tamai N.
        • Saito Y.
        • Sakamoto T.
        • et al.
        Effectiveness of computer-aided diagnosis of colorectal lesions using novel software for magnifying narrow-band imaging: a pilot study.
        Endosc Int Open. 2017; 5: E690-E694
        • Kaltenbach T.
        • Anderson J.C.
        • Burke C.A.
        • et al.
        Endoscopic removal of colorectal lesions—recommendations by the US Multi-Society Task Force on Colorectal Cancer.
        Gastroenterology. 2020; 158: 1095-1129
        • Deng J.
        • Dong W.
        • Socher R.
        • et al.
        Imagenet: a large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition.
        IEEE. 2009; (Available at:) (Accessed January 19, 2022): 248-255
        • Abadi M.
        • Barham P.
        • Chen J.
        • et al.
        Tensorflow: a system for large-scale machine learning.
        12th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 16). 2016; (Available at:) (Accessed January 19, 2022): 265-283
        • Baldi P.
        • Sadowski P.
        The dropout learning algorithm.
        Artif Intell. 2014; 210: 78-122
        • Mikołajczyk A.
        • Grochowski M.
        Data augmentation for improving deep learning in image classification problem. 2018 International Interdisciplinary PhD Workshop (IIPhDW).
        IEEE. 2018; (Available at:) (Accessed January 19, 2022): 117-122
        • Duvenaud D.
        • Maclaurin D.
        • Adams R.
        Early stopping as nonparametric variational inference. Artificial Intelligence and Statistics.
        PMLR, 2016: 1070-1077 (Available at:)
        • Yu C.
        • Helwig E.J.
        The role of AI technology in prediction, diagnosis and treatment of colorectal cancer.
        Artif Intell Rev. 2021; 55: 323-23
        • Lui T.K.L.
        • Wong K.K.Y.
        • Mak L.L.Y.
        • et al.
        Endoscopic prediction of deeply submucosal invasive carcinoma with use of artificial intelligence.
        Endosc Int Open. 2019; 7: E514-E520
        • Ezoe Y.
        • Muto M.
        • Uedo N.
        • et al.
        Magnifying narrowband imaging is more accurate than conventional white-light imaging in diagnosis of gastric mucosal cancer.
        Gastroenterology. 2011; 141: 2017-2025
        • Leifeld L.
        • Rogler G.
        • Stallmach A.
        • et al.
        White-light or narrow-band imaging colonoscopy in surveillance of ulcerative colitis: a prospective multicenter study.
        Clin Gastroenterol Hepatol. 2015; 13: 1776-1781
        • Fonollà R.
        • EW van der Zander Q.
        • Schreuder R.M.
        • et al.
        A CNN CADx system for multimodal classification of colorectal polyps combining WL, BLI, and LCI modalities.
        Appl Sci. 2020; 10: 5040
        • Singh D.
        • Singh B.
        Effective and efficient classification of gastrointestinal lesions: combining data preprocessing, feature weighting, and improved ant lion optimization.
        J Ambient Intell Human Comput. 2021; 12: 8683-8698
        • Xu W.
        • Lin J.
        • Gao M.
        • et al.
        Rapid computer-aided diagnosis of stroke by serum metabolic fingerprint based multi-modal recognition.
        Adv Sci. 2020; 7: 2002021
        • Chen Y.
        • Li D.
        • Zhang X.
        • et al.
        Computer aided diagnosis of thyroid nodules based on the devised small-datasets multi-view ensemble learning.
        Med Image Anal. 2021; 67: 101819
        • Shen D.
        • Wu G.
        • Suk H.I.
        Deep learning in medical image analysis.
        Annu Rev Biomed Eng. 2017; 19: 221-248
        • Yang Y.J.
        • Cho B.J.
        • Lee M.J.
        • et al.
        Automated classification of colorectal neoplasms in white-light colonoscopy images via deep learning.
        J Clin Med. 2020; 9: 1593
        • Backes Y.
        • Moss A.
        • Reitsma J.B.
        • et al.
        Narrow band imaging, magnifying chromoendoscopy, and gross morphological features for the optical diagnosis of T1 colorectal cancer and deep submucosal invasion: a systematic review and meta-analysis.
        ACG. 2017; 112: 54-64
        • Rex D.K.
        • Ahnen D.J.
        • Baron J.A.
        • et al.
        Serrated lesions of the colorectum: review and recommendations from an expert panel.
        Am J Gastroenterol. 2012; 107: 1315

      Linked Article