Identification of patients with malignant biliary strictures using a cholangioscopy-based deep learning artificial intelligence (with video)

Published:August 22, 2022DOI:

      Background and Aims

      Accurately diagnosing malignant biliary strictures (MBSs) as benign or malignant remains challenging. It has been suggested that direct visualization and interpretation of cholangioscopy images provide greater accuracy for stricture classification than current sampling techniques (ie, brush cytology and forceps biopsy sampling) using ERCP. We aimed to develop a convolutional neural network (CNN) model capable of accurate stricture classification and real-time evaluation based solely on cholangioscopy image analysis.


      Consecutive patients with cholangioscopy examinations from 2012 to 2021 were reviewed. A CNN was developed and tested using cholangioscopy images with direct expert annotations. The CNN was then applied to a multicenter, reserved test set of cholangioscopy videos. CNN performance was then directly compared with that of ERCP sampling techniques. Occlusion block heatmap analyses were used to evaluate and rank cholangioscopy features associated with MBSs.


      One hundred fifty-four patients with available cholangioscopy examinations were included in the study. The final image database comprised 2,388,439 still images. The CNN demonstrated good performance when tasked with mimicking expert annotations of high-quality malignant images (area under the receiver-operating characteristic curve, .941). Overall accuracy of CNN-based video analysis (.906) was significantly greater than that of brush cytology (.625, P = .04) or forceps biopsy sampling (.609, P = .03). Occlusion block heatmap analysis demonstrated that the most frequent image feature for an MBS was the presence of frond-like mucosa/papillary projections.


      This study demonstrates that a CNN developed using cholangioscopy data alone has greater accuracy for biliary stricture classification than traditional ERCP-based sampling techniques.

      Graphical abstract


      AI (artificial intelligence), AUROC (area under the receiver-operating characteristic curve), CCA (cholangiocarcinoma), CI (confidence interval), CNN (convolutional neural network), HQB (high quality-benign), HQM (high quality-malignant), HQS (high quality-suspicious), LQ (Low quality), MBS (malignant biliary strictures), ROC (receiver-operating characteristic), UNI (uninformative)
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Gastrointestinal Endoscopy
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Yao K.J.
        • Jabbour S.
        • Parekh N.
        • et al.
        Increasing mortality in the United States from cholangiocarcinoma: an analysis of the National Center for Health Statistics Database.
        BMC Gastroenterol. 2016; 16: 117
        • Lindner P.
        • Rizell M.
        • Hafstrom L.
        The impact of changed strategies for patients with cholangiocarcinoma in this millenium.
        HPB Surg. 2015; 2015736049
        • Strijker M.
        • Belkouz A.
        • van der Geest L.G.
        • et al.
        Treatment and survival of resected and unresected distal cholangiocarcinoma: a nationwide study.
        Acta Oncol. 2019; 58: 1048-1055
        • Kamsa-Ard S.
        • Luvira V.
        • Suwanrungruang K.
        • et al.
        Cholangiocarcinoma trends, incidence, and relative survival in Khon Kaen, Thailand from 1989 through 2013: a population-based cancer registry study.
        J Epidemiol. 2019; 29: 197-204
        • Gerhards M.F.
        • Vos P.
        • van Gulik T.M.
        • et al.
        Incidence of benign lesions in patients resected for suspicious hilar obstruction.
        Br J Surg. 2001; 88: 48-51
        • Clayton R.A.
        • Clarke D.L.
        • Currie E.J.
        • et al.
        Incidence of benign pathology in patients undergoing hepatic resection for suspected malignancy.
        Surgeon. 2003; 1: 32-38
        • Sun B.
        • Moon J.H.
        • Cai Q.
        • et al.
        Asia-Pacific consensus recommendations on endoscopic tissue acquisition for biliary strictures [Review].
        Aliment Pharmacol Ther. 2018; 48: 138-151
        • Navaneethan U.
        • Njei B.
        • Lourdusamy V.
        • et al.
        Comparative effectiveness of biliary brush cytology and intraductal biopsy for detection of malignant biliary strictures: a systematic review and meta-analysis.
        Gastrointest Endosc. 2015; 81: 168-176
        • Baroud S.
        • Sahakian A.J.
        • Sawas T.
        • et al.
        Impact of trimodality sampling on detection of malignant biliary strictures compared with patients with primary sclerosing cholangitis.
        Gastrointest Endosc. 2022; 95: 884-892
        • Sadeghi A.
        • Mohamadnejad M.
        • Islami F.
        • et al.
        Diagnostic yield of EUS-guided FNA for malignant biliary stricture: a systematic review and meta-analysis.
        Gastrointest Endosc. 2016; 83: 290-298
        • Heimbach J.K.
        • Sanchez W.
        • Rosen C.B.
        • et al.
        Trans-peritoneal fine needle aspiration biopsy of hilar cholangiocarcinoma is associated with disease dissemination.
        HPB. 2011; 13: 356-360
        • Levy M.J.
        • Gleeson F.C.
        • Campion M.B.
        • et al.
        Prospective cytological assessment of gastrointestinal luminal fluid acquired during EUS: a potential source of false-positive FNA and needle tract seeding.
        Am J Gastroenterol. 2010; 105: 1311-1318
        • Navaneethan U.
        • Hasan M.K.
        • Lourdusamy V.
        • et al.
        Single-operator cholangioscopy and targeted biopsies in the diagnosis of indeterminate biliary strictures: a systematic review.
        Gastrointest Endosc. 2015; 82: 608-614
        • Sun X.
        • Zhou Z.
        • Tian J.
        • et al.
        Is single-operator peroral cholangioscopy a useful tool for the diagnosis of indeterminate biliary lesion? A systematic review and meta-analysis.
        Gastrointest Endosc. 2015; 82: 79-87
        • Kaura K.
        • Sawas T.
        • Bazerbachi F.
        • et al.
        Cholangioscopy biopsies improve detection of cholangiocarcinoma when combined with cytology and FISH, but not in patients with PSC.
        Dig Dis Sci. 2020; 65: 1471-1478
        • Lenze F.
        • Bokemeyer A.
        • Gross D.
        • et al.
        Safety, diagnostic accuracy and therapeutic efficacy of digital single-operator cholangioscopy.
        United Eur Gastroenterol J. 2018; 6: 902-909
        • Navaneethan U.
        • Hasan M.K.
        • Kommaraju K.
        • et al.
        Digital, single-operator cholangiopancreatoscopy in the diagnosis and management of pancreatobiliary disorders: a multicenter clinical experience (with video).
        Gastrointest Endosc. 2016; 84: 649-655
        • Shah R.J.
        • Raijman I.
        • Brauer B.
        • et al.
        Performance of a fully disposable, digital, single-operator cholangiopancreatoscope.
        Endoscopy. 2017; 49: 651-658
        • Turowski F.
        • Hugle U.
        • Dormann A.
        • et al.
        Diagnostic and therapeutic single-operator cholangiopancreatoscopy with SpyGlassDS: results of a multicenter retrospective cohort study.
        Surg Endosc. 2018; 32: 3981-3988
        • Stassen P.M.C.
        • Goodchild G.
        • de Jonge P.J.F.
        • et al.
        Diagnostic accuracy and interobserver agreement of digital single-operator cholangioscopy for indeterminate biliary strictures.
        Gastrointest Endosc. 2021; 94: 1059-1068
        • Hirasawa T.
        • Aoyama K.
        • Tanimoto T.
        • et al.
        Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images.
        Gastric Cancer. 2018; 21: 653-660
        • Ribeiro T.
        • Saraiva M.M.
        • Ferreira J.P.S.
        • et al.
        Artificial intelligence and capsule endoscopy: automatic detection of vascular lesions using a convolutional neural network.
        Ann Gastroenterol. 2021; 34: 820-828
        • Marya N.B.
        • Powers P.D.
        • Chari S.T.
        • et al.
        Utilisation of artificial intelligence for the development of an EUS-convolutional neural network model trained to enhance the diagnosis of autoimmune pancreatitis.
        Gut. 2021; 70: 1335-1344
        • Marya N.B.
        • Powers P.D.
        • Fujii-Lau L.
        • et al.
        Application of artificial intelligence using a novel EUS-based convolutional neural network model to identify and distinguish benign and malignant hepatic masses.
        Gastrointest Endosc. 2021; 93: 1121-1130
        • Blechacz B.
        • Komuta M.
        • Roskams T.
        • et al.
        Clinical diagnosis and staging of cholangiocarcinoma.
        Nat Rev Gastroenterol Hepatol. 2011; 8: 512-522
        • He K.
        • Zhang X.
        • Ren S.
        • et al.
        Deep residual learning for image recognition.
        ArXiv. Epub. 2015; (Dec 10. Available at Accessed December 2, 2022)
      1. Deng J, Dong W, Socher R, et al. Imagenet: a large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition; 2009. p. 248-255.

        • Gulli A.
        • Pal S.
        Deep learning with Keras.
        Birmingham, UK: Packt Publishing Ltd, 2017
      2. Martin A, Agarwal A, Barham P, et al. TensorFlow: large-scale machine learning on heterogeneous distributed systems. ArXiv. Epub 2015 Mar 16. Available at: Accessed December 2, 2022.

        • Pedregosa F.
        • Varoquax G.
        • Gramfort A.
        • et al.
        Scikit-learn: machine learning in Python.
        Journal of Machine Learning Research. 2011; 12: 2825-2830
        • Saraiva M.M.
        • Ribeiro T.
        • Ferreira J.P.S.
        • et al.
        Artificial intelligence for automatic diagnosis of biliary stricture malignancy status in single-operator cholangioscopy: a pilot study.
        Gastrointest Endosc. 2022; 95: 339-348
        • Gong Z.
        • Zhong P.
        • Hu W.
        Diversity in machine learning.
        IEEE Access. 2019; 7: 64323-64350
        • Sethi A.
        • Tyberg A.
        • Slivka A.
        • et al.
        Digital single-operator cholangioscopy (DSOC) improves interobserver agreement (IOA) and accuracy for evaluation of indeterminate biliary strictures: the Monaco classification.
        J Clin Gastroenterol. 2022; 56: e94-e97
        • Okagawa Y.
        • Abe S.
        • Yamada M.
        • et al.
        Artificial intelligence in endoscopy.
        Dig Dis Sci. 2022; 67: 1553-1572
        • Guo L.
        • Xiao X.
        • Wu C.
        • et al.
        Real-time automated diagnosis of precancerous lesions and early esophageal squamous cell carcinoma using a deep learning model (with videos).
        Gastrointest Endosc. 2020; 91: 41-51
        • Marusic M.
        • Paic M.
        • Knobloch M.
        • et al.
        Klatskin-mimicking lesions.
        Diagnostics. 2021; : 11
        • Shah R.J.
        • Langer D.A.
        • Antillon M.R.
        • et al.
        Cholangioscopy and cholangioscopic forceps biopsy in patients with indeterminate pancreaticobiliary pathology.
        Clin Gastroenterol Hepatol. 2006; 4: 219-225
        • Torrealba M.
        • Berzosa M.
        • Raijman I.
        Single-operator cholangioscopy for the diagnosis of bile duct lymphoma: a case report and brief review of the literature.
        Endoscopy. 2013; 45: E359-E360