Advertisement

Comparing the number and relevance of false activations between 2 artificial intelligence computer-aided detection systems: the NOISE study

Published:January 03, 2022DOI:https://doi.org/10.1016/j.gie.2021.12.031

      Background and Aims

      Artificial intelligence has been shown to be effective in polyp detection, and multiple computer-aided detection (CADe) systems have been developed. False-positive (FP) activation emerged as a possible way to benchmark CADe performance in clinical practice. The aim of this study was to validate a previously developed classification of FPs comparing the performances of different brands of approved CADe systems.

      Methods

      We compared 2 different consecutive video libraries (40 video per arm) collected at Humanitas Research Hospital with 2 different CADe system brands (CADe A and CADe B). For each video, the number of CADe false activations, cause, and time spent by the endoscopist to examine the area erroneously highlighted were reported. The FP activations were classified according to the previously developed classification of FPs (the NOISE classification) according to their cause and relevance.

      Results

      In CADe A 1021 FP activations were registered across the 40 videos (25.5 ± 12.2 FPs per colonoscopy), whereas in CADe B 1028 were identified (25.7 ± 13.2 FPs per colonoscopy; P = .53). Among them, 22.9 ± 9.9 (89.8% in CADe A) and 22.1 ± 10.0 (86.0% in CADe B) were because of artifacts from the bowel wall. Conversely, 2.6 ± 1.9 (10.2% in CADe A) and 3.5 ± 2.1 (14% in CADe B) were caused by bowel content (P = .45). Within CADe A each false activation required .2 ± .9 seconds, with 1.6 ± 1.0 FPs (6.3%) requiring additional time for endoscopic assessment. Comparable results were reported within CADe B with .2 ± .8 seconds spent per false activation and 1.8 ± 1.2 FPs per colonoscopy requiring additional inspection.

      Conclusions

      The use of a standardized nomenclature provided comparable results with either of the 2 recently approved CADe systems. (Clinical trial registration number: NCT04399590.)

      Abbreviations:

      AI (artificial intelligence), CADe (computer-aided detection), FP (false positive)
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Gastrointestinal Endoscopy
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Hassan C.
        • Wallace M.B.
        • Sharma P.
        • et al.
        New artificial intelligence system: first validation study versus experienced endoscopists for colorectal polyp detection.
        Gut. 2020; 69: 799-800
        • Fernández-Esparrach G.
        • Bernal J.
        • López-Cerón M.
        • et al.
        Exploring the clinical potential of an automatic colonic polyp detection method based on the creation of energy maps.
        Endoscopy. 2016; 48: 837-842
        • Misawa M.
        • Kudo S.E.
        • Mori Y.
        • et al.
        Artificial intelligence-assisted polyp detection for colonoscopy: initial experience.
        Gastroenterology. 2018; 154: 2027-2029
        • Attardo S.
        • Chandrasekar V.T.
        • Spadaccini M.
        • et al.
        Artificial intelligence technologies for the detection of colorectal lesions: the future is now.
        World J Gastroenterol. 2020; 26: 5606-5616
        • Wang P.
        • Berzin T.M.
        • Glissen Brown J.R.
        • et al.
        Real-time automatic detection system increases colonoscopic polyp and adenoma detection rates: a prospective randomised controlled study.
        Gut. 2019; 68: 1813-1819
        • Su J.R.
        • Li Z.
        • Shao X.J.
        • et al.
        Impact of a real-time automatic quality control system on colorectal polyp and adenoma detection: a prospective randomized controlled study (with videos).
        Gastrointest. Endosc. 2020; 91: 415-424
        • Gong D.
        • Wu L.
        • Zhang J.
        • et al.
        Detection of colorectal adenomas with a real-time computer-aided system (ENDOANGEL): a randomised controlled study.
        Lancet Gastroenterol Hepatol. 2020; 5: 352-361
        • Liu W.N.
        • Zhang Y.Y.
        • Bian X.Q.
        • et al.
        Study on detection rate of polyps and adenomas in artificial-intelligence-aided colonoscopy.
        Saudi J Gastroenterol. 2020; 26: 13-19
        • Repici A.
        • Badalamenti M.
        • Maselli R.
        • et al.
        Efficacy of real-time computer-aided detection of colorectal neoplasia in a randomized trial.
        Gastroenterology. 2020; 159: 512-520
        • Repici A.
        • Spadaccini M.
        • Antonelli G.
        • et al.
        Artificial intelligence and colonoscopy experience: lessons from two randomised trials.
        Gut. 2022; 71: 757-765
        • Hassan C.
        • Spadaccini M.
        • Iannone A.
        • et al.
        Performance of artificial intelligence in colonoscopy for adenoma and polyp detection: a systematic review and meta-analysis.
        Gastrointest Endosc. 2021; 93: 77-85
        • Glissen Brown J.R.
        • Mansour N.M.
        • Wang P.
        • et al.
        Deep learning computer-aided polyp detection reduces adenoma miss rate: a U.S. multi-center randomized tandem colonoscopy study (CADeT-CS Trial).
        Clin Gastroenterol Hepatol. 2021; (Epub 2021 Sep 14)
        • Wang P.
        • Liu P.
        • Glissen Brown J.R.
        • et al.
        Lower adenoma miss rate of computer-aided detection-assisted colonoscopy vs routine white-light colonoscopy in a prospective tandem study.
        Gastroenterology. 2020; 159: 1252-1261
        • Spadaccini M.
        • Iannone A.
        • Maselli R.
        • et al.
        Computer-aided detection versus advanced imaging for detection of colorectal neoplasia: a systematic review and network meta-analysis.
        Lancet Gastroenterol Hepatol. 2021; 6: 793-802
        • Van der Sommen F.
        • de Groof J.
        • Struyvenberg M.
        • et al.
        Machine learning in GI endoscopy: practical guidance in how to interpret a novel field.
        Gut. 2020; 69: 2035-2045
        • Hoogenboom S.A.
        • Bagci U.
        • Wallace M.B.
        AI in gastroenterology. The current state of play and the potential. How will it affect our practice and when?.
        Techn Gastrointest Endosc. 2020; 22: 42-47
        • Lui T.K.
        • Guo C.-G.
        • Leung W.K.
        Accuracy of artificial intelligence on histology prediction and detection of colorectal polyps: a systematic review and meta-analysis.
        Gastrointest Endosc. 2020; 92: 11-22
        • Greenhill A.T.
        • Edmunds B.R.
        A primer of artificial intelligence in medicine.
        Techn Gastrointest Endosc. 2020; 22: 85-89
        • Vinsard D.G.
        • Mori Y.
        • Misawa M.
        • et al.
        Quality assurance of computer-aided detection and diagnosis in colonoscopy.
        Gastrointest Endosc. 2019; 90: 55-63
        • Antonelli G.
        • Gkolfakis P.
        • Tziatzios G.
        • et al.
        Artificial intelligence-aided colonoscopy: recent developments and future perspectives.
        World J Gastroenterol. 2020; 26: 7436-7443
        • Hsieh Y.-H.
        • Tang C.-P.
        • Tseng C.-W.
        • et al.
        Computer-aided detection false positives in colonoscopy.
        Diagnostics. 2021; 11: 1113
        • Gilbert F.J.
        • Astley S.M.
        • Gillan M.G.
        • et al.
        Single reading with computer-aided detection for screening mammography.
        N Engl J Med. 2008; 359: 1675-1684
        • Ahmad O.F.
        • Mori Y.
        • Misawa M.
        • et al.
        Establishing key research questions for the implementation of artificial intelligence in colonoscopy—a modified Delphi method.
        Endoscopy. 2021; 53: 893-901
        • Hassan C.
        • Badalamenti M.
        • Maselli R.
        • et al.
        Computer-aided detection-assisted colonoscopy: classification and relevance of false positives.
        Gastrointest Endosc. 2020; 92: 900-904
        • Neumann H.
        • Kreft A.
        • Sivanathan V.
        • et al.
        Evaluation of novel LCI CAD EYE system for real time detection of colon polyps.
        PLoS One. 2021; 16e0255955
        • Guo Z.
        • Nemoto D.
        • Zhu X.
        • et al.
        Polyp detection algorithm can detect small polyps: ex vivo reading test compared with endoscopists.
        Dig Endosc. 2021; 33: 162-169
        • Ahmad O.F.
        • González-Bueno Puyal J.
        • Brandao P.
        • et al.
        Performance of artificial intelligence for detection of subtle and advanced colorectal neoplasia.
        Dig Endosc. 2021; (Epub 2021 Nov 8)
        • Holzwanger E.A.
        • Bilal M.
        • Glissen Brown J.R.
        • et al.
        Benchmarking definitions of false-positive alerts during computer-aided polyp detection in colonoscopy.
        Endoscopy. 2021; 53: 937-940