Advertisement

Artificial intelligence using convolutional neural networks for real-time detection of early esophageal neoplasia in Barrett’s esophagus (with video)

Published:January 10, 2020DOI:https://doi.org/10.1016/j.gie.2019.12.049

      Background and Aims

      The visual detection of early esophageal neoplasia (high-grade dysplasia and T1 cancer) in Barrett’s esophagus (BE) with white-light and virtual chromoendoscopy still remains challenging. The aim of this study was to assess whether a convolutional neural artificial intelligence network can aid in the recognition of early esophageal neoplasia in BE.

      Methods

      Nine hundred sixteen images from 65 patients of histology-proven early esophageal neoplasia in BE containing high-grade dysplasia or T1 cancer were collected. The area of neoplasia was masked using image annotation software. Nine hundred nineteen control images were collected of BE without high-grade dysplasia. A convolutional neural network (CNN) algorithm was pretrained on ImageNet and then fine-tuned with the goal of providing the correct binary classification of “dysplastic” or “nondysplastic.” We developed an object detection algorithm that drew localization boxes around regions classified as dysplasia.

      Results

      The CNN analyzed 458 test images (225 dysplasia and 233 nondysplasia) and correctly detected early neoplasia with sensitivity of 96.4%, specificity of 94.2%, and accuracy of 95.4%. With regard to the object detection algorithm for all images in the validation set, the system was able to achieve a mean average precision of .7533 at an intersection over union of .3

      Conclusions

      In this pilot study, our artificial intelligence model was able to detect early esophageal neoplasia in BE images with high accuracy. In addition, the object detection algorithm was able to draw a localization box around the areas of dysplasia with high precision and at a speed that allows for real-time implementation.

      Abbreviations:

      AI (artificial intelligence), BE (Barrett’s esophagus), CNN (convolutional neural network), DL (deep learning), mAP (mean average precision), NBI (narrow-band imaging), WLI (white-light imaging)
      Esophageal cancer is the eighth most common cancer and the sixth leading cause of cancer death worldwide with an estimated incidence of 52,000 cases in 2012.
      • Arnold M.
      • Soerjomataram I.
      • Ferlay J.
      • et al.
      Global incidence of oesophageal cancer by histological subtype in 2012.
      Barrett’s esophagus (BE) is a known risk factor for the development of esophageal adenocarcinoma.
      • Hur C.
      • Miller M.
      • Kong C.Y.
      • et al.
      Trends in esophageal adenocarcinoma incidence and mortality.
      Prognosis is strongly related to stage at diagnosis.
      • Hur C.
      • Miller M.
      • Kong C.Y.
      • et al.
      Trends in esophageal adenocarcinoma incidence and mortality.
      However, more than 40% of patients with esophageal adenocarcinoma are diagnosed after the disease has metastasized, and the 5-year survival rate is less than 20%.
      • Hur C.
      • Miller M.
      • Kong C.Y.
      • et al.
      Trends in esophageal adenocarcinoma incidence and mortality.
      ,
      • Thrift A.P.
      The epidemic of oesophageal carcinoma: Where are we now?.
      The risk of esophageal adenocarcinoma rises when BE is associated with dysplasia.
      • Curvers W.L.
      • Ten Kate F.J.
      • Krishnadath K.K.
      • et al.
      Low-grade dysplasia in Barrett’s esophagus: overdiagnosed and underestimated.
      ,
      • Rastogi A.
      • Puli S.
      • El-Serag H.B.
      • et al.
      Incidence of esophageal adenocarcinoma in patients with Barrett’s esophagus and high-grade dysplasia: a meta-analysis.
      Most GI societies recommend endoscopic surveillance for BE subjects to enable early detection of dysplasia or carcinoma.
      • Shaheen N.J.
      • Falk G.W.
      • Iyer P.G.
      • et al.
      ACG clinical guideline: diagnosis and management of Barrett’s esophagus.
      • Fitzgerald R.C.
      • di Pietro M.
      • Ragunath K.
      • et al.
      British Society of Gastroenterology guidelines on the diagnosis and management of Barrett’s oesophagus.
      • Evans J.A.
      • Early D.S.
      • Fukami N.
      • et al.
      The role of endoscopy in Barrett’s esophagus and other premalignant conditions of the esophagus.
      However, the effect of endoscopic surveillance for BE has been disappointing.
      • Codipilly D.C.
      • Chandar A.K.
      • Singh S.
      • et al.
      The effect of endoscopic surveillance in patients with Barrett's esophagus: a systematic review and meta-analysis.
      Current guidelines recommend endoscopic surveillance in patients with BE with random 4-quadrant biopsy specimens obtained every 1 to 2 cm to detect dysplasia,
      • Sharma P.
      • Savides T.J.
      • Canto M.I.
      • et al.
      The American Society for Gastrointestinal Endoscopy PIVI (Preservation and Incorporation of Valuable Endoscopic Innovations) on imaging in Barrett's Esophagus.
      but this method is invasive, time-consuming, and difficult to comply with.
      • Tavakkoli A.
      • Appelman H.D.
      • Beer D.G.
      • et al.
      Use of appropriate surveillance for patients with nondysplastic Barrett's esophagus.
      As a result, the American Society for Gastrointestinal Endoscopy set a performance threshold for optical diagnosis. That is, if an imaging technology is able to achieve a per-patient sensitivity of 90%, a negative predictive value of 98%, and a specificity of 80% for detecting early esophageal neoplasm,
      • Evans J.A.
      • Early D.S.
      • Fukami N.
      • et al.
      The role of endoscopy in Barrett’s esophagus and other premalignant conditions of the esophagus.
      then random 4-quadrant biopsy sampling could be avoided. Because this requirement is believed to be difficult to achieve for nonexperts,
      • Thosani N.
      • Abu Dayyeh B.K.
      • et al.
      ASGE Technology Committee
      ASGE Technology Committee systematic review and meta-analysis assessing the ASGE Preservation and Incorporation of Valuable Endoscopic Innovations thresholds for adopting real-time imaging-assisted endoscopic targeted biopsy during endoscopic surveillance of Barrett's esophagus.
      a great number of technologies and imaging enhancements have been studied and developed to help improve detection. Among these are chromoendoscopy, endocytoscopy, probe-based confocal laser–induced endomicroscopy, image-enhanced endoscopy, volumetric laser endomicroscopy, magnification endoscopy, and wide-area transepithelial sampling with computer-assisted 3-dimensional analysis.
      • Sami S.S.
      • Iyer P.G.
      Recent advances in screening for Barrett's esophagus.
      Many of these have disadvantages with respect to learning curve, cost, and time issues.
      A simple real-time diagnosis support system would be of great help for endoscopists in the detection of Barrett’s dysplasia. Recently, artificial intelligence (AI) using deep learning (DL) with convolutional neural networks (CNNs) has emerged and showed great results in the diagnosis and detection of lesions in the esophagus,
      • Horie Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
      • Cai S.L.
      • Li B.
      • Tan W.M.
      • et al.
      Using deep learning system in endoscopy for screening of early esophageal squamous cell carcinoma (with video).
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Computer-aided diagnosis using deep learning in the evaluation of early oesophageal adenocarcinoma.
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Real-time use of artificial intelligence in the evaluation of cancer in Barrett's oesophagus.
      stomach,
      • Hirasawa T.
      • Aoyama K.
      • Tanimoto T.
      • et al.
      Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images.
      • Kanesaka T.
      • Lee T.C.
      • Uedo N.
      • et al.
      Computer-aided diagnosis for identifying and delineating early gastric cancers in magnifying narrow-band imaging.
      • Zhang X.
      • Chen F.
      • Yu T.
      • et al.
      Real-time gastric polyp detection using convolutional neural networks.
      • Li L.
      • Chen Y.
      • Shen Z.
      • et al.
      Convolutional neural network for the diagnosis of early gastric cancer based on magnifying narrow band imaging.
      small bowel,
      • Aoki T.
      • Yamada A.
      • Aoyama K.
      • et al.
      Automatic detection of erosions and ulcerations in wireless capsule endoscopy images based on a deep convolutional neural network.
      and colon.
      • Urban G.
      • Tripathi P.
      • Alkayali T.
      • et al.
      Deep learning localizes and identifies polyps in real time with 96% accuracy in screening colonoscopy.
      • Misawa M.
      • Kudo S.E.
      • Mori Y.
      • et al.
      Artificial intelligence-assisted polyp detection for colonoscopy: initial experience.
      • Klare P.
      • Sander C.
      • Prinzen M.
      • et al.
      Automated polyp detection in the colorectum: a prospective study (with videos).
      • Wang P.
      • Xiao X.
      • Brown J.R.
      • et al.
      Development and validation of a deep-learning algorithm for the detection of polyps during colonoscopy.
      However, no study has been reported on an application of DL for detection of early neoplasia within BE. We conducted a pilot study on the endoscopic detection of early esophageal neoplasia on BE using a DL system showing promising results.

      Methods

      Upper endoscopy and dysplasia image dataset

      We retrospectively reviewed all endoscopic images of patients with early esophageal neoplasia in BE, defined in this case as high-grade dysplasia or T1 stage adenocarcinoma, proven by histology at our institution between January 2016 and November 2018 using the electronic endoscopy database. The endoscope used over this time period was the Olympus 190 series upper endoscope (190 HQ and 190 H; Olympus, Center Valley, Pa, USA). The images were categorized by image quality (excellent, adequate, or poor) and by the following imaging techniques: white-light imaging (WLI), narrow-band imaging (NBI), standard focus, and Near Focus (Olympus). Near focus is a proprietary imaging technique to Olympus endoscopes that enables a change to the focal length of the scope lens such that objects within 2 mm of the endoscope tip are maintained in clear focus. Only excellent and adequate images were used.
      Nine hundred sixteen images from 70 patients were retrospectively collected of histology-proven dysplasia (high-grade dysplasia and T1 adenocarcinoma) in BE, and 916 control images from 30 patients were collected of either histology-proven or confocal laser endomicroscopy–proven BE without dysplasia. We did not include cases with low-grade dysplasia for this pilot study. A large diversity of dysplastic lesions was chosen for training in this study to optimize the algorithm. Neoplastic lesions in the image dataset ranged from 3 to 20 mm in size, and most lesions were deemed subtle, such that a lesser-trained endoscopist could miss them.

      Annotation of images

      All neoplastic lesions in the selected images were annotated independently by 2 expert endoscopists (R.H. and J.S.) who were not blinded to endoscopic findings and pathology. Images were annotated with a single rectangular box using image annotation software (Fig. 1, red box). When multiple neoplastic lesions were noted in an image, multiple rectangular boxes were used. This annotation was used as the ground truth for the training of the automated detection algorithm.
      Figure thumbnail gr1
      Figure 1White-light endoscopic image of a dysplastic area of Barrett’s esophagus. Red box is expert annotation. Green box is prediction by artificial intelligence algorithm.

      CNN architecture

      We developed and designed a CNN model architecture from 2 primary modules: feature extraction and classification. The base module of our algorithm is responsible for automated feature extraction and borrows from the Inception-ResNet-v2 algorithm developed by Google AI (Mountain View, Calif, USA) that achieves state-of-the-art accuracy on popular image classification benchmarks. The head module of our algorithm is designed for transforming extracted features from the base layers (50 mm + parameters) into a graded scale that allows for pathologic classification. The sigmoid activation function maps the model’s output logits into a float value ranging between 0 and 1. An Nvidia Gtx 1070 GPU (Nvidia, Santa Clara, Calif, USA) was used for algorithm development. Running inference on a frame requires 13 ms (77 fps), essentially real time.

      CNN training and validation

      CNN training and validation proceeded in 3 stages: preprocessing, training, and inference. In the preprocessing stage, data were formatted for input in the algorithm by resizing images, normalizing pixels, and then using transfer learning to help initiate the base layer imaging weights. The base layer imaging weights were pretrained on ImageNet, which is a large visual database designed for object recognition software. In the training stage, the CNN was trained with Tensorflow, an open-source machine learning framework. Data augmentation provided standardization for training data on the fly, including various image adjustments. To prevent overfitting, we used various regularization techniques including dropout, weight decay (L2 regularization), and data augmentation, that is, horizontal/vertical flips, rotations, scaling, color augmentations, Gaussian noise, and so on. Additionally, to prevent data leakage, we carefully split the training and validation sets such that each set only contained unique images from different patients. This ensured the algorithm metrics were validated on patients who were not part of the training set. In the inference stage, the data produced by the training were used to generate test set predictions. Various augmentations to the test set generated an additional layer of predictions. Predictions were then averaged to create a composite probability and then fed into an Adam optimizer to produce a binary result between 0 to .5 and .5 to 1 for final classification data. Any value ≥.5 was classified as the presence of dysplasia and <.5 as no dysplasia.

      Algorithm design

      Our algorithm was composed of 2 steps. The first was a binary classification assessing the presence of any neoplastic lesion and/or area on the image. If the binary classification classified the image as containing neoplasia, the second step was object detection (ie, localization of the lesion). For our 2-stage strategy, we used CNN based on Xception architecture for binary classification to quickly flag frames of interest and YOLO v2
      • Redmon J.
      • Farhadi A.
      YOLO9000: better, faster, stronger.
      for detection and localization of neoplasia on positively identified frames from the binary classifier. The algorithm was designed to draw a predicted rectangular bounding box over the area of neoplasia (Figs. 1 and 2).
      Figure thumbnail gr2
      Figure 2Illustration of the deep learning system. CNN, Convolutional neural network; NMS, non-maximum suppression.

      Internal validation

      Using the algorithm developed by training, we performed internal validation experiments on 458 test images (Supplementary Fig. 1, available online at www.giejournal.org). For binary classification the sensitivity, specificity, and accuracy were calculated per image, per patient, and by WLI, NBI, standard focus, and near focus.
      For localization accuracy, we used mean average precision (mAP), a standard metric used in AI. Average precision serves to determine the area under the precision recall curve and carries a value between 0 and 1. To calculate mAP, we need to use the intersection of union, which measures the overlap between prediction and ground truth. We predefined an intersection over union threshold at .3 to classify whether the box prediction by the algorithm was a true positive or a false positive. An intersection over union threshold of .3 was chosen because it was believed that in real-time clinical endoscopy this level of lesion targeting was more than sufficient to aid in the clinical identification of a region of neoplasia during a screening or surveillance BE examination (Fig. 3).
      Figure thumbnail gr3
      Figure 3Endoscopic images with varying intersections of union (IoUs): A, IoU .3. B, IoU .5. C, IoU .8. In this study, we set the IoU threshold as ≥.3 for true positive on object detection.

      Statistical analysis

      All statistical analyses were performed using R software (version 3.3.3; The R Foundation for Statistical Computing, Vienna, Austria). Categorical data were compared with the χ2 and t tests and continuous variables with the Mann-Whitney U test. Differences between variables with P < .05 were considered significant.

      Ethics

      This study was approved by the Institutional Review Board of the University of California Irvine Medical Center (UCI IRB HS no. 2008-6258).

      Results

      Binary classification per image

      A total of 458 images, 225 images of dysplastic lesions from 26 patients and 233 images of nondysplastic BE from 13 patients, were used for validation. For binary classification (dysplasia vs nondysplasia) sensitivity, specificity, and accuracy per image were 96.4%, 94.2%, and 95.4%, respectively (Fig. 4, Table 1). The sensitivity per image for WLI only was 98.6% and for NBI was 92.4%. The sensitivity per image for standard focus images only was 96.6% and for near focus was 96.2%. NBI specificity (99.2%) was higher than WLI specificity (88.8%) (P = .0007). Near focus specificity (98.4%) was higher than standard focus (89.9%, P = .005).
      Figure thumbnail gr4
      Figure 4A 2 × 2 table showing results of the artificial intelligence algorithm performance of binary classification of dysplasia versus no dysplasia per image.
      Table 1Results of AI binary diagnosis: per-image analysis
      Sensitivity (%)P valueSpecificity (%)P value
      AI diagnosis by white-light imaging98.6 (144/146).02388.8 (95/107).0007
      AI diagnosis by narrow-band imaging92.4 (73/79)99.2 (125/126)
      AI diagnosis by standard focus96.6 (141/146).8989.9 (98/109).005
      AI diagnosis by near focus96.2 (76/79)98.4 (122/124)
      Comprehensive AI diagnosis96.4 (217/225)94.2 (220/233)
      Values in parentheses are n/N. A total of 458 images (dysplastic 225, nondysplastic 233) were used for validation. The number of dysplastic images of white-light imaging, narrow-band imaging, standard focus, and near focus was 146, 79, 146, and 79 and nondysplastic images 107, 126, 109, and 124, respectively. AI, Artificial intelligence.

      Binary classification per patient

      The CNN correctly diagnosed 24 of 26 patients (92.3%) as having dysplasia. The sensitivities of dysplasia diagnosis per patient by WLI, NBI, false standard focus, and near focus were 94.7%, 91.7%, 95.2%, and 91.7%, respectively (Table 2).
      Table 2Results of AI binary diagnosis: per-patient analysis
      Sensitivity (%)P value
      AI diagnosis by white-light imaging94.7 (18/19).74
      AI diagnosis by narrow-band imaging91.7 (11/12)
      AI diagnosis by standard focus95.2 (20/21).68
      AI diagnosis by near focus91.7 (11/12)
      Comprehensive AI diagnosis92.3 (24/26)
      Values in parentheses are n/N. Validation set includes images of dysplasia from 26 patients in total. Images of white-light imaging, narrow-band imaging, standard focus, and near focus came from 19, 12, 21, and 12 patients, respectively. AI, Artificial intelligence.

      False-negative and -positive lesions on binary classification

      Eight of 225 dysplastic images (3.6%) with 4 patients showed a false negative in the binary classification. Six of 8 false-negative images were from 2 patients but had another true-positive image and therefore did not affect per-patient sensitivity. Two of 8 false-negative images from 2 patients had no true-positive images. All false negatives were small subtle lesions less than 5 mm in size (Fig. 5). For most false-positive lesions, it appears that the algorithm detected an area of nodularity and/or raised tissue as dysplastic.
      Figure thumbnail gr5
      Figure 5Two cases with negative binary result. Both lesions showed high-grade dysplasia in histopathologic examination. A, A 4-mm nodular lesion from buried Barrett’s esophagus. B, A 4-mm slightly depressed lesion.

      Object detection

      In our validation set with an intersection of union at .3, overall mAP was .7533. mAP for NBI only was higher at .802, and mAP for near focus images was .819. The overall sensitivity (ie, recall) was 95.6%. The sensitivity for WLI was 94.1%, NBI 98.2%, standard focus 96.8%, and near focus 97.5%. The overall positive predictive value (ie, precision) was 89.2%. The positive predictive value for WLI was 87.8%, NBI 92.1%, standard focus 86.3%, and near focus 96.3% (Table 3).
      Table 3Results of AI object detection for binary positive image: per-image analysis
      Sensitivity (%)P valuePositive predictive value (%)P value
      AI diagnosis by white-light imaging94.1 (318/338).01887.8 (318/362).14
      AI diagnosis by narrow-band imaging98.2 (164/166)92.1 (164/178)
      AI diagnosis by standard focus96.8 (328/346).2486.3 (328/380).0003
      AI diagnosis by near focus97.5 (154/158)96.3 (154/160)
      Comprehensive AI diagnosis95.6 (482/504)89.2 (482/540)
      Values in parentheses are n/N. AI, Artificial intelligence.

      Speed

      On GPU gtx1070 setting, the binary classifier runs at around 72 frames per s (299x299 pixel image size). The YOLO v2 runs at around 45 frames per s (416 x 416p x image size). Video 1 (available online at www.giejournal.org) demonstrates how this algorithm could work in real time.

      Discussion

      Our pilot study for detection of early esophageal neoplasm in BE using CNN demonstrated a high sensitivity (96.4%), specificity (94.2%), and accuracy (95.4%) per image for binary classification (dysplasia vs no dysplasia) on an internal validation study. Our localization algorithm also detected most lesions correctly with a mAP of .7533, sensitivity of 95.6%, and positive predictive value of 89.2%.
      DL for the detection of lesions in the GI tract is rapidly emerging.
      • Horie Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
      • Cai S.L.
      • Li B.
      • Tan W.M.
      • et al.
      Using deep learning system in endoscopy for screening of early esophageal squamous cell carcinoma (with video).
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Computer-aided diagnosis using deep learning in the evaluation of early oesophageal adenocarcinoma.
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Real-time use of artificial intelligence in the evaluation of cancer in Barrett's oesophagus.
      • Hirasawa T.
      • Aoyama K.
      • Tanimoto T.
      • et al.
      Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images.
      • Kanesaka T.
      • Lee T.C.
      • Uedo N.
      • et al.
      Computer-aided diagnosis for identifying and delineating early gastric cancers in magnifying narrow-band imaging.
      • Zhang X.
      • Chen F.
      • Yu T.
      • et al.
      Real-time gastric polyp detection using convolutional neural networks.
      • Li L.
      • Chen Y.
      • Shen Z.
      • et al.
      Convolutional neural network for the diagnosis of early gastric cancer based on magnifying narrow band imaging.
      • Aoki T.
      • Yamada A.
      • Aoyama K.
      • et al.
      Automatic detection of erosions and ulcerations in wireless capsule endoscopy images based on a deep convolutional neural network.
      • Urban G.
      • Tripathi P.
      • Alkayali T.
      • et al.
      Deep learning localizes and identifies polyps in real time with 96% accuracy in screening colonoscopy.
      • Misawa M.
      • Kudo S.E.
      • Mori Y.
      • et al.
      Artificial intelligence-assisted polyp detection for colonoscopy: initial experience.
      • Klare P.
      • Sander C.
      • Prinzen M.
      • et al.
      Automated polyp detection in the colorectum: a prospective study (with videos).
      • Wang P.
      • Xiao X.
      • Brown J.R.
      • et al.
      Development and validation of a deep-learning algorithm for the detection of polyps during colonoscopy.
      • Redmon J.
      • Farhadi A.
      YOLO9000: better, faster, stronger.
      • van der Sommen F.
      • Zinger S.
      • Curvers W.L.
      • et al.
      Computer-aided detection of early neoplastic lesions in Barrett's esophagus.
      • de Groof J.
      • van der Sommen F.
      • van der Putten J.
      • et al.
      The Argos project: the development of a computer-aided detection system to improve detection of Barrett's neoplasia on white light endoscopy.
      • Swager A.F.
      • van der Sommen F.
      • Klomp S.R.
      • et al.
      Computer-aided detection of early Barrett's neoplasia using volumetric laser endomicroscopy.
      • Fonollà R.
      • Scheeve T.
      • Struyvenberg M.R.
      • et al.
      Ensemble of deep convolutional neural networks for classification of early barrett's neoplasia using volumetric laser endomicroscopy.
      However, there still remains limited literature that focuses on early esophageal neoplasia in BE. de Groof et al
      • de Groof J.
      • van der Sommen F.
      • van der Putten J.
      • et al.
      The Argos project: the development of a computer-aided detection system to improve detection of Barrett's neoplasia on white light endoscopy.
      reported a study of automated detection of early esophageal neoplasia in BE using machine learning with a hand-crafted algorithm. They used 100 endoscopic images from 44 patients with a sensitivity of 83% and a specificity of 83%.
      • van der Sommen F.
      • Zinger S.
      • Curvers W.L.
      • et al.
      Computer-aided detection of early neoplastic lesions in Barrett's esophagus.
      They also published a prospective pilot study of this algorithm with promising data. However, this algorithm was not fast enough to be used in real time. Also reported was the application of AI for volumetric laser endomicroscopy, resulting in a sensitivity and specificity of 90% and 93%.
      • Swager A.F.
      • van der Sommen F.
      • Klomp S.R.
      • et al.
      Computer-aided detection of early Barrett's neoplasia using volumetric laser endomicroscopy.
      ,
      • Fonollà R.
      • Scheeve T.
      • Struyvenberg M.R.
      • et al.
      Ensemble of deep convolutional neural networks for classification of early barrett's neoplasia using volumetric laser endomicroscopy.
      However, volumetric laser endomicroscopy requires time and equipment. A Japanese group recently reported a pilot study using DL with CNN on detection mainly of early squamous cell carcinoma with a sensitivity of 98%.
      • Horie Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
      Ebigbo et al
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Computer-aided diagnosis using deep learning in the evaluation of early oesophageal adenocarcinoma.
      used computer-aided diagnosis in the evaluation of early esophageal adenocarcinoma in BE. This system showed sensitivity and specificity of 97% and 88% and was trained on only 148 images, significantly less than our training set. In a follow-up study piloting this system in real time, the same group was able to assess the system in 14 patients with promising results, and their system was trained on 129 images.
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Real-time use of artificial intelligence in the evaluation of cancer in Barrett's oesophagus.
      The speed of the algorithm prediction required pausing the live video display to make a prediction. Our current prototype as seen in Video 1 does not require pausing and also does not require a second monitor to display the user interface as in the system shown in their publication.
      To improve the impact of endoscopic surveillance of BE, a real-time AI video overlay assisting an endoscopist detect neoplastic lesions is certainly desirable. Our group has already achieved a real-time video overlay with colonoscopy algorithms including polyp detection and optical pathology and believe this is certainly feasible.
      • Urban G.
      • Tripathi P.
      • Alkayali T.
      • et al.
      Deep learning localizes and identifies polyps in real time with 96% accuracy in screening colonoscopy.
      The advantage of real-time AI over other technologies aimed to improve dysplasia detection is that the learning curve is negligible. Ideally, the system would simply flag areas of concern for the endoscopist to interrogate further with more detailed imaging or a biopsy sampling. An algorithm, well trained by expert endoscopists, has the potential to elevate the image interpretation skill set of an endoscopist to a level near that of an imaging expert. If an AI algorithm were able to achieve American Society for Gastrointestinal Endoscopy Preservation and Incorporation of Valuable Endoscopic Innovations threshold for the detection of early esophageal neoplasia, it could decrease the number of unnecessary biopsy samplings performed during surveillance examinations.
      There were performance differences on binary classification with specific imaging modalities. NBI showed better sensitivity and specificity than WLI, and near focus showed better specificity than standard focus. Although the effect of NBI on detection of neoplasia is controversial,
      • Beg S.
      • Mensa M.
      • Fullard M.
      • et al.
      Impact of advanced endoscopic imaging on Barrett's esophagus in daily clinical practice.
      ,
      • Xiong Y.Q.
      • Ma S.J.
      • Hu H.Y.
      • et al.
      Comparison of narrow-band imaging and confocal laser endomicroscopy for the detection of neoplasia in Barrett's esophagus: a meta-analysis.
      our study suggests that NBI and near focus may alter tissue imaging characteristics and detail in such a way that subtle imaging differences between dysplastic and nondysplastic tissue are enhanced and are better differentiated by the algorithm.
      Because overfitting and spectrum bias can cause overestimation of accuracy, we were extremely careful in selecting a highly diverse group of images from each patient. Furthermore, images of specific patients used in the training set were not used in the validation set. Rectangular polygons for annotation and prediction were chosen to minimize processing power such that the algorithm can make predications at >40 frames per s and real-time evaluation is possible (as evidenced in Video 1). With more complex polygons with independent corners and/or vertices there becomes a significantly increased level of processing power needed, thereby slowing down the frame rate for prediction and diminishing the utility in real time.
      Performance detection depends on the neural network architecture used. Ghatwary et al
      • Ghatwary N.
      • Zolgharni M.
      • Ye X.
      Early esophageal adenocarcinoma detection using deep learning methods.
      compared several DL object detection methods to identify esophageal adenocarcinoma using 100 WL images from 39 patients and concluded that single-shot multibox detector methods outperformed regional-based CNN, fast regional-based CNN, and faster regional-based CNN with a sensitivity of .90, specificity of .88, precision of .70, and recall of .79.
      • Ghatwary N.
      • Zolgharni M.
      • Ye X.
      Early esophageal adenocarcinoma detection using deep learning methods.
      We chose to use YOLO v2, which is similar to the single-shot multibox detector but has a faster frames per second prediction to use this algorithm in real time.
      Our study has limitations. First, this study was a single-center retrospective study and lacked external validation. That is, although this algorithm worked very well on our validation set of images, it was not tested on endoscopic images from other centers or other scope manufacturers, and this is a future goal for this project. Second, the number of early esophageal neoplasia in BE was not large.
      In summary, this is among the first studies in which AI using CNNs was examined for detection of early esophageal neoplasm in BE. Although this is a pilot study, the results were very promising. The speed of the algorithm is fast enough to enable true real-time detection, and our results suggest a real-time AI detection system for dysplasia in BE can likely be achieved in the near future.

      Appendix

      Figure thumbnail fx1
      Supplementary Figure 1Study flow of this study. CNN, Convolutional neural network.

      Supplementary data

      References

        • Arnold M.
        • Soerjomataram I.
        • Ferlay J.
        • et al.
        Global incidence of oesophageal cancer by histological subtype in 2012.
        Gut. 2015; 64: 381-387
        • Hur C.
        • Miller M.
        • Kong C.Y.
        • et al.
        Trends in esophageal adenocarcinoma incidence and mortality.
        Cancer. 2013; 119: 1149-1158
        • Thrift A.P.
        The epidemic of oesophageal carcinoma: Where are we now?.
        Cancer Epidemiol. 2016; 41: 88-95
        • Curvers W.L.
        • Ten Kate F.J.
        • Krishnadath K.K.
        • et al.
        Low-grade dysplasia in Barrett’s esophagus: overdiagnosed and underestimated.
        Am J Gastroenterol. 2010; 105: 1523-1530
        • Rastogi A.
        • Puli S.
        • El-Serag H.B.
        • et al.
        Incidence of esophageal adenocarcinoma in patients with Barrett’s esophagus and high-grade dysplasia: a meta-analysis.
        Gastrointest Endosc. 2008; 67: 394-398
        • Shaheen N.J.
        • Falk G.W.
        • Iyer P.G.
        • et al.
        ACG clinical guideline: diagnosis and management of Barrett’s esophagus.
        Am J Gastroenterol. 2016; 111: 30-50
        • Fitzgerald R.C.
        • di Pietro M.
        • Ragunath K.
        • et al.
        British Society of Gastroenterology guidelines on the diagnosis and management of Barrett’s oesophagus.
        Gut. 2014; 63: 7-42
        • Evans J.A.
        • Early D.S.
        • Fukami N.
        • et al.
        The role of endoscopy in Barrett’s esophagus and other premalignant conditions of the esophagus.
        Gastrointest Endosc. 2012; 76: 1087-1094
        • Codipilly D.C.
        • Chandar A.K.
        • Singh S.
        • et al.
        The effect of endoscopic surveillance in patients with Barrett's esophagus: a systematic review and meta-analysis.
        Gastroenterology. 2018; 154: 2068-2086
        • Sharma P.
        • Savides T.J.
        • Canto M.I.
        • et al.
        The American Society for Gastrointestinal Endoscopy PIVI (Preservation and Incorporation of Valuable Endoscopic Innovations) on imaging in Barrett's Esophagus.
        Gastrointest Endosc. 2012; 76: 252-254
        • Tavakkoli A.
        • Appelman H.D.
        • Beer D.G.
        • et al.
        Use of appropriate surveillance for patients with nondysplastic Barrett's esophagus.
        Clin Gastroenterol Hepatol. 2018; 16: 862-869
        • Thosani N.
        • Abu Dayyeh B.K.
        • et al.
        • ASGE Technology Committee
        ASGE Technology Committee systematic review and meta-analysis assessing the ASGE Preservation and Incorporation of Valuable Endoscopic Innovations thresholds for adopting real-time imaging-assisted endoscopic targeted biopsy during endoscopic surveillance of Barrett's esophagus.
        Gastrointest Endosc. 2016; 83: 684-698
        • Sami S.S.
        • Iyer P.G.
        Recent advances in screening for Barrett's esophagus.
        Curr Treat Options Gastroenterol. 2018; 16: 1-14
        • Horie Y.
        • Yoshio T.
        • Aoyama K.
        • et al.
        Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
        Gastrointest Endosc. 2019; 89: 25-32
        • Cai S.L.
        • Li B.
        • Tan W.M.
        • et al.
        Using deep learning system in endoscopy for screening of early esophageal squamous cell carcinoma (with video).
        Gastrointest Endosc. 2019; 90: 745-753
        • Ebigbo A.
        • Mendel R.
        • Probst A.
        • et al.
        Computer-aided diagnosis using deep learning in the evaluation of early oesophageal adenocarcinoma.
        Gut. 2019; 68: 1143-1145
        • Ebigbo A.
        • Mendel R.
        • Probst A.
        • et al.
        Real-time use of artificial intelligence in the evaluation of cancer in Barrett's oesophagus.
        Gut. 2020; 69: 615-616
        • Hirasawa T.
        • Aoyama K.
        • Tanimoto T.
        • et al.
        Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images.
        Gastric Cancer. 2018; 21: 653-660
        • Kanesaka T.
        • Lee T.C.
        • Uedo N.
        • et al.
        Computer-aided diagnosis for identifying and delineating early gastric cancers in magnifying narrow-band imaging.
        Gastrointest Endosc. 2018; 87: 1339-1344
        • Zhang X.
        • Chen F.
        • Yu T.
        • et al.
        Real-time gastric polyp detection using convolutional neural networks.
        PLoS One. 2019; 14e0214133
        • Li L.
        • Chen Y.
        • Shen Z.
        • et al.
        Convolutional neural network for the diagnosis of early gastric cancer based on magnifying narrow band imaging.
        Gastric Cancer. 2020; 23: 126-132
        • Aoki T.
        • Yamada A.
        • Aoyama K.
        • et al.
        Automatic detection of erosions and ulcerations in wireless capsule endoscopy images based on a deep convolutional neural network.
        Gastrointest Endosc. 2019; 89: 357-363
        • Urban G.
        • Tripathi P.
        • Alkayali T.
        • et al.
        Deep learning localizes and identifies polyps in real time with 96% accuracy in screening colonoscopy.
        Gastroenterology. 2018; 155: 1069-1078
        • Misawa M.
        • Kudo S.E.
        • Mori Y.
        • et al.
        Artificial intelligence-assisted polyp detection for colonoscopy: initial experience.
        Gastroenterology. 2018; 154: 2027-2029
        • Klare P.
        • Sander C.
        • Prinzen M.
        • et al.
        Automated polyp detection in the colorectum: a prospective study (with videos).
        Gastrointest Endosc. 2019; 89: 576-582
        • Wang P.
        • Xiao X.
        • Brown J.R.
        • et al.
        Development and validation of a deep-learning algorithm for the detection of polyps during colonoscopy.
        Nat Biomed Eng. 2018; 2: 741
        • Redmon J.
        • Farhadi A.
        YOLO9000: better, faster, stronger.
        The IEEE Conference on Computer Vision and Pattern Recognition. 2017; (Available at: https://arxiv.org/abs/1612.08242. Accessed November 22, 2019): 7263-7271
        • van der Sommen F.
        • Zinger S.
        • Curvers W.L.
        • et al.
        Computer-aided detection of early neoplastic lesions in Barrett's esophagus.
        Endoscopy. 2016; 48: 617-624
        • de Groof J.
        • van der Sommen F.
        • van der Putten J.
        • et al.
        The Argos project: the development of a computer-aided detection system to improve detection of Barrett's neoplasia on white light endoscopy.
        United Eur Gastroenterol J. 2019; 7: 538-547
        • Swager A.F.
        • van der Sommen F.
        • Klomp S.R.
        • et al.
        Computer-aided detection of early Barrett's neoplasia using volumetric laser endomicroscopy.
        Gastrointest Endosc. 2017; 86: 839-846
        • Fonollà R.
        • Scheeve T.
        • Struyvenberg M.R.
        • et al.
        Ensemble of deep convolutional neural networks for classification of early barrett's neoplasia using volumetric laser endomicroscopy.
        Appl Sci. 2019; 9: 2183
        • Beg S.
        • Mensa M.
        • Fullard M.
        • et al.
        Impact of advanced endoscopic imaging on Barrett's esophagus in daily clinical practice.
        Gastrointest Endosc. 2018; 87: 1189-1194
        • Xiong Y.Q.
        • Ma S.J.
        • Hu H.Y.
        • et al.
        Comparison of narrow-band imaging and confocal laser endomicroscopy for the detection of neoplasia in Barrett's esophagus: a meta-analysis.
        Clin Res Hepatol Gastroenterol. 2018; 42: 31-39
        • Ghatwary N.
        • Zolgharni M.
        • Ye X.
        Early esophageal adenocarcinoma detection using deep learning methods.
        Int J Comput Assist Radiol Surg. 2019; 14: 611-621