Advertisement

Differential diagnosis for esophageal protruded lesions using a deep convolution neural network in endoscopic images

  • Author Footnotes
    ∗ Drs Zhang, Zhu, Wang, Kong and Hua contributed equally to this article.
    Min Zhang
    Footnotes
    ∗ Drs Zhang, Zhu, Wang, Kong and Hua contributed equally to this article.
    Affiliations
    Department of Gastroenterology, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China
    Search for articles by this author
  • Author Footnotes
    ∗ Drs Zhang, Zhu, Wang, Kong and Hua contributed equally to this article.
    Chang Zhu
    Footnotes
    ∗ Drs Zhang, Zhu, Wang, Kong and Hua contributed equally to this article.
    Affiliations
    Department of Gastroenterology, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China
    Search for articles by this author
  • Author Footnotes
    ∗ Drs Zhang, Zhu, Wang, Kong and Hua contributed equally to this article.
    Yun Wang
    Footnotes
    ∗ Drs Zhang, Zhu, Wang, Kong and Hua contributed equally to this article.
    Affiliations
    Department of Gastroenterology, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China
    Search for articles by this author
  • Author Footnotes
    ∗ Drs Zhang, Zhu, Wang, Kong and Hua contributed equally to this article.
    Zihao Kong
    Footnotes
    ∗ Drs Zhang, Zhu, Wang, Kong and Hua contributed equally to this article.
    Affiliations
    Department of Gastroenterology, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China
    Search for articles by this author
  • Author Footnotes
    ∗ Drs Zhang, Zhu, Wang, Kong and Hua contributed equally to this article.
    Yifei Hua
    Footnotes
    ∗ Drs Zhang, Zhu, Wang, Kong and Hua contributed equally to this article.
    Affiliations
    Department of Gastroenterology, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China
    Search for articles by this author
  • Weifeng Zhang
    Affiliations
    Department of Gastroenterology, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China
    Search for articles by this author
  • Xinmin Si
    Affiliations
    Department of Gastroenterology, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China
    Search for articles by this author
  • Bixing Ye
    Affiliations
    Department of Gastroenterology, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China
    Search for articles by this author
  • Xiaobing Xu
    Affiliations
    Department of Gastroenterology, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China
    Search for articles by this author
  • Lurong Li
    Affiliations
    Department of Gastroenterology, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China
    Search for articles by this author
  • Ding Heng
    Affiliations
    Department of Gastroenterology, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China
    Search for articles by this author
  • Baiyun Liu
    Affiliations
    Infervision, Beijing, China
    Search for articles by this author
  • Song Tian
    Affiliations
    Infervision, Beijing, China
    Search for articles by this author
  • Jiangfen Wu
    Affiliations
    Infervision, Beijing, China
    Search for articles by this author
  • Yini Dang
    Correspondence
    Reprint requests: Guoxin Zhang or Yini Dang, Department of Gastroenterology, The First Affiliated Hospital of Nanjing Medical University, 300 Guang-Zhou Road, Nanjing 210029, China.
    Affiliations
    Department of Gastroenterology, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China
    Search for articles by this author
  • Guoxin Zhang
    Correspondence
    Reprint requests: Guoxin Zhang or Yini Dang, Department of Gastroenterology, The First Affiliated Hospital of Nanjing Medical University, 300 Guang-Zhou Road, Nanjing 210029, China.
    Affiliations
    Department of Gastroenterology, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China
    Search for articles by this author
  • Author Footnotes
    ∗ Drs Zhang, Zhu, Wang, Kong and Hua contributed equally to this article.
Open AccessPublished:October 12, 2020DOI:https://doi.org/10.1016/j.gie.2020.10.005

      Background and Aims

      Recent advances in deep convolutional neural networks (CNNs) have led to remarkable results in digestive endoscopy. In this study, we aimed to develop CNN-based models for the differential diagnosis of benign esophageal protruded lesions using endoscopic images acquired during real clinical settings.

      Methods

      We retrospectively reviewed the images from 1217 patients who underwent white-light endoscopy (WLE) and EUS between January 2015 and April 2020. Three deep CNN models were developed to accomplish the following tasks: (1) identification of esophageal benign lesions from healthy controls using WLE images; (2) differentiation of 3 subtypes of esophageal protruded lesions (including esophageal leiomyoma [EL], esophageal cyst (EC], and esophageal papilloma [EP]) using WLE images; and (3) discrimination between EL and EC using EUS images. Six endoscopists blinded to the patients’ clinical status were enrolled to interpret all images independently. Their diagnostic performances were evaluated and compared with the CNN models using the area under the receiver operating characteristic curve (AUC).

      Results

      For task 1, the CNN model achieved an AUC of 0.751 (95% confidence interval [CI], 0.652-0.850) in identifying benign esophageal lesions. For task 2, the proposed model using WLE images for differentiation of esophageal protruded lesions achieved an AUC of 0.907 (95% CI, 0.835-0.979), 0.897 (95% CI, 0.841-0.953), and 0.868 (95% CI, 0.769-0.968) for EP, EL, and EC, respectively. The CNN model achieved equivalent or higher identification accuracy for EL and EC compared with skilled endoscopists. In the task of discriminating EL from EC (task 3), the proposed CNN model had AUC values of 0.739 (EL, 95% CI, 0.600-0.878) and 0.724 (EC, 95% CI, 0.567-0.881), which outperformed seniors and novices. Attempts to combine the CNN and endoscopist predictions led to significantly improved diagnostic accuracy compared with endoscopists interpretations alone.

      Conclusions

      Our team established CNN-based methodologies to recognize benign esophageal protruded lesions using routinely obtained WLE and EUS images. Preliminary results combining the results from the models and the endoscopists underscored the potential of ensemble models for improved differentiation of lesions in real endoscopic settings.

      Abbreviations:

      AI (artificial intelligence), AUC (area under curve), CI (confidence interval), CNN (convolutional neural network), EC (esophageal cyst), EL (esophageal leiomyoma), EP (esophageal papilloma), ROC (receiver operating characteristic), WLE (white-light endoscopy)

      Introduction

      Protruded lesions that derive from the esophagus may account for approximately 30% of upper GI lesions.
      • Hoda K.M.
      • Rodriguez S.A.
      • Faigel D.O.
      EUS-guided sampling of suspected GI stromal tumors.
      These lesions can often be found with white-light endoscopy imaging (WLE) and may arise from any layer along the wall, from deep mucosa to the muscle or the serosa.
      • Baysal B.
      • Masri O.A.
      • Eloubeidi M.A.
      • et al.
      The role of EUS and EUS-guided FNA in the management of subepithelial lesions of the esophagus: a large, single-center experience.
      In our clinical practice, increasing interest has been paid to rapid and accurate recognition of these esophageal protruded lesions. Clinically, benign protruded lesions can be classified into several subtypes based on their histologic entities, including esophageal leiomyoma (EL), esophageal papilloma (EP), and esophageal cyst (EC). Most esophageal protruded lesions were visually distinct with WLE imaging, whereas some may appear similar making it difficult to assess their characteristics using WLE alone. Recent advances in EUS imaging have become available to reveal fine lesion structures, such as size, margins, vascularity, layer of origin, and specific echogenicity.
      • Mortensen M.B.
      • Fristrup C.
      • Holm F.S.
      • et al.
      Prospective evaluation of patient tolerability, satisfaction with patient information, and complications in endoscopic ultrasonography.
      • Buscarini E.
      • Stasi M.D.
      • Rossi S.
      • et al.
      Endosonographic diagnosis of submucosal upper gastrointestinal tract lesions and large fold gastropathies by catheter ultrasound probe.
      • Chak A.
      • Canto M.
      • Stevens P.D.
      • et al.
      Clinical applications of a new through-the-scope ultrasound probe: prospective comparison with an ultrasound endoscope.
      However, operating EUS and handling subsequent image interpretation are highly demanding and require endoscopists with extensive knowledge and years of experience.
      • Van Dam J.
      • Brady P.G.
      • Freeman M.
      • et al.
      Guidelines for training in electronic ultrasound: guidelines for clinical application. From the ASGE. American Society for Gastrointestinal Endoscopy.
      The differential diagnosis of esophageal protruded lesions can lead to misinterpretations and inconsistency among endoscopists
      • Park C.
      • Lee S.
      • Oh M.
      • et al.
      The analysis of endoscopic ultrasonographic finding for esophageal subepithelial lesion.
      • Jeon H.W.
      • Choi M.G.
      • Lim C.H.
      • et al.
      Intraoperative esophagoscopy provides accuracy and safety in video-assisted thoracoscopic enucleation of benign esophageal submucosal tumors.
      • Liu R.
      • Adler D.G.
      Duplication cysts: diagnosis, management, and the role of endoscopic ultrasound.
      and remains a challenging task for endoscopists at all levels of experience.
      Recent advances in artificial intelligence (AI) have shown its remarkable success in diagnostic imaging in various medical fields.
      • Mendelson E.B.
      Artificial intelligence in breast imaging: potentials and limitations.
      • Shkolyar E.
      • Jia X.
      • Chang T.C.
      • et al.
      augmented bladder tumor detection using deep learning.
      • Willemink M.J.
      • Noel P.B.
      The evolution of image reconstruction for CT-from filtered back projection to artificial intelligence.
      In endoscopy, several studies have demonstrated valuable applications of AI in endoscopic diagnosis and characterization, such as early detection of polyps and neoplasia, determination of the invasion depth of carcinomas,
      • Zhu Y.
      • Wang Q.C.
      • Xu M.D.
      • et al.
      Application of convolutional neural network in the diagnosis of the invasion depth of gastric cancer based on conventional endoscopy.
      and examination quality.
      • Zhou J.
      • Wu L.
      • Wan X.
      • et al.
      A novel artificial intelligence system for the assessment of bowel preparation (with video).
      In particular, deep learning algorithms with convolutional neural networks (CNNs) have accelerated the progress toward endoscopic image analysis and augmented clinician performance.
      • Aziz M.
      • Fatima R.
      • Dong C.
      • et al.
      The impact of deep convoluted neural network based artificial intelligence on colonoscopy outcomes: a systematic review with meta-analysis.
      ,
      • Wu L.
      • Zhou W.
      • Wan X.
      • et al.
      A deep neural network improves endoscopic detection of early gastric cancer without blind spots.
      Their proven merits include automated phenotypic characterization, quantitative image analysis, and definition of abstract features.
      • Wu L.
      • Zhou W.
      • Wan X.
      • et al.
      A deep neural network improves endoscopic detection of early gastric cancer without blind spots.
      • Jing Y.
      • Bian Y.
      • Hu Z.
      • et al.
      Deep learning for drug design: an artificial intelligence paradigm for drug discovery in the big data era.
      • Zhao L.
      • Jia K.
      Multiscale CNNs for brain tumor segmentation and diagnosis.
      • Jisu H.
      • Bo-Yong P.
      • Hyunjin P.
      Convolutional neural network classifier for distinguishing Barrett's esophagus and neoplasia endomicroscopy images.
      For example, Saito et al
      • Saito H.
      • Aoki T.
      • Aoyama K.
      • et al.
      Automatic detection and classification of protruding lesions in wireless capsule endoscopy images based on a deep convolutional neural network.
      developed a CNN model using wireless capsule endoscopy images for detection of small-bowel protruding lesions and achieved an area under the curve (AUC) of 0.911 (95% Cl, 0.9069-0.9155). Moreover, Urban et al
      • Urban G.
      • Tripathi P.
      • Alkayali T.
      • et al.
      Deep learning localizes and identifies polyps in real time with 96% accuracy in screening colonoscopy.
      reported a CNN model that successfully localized and distinguished polyps in screening colonoscopies with an accuracy up to 96%. Despite all the research reporting favorable performance in endoscopy, there has been no relevant study on differentiating subtypes of benign esophageal protruded lesions using a CNN model and assessment of its capacity in assisting endoscopists.
      Therefore, in this study we aimed to establish a methodological framework powered by deep learning for the differential diagnosis of esophageal protruded lesions on WLE and EUS images. Preliminary findings on combining predictions from the model and endoscopists showed the potential for improvement of diagnostic accuracy across all levels of endoscopists.

      Methods

      This study was approved by the Ethics Committee of the First Affiliated Hospital of Nanjing Medical University, Jiangsu Province Hospital (ethics approval number 2020-SR-201), and written informed consent was waived due to the retrospective nature of this study.

      Study design

      In this retrospective study, a total of 1217 patients who underwent the WLE (GIF-Q260J, Olympus, Tokyo, Japan; GIF-Q290J, Olympus, Japan) and EUS (P2620, FUJI, Tokyo, Japan) between January 2015 and April 2020 were reviewed. CNN models were introduced to detect and classify esophageal protruded lesions on WLE and EUS images automatically. The training procedure for the CNN models is shown in Figure 1.
      Figure thumbnail gr1
      Figure 1Flowchart of the study design. The study consisted of 3 sections as shown in the blue boxes. The deep convolutional neural network (CNN) procedure included images for preprocessing, training, testing, and comparison. EC, Esophageal cyst; EL, esophageal leiomyoma; EP, esophageal papilloma; WLI, white light endoscopy.
      To imitate realistic clinical practice procedures, our study was composed of 3 sections: (1) identification of esophageal benign lesions from healthy controls using WLE images; (2) distinguishing subtypes of esophageal protruded lesions using WLE images; (3) discrimination of EL and EC using EUS images.

      Sample size calculation

      On the basis of our primary outcome for identifying each subtype of esophageal protruded lesions, we designed the study with 80% power to find the difference between the minimal diagnostic rate and the proposed CNN model with a significance level (type I error) of 5% (2-sided). Based on the data from previous research and exploratory clinical experiments, the clinically acceptable diagnostic rate was set at 0.8 and 0.9 for the CNN models. Therefore, the sample size required for the single-group clinical target value calculated by the exact estimation method was 107 (calculated using G-power software). Taking into account the possibility of missing data, the sample size was increased by 10%, and 117 patients were finally included in each group. The same sets of patients were used for the initial testing of the CNN model trained on EUS images.

      Image annotation and datasets preparation

      Image datasets were originally obtained from endoscopic videos under white light. All datasets were deidentified and reviewed by 6 experienced endoscopists divided into 2 groups (3 endoscopists in each group) to interpret the WLE and EUS images. All endoscopists were asked to outline the boundaries of any esophageal protruded lesions present in the images, along with the disease label based on the whole images. These masks and image labels were subsequently reviewed by the most experienced endoscopists (Y.W. and G.X.Z.).
      Finally, the distribution of each section in the datasets were as follows. For section 1, there were 17,279 WLE examinations from 598 patients, comprising 3 groups, including normal (200 patients with 5536 graphs), esophagitis (200 patients with 6280 graphs), and esophageal protruded lesions (198 patients with 5463 graphs). Normal images were defined as negative cases (patients with no lesions). For section 2, there were 3226 WLE examinations from 619 patients who were diagnosed with subtypes of esophageal protruded lesions, including 161 patients with EP (829 graphs), 119 patients with EC (814 graphs), and 339 patients with EL (1583 graphs). For section 3, there were 3411 EUS examinations from 248 patients, including 85 patients with EC (1197 graphs), and 163 EL patients (2214 graphs).
      For all sections, patients included in the study had a definitive diagnosis and clear presence of lesions. Images from patients containing noise or obvious blurring, apparent mucus, foam, or food residues were excluded. For each section, disease classes were described as follows. For section 1, the diagnosis of esophagitis was confirmed by pathology records, and protruded lesions were based on WLE images and consensus readings by experienced endoscopists (Y.W. and G.X.Z, who did not participate in the readers’ experiment). For section 2, the diagnosis of EP and EL was confirmed by pathology records, whereas EC was verified based on WLE and EUS images and endoscopist readings in consensus. Section 3 was as described in section 2 for EL and EC.
      All datasets included images of the same lesion from multiple viewpoints or similar lesions from the same person. Each dataset was separated into training (including validation) and test cohorts in a4:1 ratio, respectively. Details of the distribution in the datasets are shown in Table 1.
      Table 1The details of sections 1, 2, and 3
      TotalTraining setsTest sets
      PatientsSeriesPatientsSeries
      Section 1
      Normal1604340401196
      Esophagitis1605036401244
      Protruded lesions1594350391113
      Total479137261193553
      Section 2
      EP12964232187
      EC9766722147
      EL272129067293
      Total4982599121627
      Section 3
      EC68103217165
      EL131178332431
      Total199281549596
      EC, Esophageal cyst; EP, esophageal papilloma; EL, esophageal leiomyoma.

      Development of the CNN algorithm

      Our proposed scheme for training the CNN models was based on the 3 sections. For different image modalities (WLE vs EUS), multicategory classification models were developed and validated to obtain accurate prediction outcomes (Figs. 2 and 3).
      Figure thumbnail gr2
      Figure 2The workflow and architecture of the convolutional neural network model. EC, Esophageal cyst; EL, esophageal leiomyoma; EP, esophageal papilloma; HardSwish Act, Hard Swish activation.
      Figure thumbnail gr3
      Figure 3The workflow of the inference algorithm and patient-level diagnosis. CNN, Convolutional neural network.

      Image preprocessing and data augmentation

      Image preprocessing is a mandatory step when building a dataset before model development. In this study, both WLE and EUS images were stored in JPG format with good image resolution. Random data augmentation techniques were used to improve the invariance of the model to noise, including combination of horizontal and vertical flipping, brightness, contrast, saturation shifting. Moreover, all WLE images were processed with hue transformation and resized to 512 × 512 pixels with OpenCV bilinear interpolation (version 4.1.0). All pixel values were normalized with mean of 0.485 (standard deviation, 0.22).

      Model training

      The training protocol of the model determined by the searching method is presented in the section on training details in Appendix 1 (available online at www.giejournal.org). Briefly, we adopted MobiLeNetv3 large
      • Howard A.
      • Sandler M.
      • Chu G.
      • et al.
      Searching for mobilenetv3.
      as the primary approach to classify each category (eg, EL vs EC vs EP) in WLE and EUS images. MobiLeNetv3 is a mature architecture that integrates several well-established CNN structures and convolution methods. This state-of-the-art model targets different image modalities and can recognize blurred frames with low resource use
      • Howard A.
      • Sandler M.
      • Chu G.
      • et al.
      Searching for mobilenetv3.
      by using a single labeled graph as model input, and the output is the probability of each given class. For multicategory classifications, different loss functions were applied in the model training to ensure the decent prediction outcomes. Specifically, channels in MobiLeNetv3 were multiplied by 0.75 to overcome overfitting in section 2; we then used weight cross entropy loss and sigmoid loss for classification purposes in sections 1 and 3 because similar accuracy was obtained. Subsequently, similar hyperparameters were applied to 3 training tasks. Batch size, weighted decay, and the initial learning rate were set to 16 per gaphics processing unit (GPU), 0.001, and 0.01, respectively. Model training and testing were done using Mxnet (version 1.6.0) and CUDA (version 10.0). GPUs were 4 NVIDIA GeForce RTX 2070.
      Weighted cross entropy loss=ilogipi,labeli


      Weighted sigmoid loss=ipipilabeli+wi(spftrelu(p)+relu(p))


      Model evaluation and statistics

      Model efficiency was evaluated with accuracy, sensitivity, specificity, positive predictive value (precision), F1 score and g mean. The receiver operating characteristic (ROC) curve was used to compare the specificity and sensitivity of readers and models. The point closest to point (0, 1) represented good classification performance. For ROC curves, AUC values and 95% confidence intervals (CIs)
      • Cortes C.
      • Mohri M.
      Confidence intervals for the area under the ROC curve.
      were also calculated. Heatmaps were generated using Grad-CAM
      • Selvaraju R.R.
      • Cogswell M.
      • Das A.
      • et al.
      Grad-CAM: visual explanations from deep networks via gradient-based localization.
      to localize the pixel-level abnormality scores predicted by the model. The pixels that appeared to be red (hot) represented greater gradient and prediction scores.

      Observation experiments

      Six endoscopists (different from the endoscopists who annotated the images) participated in this study in 3 groups: experts (readers 1 and 2, abundant EUS experience with over 5000 gastroscopies); seniors (readers 3 and 4, basic EUS experience with over 1000 gastroscopies); and novices (readers 5 and 6, inadequate EUS experience with fewer than 1000 gastroscopies).
      In the test sets in sections 2 and 3, each endoscopist who was blinded to the patient’s clinical status independently evaluated the digital WLE and EUS images. Endoscopists were asked to classify the subtypes of esophageal protruded lesions in the images in the test dataset and to assign a confidence score (0%-100%) for each class. The sum of the scores should equal to 100%: eg, EL, 70%; EP, 20%, EC, 10%.
      The Grit-S
      • Duckworth A.L.
      • Quinn P.D.
      Development and validation of the short grit scale (grit-s).
      survey was also conducted based on a questionnaire, which comprised 8 items scored on a 5-point scale (from 1 to 5). A higher Grit score can be associated with greater will-power based on personal characteristics.

      Results

      Clinical characteristics in the study

      The clinical characteristics of the enrolled patients are listed in Table 2. We included 3226 WLE graphs from 613 patients in section 2: 339 patients with EL (mean age, 54.5 ± 10.1 years; 55% were male), 161 patients with EP (mean age, 52.3 ± 15.2 years; 39% were male). and 115 patients with EC (mean age, 59.8 ± 12.4 years; 57% were male). The average lesion sizes for EL, EP, and EC were 0.87 ± 0.68 cm, 0.41 ± 0.17 cm, and 0.68 ± 0.41 cm, respectively. Lesions in EP were generally smaller in size than that in EL and EC (P < .05). Section 3 comprised 3411 EUS images from 248 patients, including 163 patients with EL (mean age, 54.1 ± 10.1 years; 59% were male) and 85 age-matched patients with EC (mean age, 60.2 ± 11.2 years; 47% were male). The distribution of the lesions in each group was as follows: upper-third of the esophagus (144 of 339 in EL, 46 of 119 in EC), middle-third of the esophagus (104 of 339 in EL, 26 of 119 in EC), and lower-third of the esophagus (91 of 339 in EL and 47 of 119 in EC).
      Table 2Characteristics of the patients
      ELEPEC
      White-light endoscopyn = 339n = 161n = 119
      Age (years), mean ± SD54.47 ± 10.0852.29 ± 15.1859.76 ± 12.36
      Sex
       Male1866265
       Female1539954
      Location
       Upper third of the esophagus1445346
       Middle third of the esophagus1045726
       Lower third of the esophagus915147
      Lesion size (cm), mean ± SD0.87 ± 0.680.41 ± 0.170.68 ± 0.41
      EUSn = 163n = 85
      Age (years), mean ± SD54.12±10.12NA60.15±11.17
      Sex
       Male96NA45
       Female67NA40
      Location
       Upper third of the esophagus79NA26
       Middle third of the esophagus46NA19
       Lower third of the esophagus38NA40
      Lesion size (cm), mean ± SD0.88 ± 0.69NA0.65 ± 0.36
      EL, Esophageal leiomyoma; EP, esophageal papilloma; EC, esophageal cyst; SD, standard deviation; NA, not available.

      Section 1: classification of benign esophageal lesions and healthy controls

      Model performance

      Regarding identification of common benign lesions of the esophagus, we studied esophagitis and protruded lesions. In the test cohort of 1244 for esophagitis (the grade of esophagitis was evaluated by the Los Angeles classification
      • Lundell L.R.
      • Dent J.
      • Bennett J.R.
      • et al.
      Endoscopic assessment of oesophagitis: clinical and functional correlates and further validation of the Los Angeles classification.
      and is displayed in Supplementary Table 1, available online at www.giejournal.org), there were 1113 esophageal protruded lesions and 1196 healthy images. As shown in Figure 4A, the CNN model achieved the highest AUC of 0.851 (95% CI, 0.770-0.993) in correctly identifying patients with esophagitis, followed by 0.799 (95% CI, 0.708-0.890) and 0.751 (95% CI, 0.652-0.850) for normal images and esophageal protruded lesions, respectively.
      Figure thumbnail gr4
      Figure 4The CNN model to distinguish EL, EP and EC in WLE images. A, Represents the performance of the trained CNN to identify benign esophageal lesions in section 1. B, Shows the ROC curve to estimate the performance of the trained CNN in distinguishing subtypes of esophageal protruded lesions in section 2. C, Normal images were blue with no lesions as in a-2. Benign protruded regions were highlighted in images evaluated by CNN as shown in b-2, c-2, and d-2. AUC, Area under the curve; CNN, convolutional neural network; EC, esophageal cyst; EL, esophageal leiomyoma; EP, esophageal papilloma; ROC, receiver operating characteristic.

      Section 2: classification of subtypes in esophageal protruded lesions using WLE images

      Model performance

      As shown in Figure 4B, the proposed CNN model achieved the highest AUC of 0.907 (95% CI, 0.835-0.979) in correctly classifying EP compared with EL (AUC, 0.897; 95% CI, 0.841-0.953) and EC (AUC, 0.868; 95% CI, 0.769-0.968). To evaluate the performance of the proposed CNN model in classifying each subtype, we reported sensitivity, precision, specificity, accuracy, F1 score, and G mean, as presented in Table 3. In particular, the predictive accuracy for each class was 86.78%, 89.26%, and 87.60% for EL, EC, and EP, respectively. The highest F1 score and G mean were found in the EL group, indicating a good classification ability. We also used gradient-weighted class activation mapping (Grad-CAM)
      • Yoon H.J.
      • Kim J.H.
      Lesion-based convolutional neural network in diagnosis of early gastric cancer.
      to visually assess the model prediction of each class. As illustrated in Figure 4C, the CNN model correctly predicted abnormalities that were highlighted on attention heatmaps. The color depth of the heatmaps represented the possibility of predicted lesions by the CNN model.
      Table 3The performance of endoscopists in white-light endoscopy test sets
      AUC95% CISensitivity (%)Precision (%)Specificity (%)Accuracy (%)F1 scoreG meanGrit grade
      EL
       Model0.8970.841-0.95398.5181.4872.2286.780.890.84NA
      Reader 10.7570.672-0.84286.5775.3264.8176.860.810.753
      Reader 20.7680.685-0.85188.0674.6862.9676.860.810.742.6
      Reader 30.8230.749-0.89679.1077.9472.2276.030.790.763
      Reader 40.8400.770-0.91095.5281.0172.2285.120.880.834.3
      Reader 50.5820.481-0.68486.5759.7927.7860.330.710.493.3
      Reader 60.8520.784-0.91992.5482.6775.9385.120.870.843.8
      EP
       Model0.9070.835-0.97968.7581.4894.3887.600.750.81NA
      Reader 10.8540.766-0.94171.8895.8398.8891.740.820.843
      Reader 20.8940.818-0.97053.1394.4498.8886.780.680.722.6
      Reader 30.8970.822-0.97378.1396.1598.8893.390.860.883
      Reader 40.9280.864-0.99284.3896.4398.8895.040.900.914.3
      Reader 50.5880.470-0.70634.3868.7594.3878.510.460.573.3
      Reader 60.9230.749-0.93187.5096.5598.8895.870.920.933.8
      EC
       Model0.8680.769-0.9685084.6297.9889.260.630.70NA
      Reader 10.6770.544-0.81045.4550.0089.9081.820.480.643
      Reader 20.7530.629-0.87859.0954.1788.8983.470.570.722.6
      Reader 30.8000.684-0.91654.5544.4484.8579.340.490.683
      Reader 40.8430.737-0.95050.0078.5796.9788.430.610.704.3
      Reader 50.4420.312-0.5719.0925.0093.9478.510.130.293.3
      Reader 60.7570.633-0.88154.5570.5994.9587.600.620.723.8
      AUC, Area under the curve; CI, confidence interval; EL, esophageal leiomyoma; NA, not available; EP, esophageal papilloma; EC, esophageal cyst.

      Reader experiment results

      Detailed performance measurements by 6 endoscopists, including 2 experts (readers 1 and 2), 2 seniors (readers 3 and 4), and 2 novices (readers 5 and 6), are presented in Table 3. Using the same test cohort, we compared the model performance and reader assessment by measuring the sensitivity and specificity for each subtype of esophageal protruded lesions. As shown in Figure 6, the performance of most endoscopists trended at or below the model’s ROC for classifying EL and EC, whereas all endoscopists (except reader 5) showed higher sensitivity and specificity than the model performance in identifying EP. The Grit score of the 6 endoscopists revealed readers with different levels of diligence and engagement that may be associated with the varying diagnostic accuracy.
      Figure thumbnail gr5
      Figure 5CNN model to distinguish EL and EC in EUS images. A, The ROC curve to show the AUC values of the CNN to clarify the esophageal protruded lesions by EUS images. B, a-2 and b-2 demonstrate a representative case with lesions predicted by the model. AUC, Area under the curve; CNN, convolutional neural network; EC, esophageal cyst; EL, esophageal leiomyoma; ROC, receiver operating characteristic.
      Figure thumbnail gr6
      Figure 6Combined model and endoscopists’ predictions in sections 2 and 3. We examined the improvement in performance with the combined model and endoscopists’ predictions using WLE and EUS images. The ROC curves of (A) EL, (B) EP, and (C) EC by WLE images and (D) EL and (E) EC by EUS images. AUC, Area under the curve; EC, esophageal cyst; EL, esophageal leiomyoma; EP, esophageal papilloma; ROC, receiver operating characteristic.

      Section 3: discrimination of EL from EC in EUS images

      Model performance

      To further differentiate between EL and EC because of the complexity in interpreting then for inexperienced endoscopists, we trained an additional CNN model with EUS images. When the test cohorts comprising 431 EL and 165 EC EUS images were used, the model achieved an AUC of 0.739 (95% CI, 0.600-0.878) and 0.724 (95% CI, 0.567-0.881) for EL and EC, respectively (Fig. 5A). We also exploited the attention heatmaps to proactively track the predicted abnormalities on EUS images for each class (Fig. 5B).

      Reader experiment results

      Table 4 showed the same analysis presented in section 2. As shown in Figure 5A, the model achieved higher sensitivity and specificity than all endoscopists in correctly classifying EL and EC using EUS images. For example, the performance of the endoscopists trended at or below the model’s ROC, except for 1 expert reader.
      Table 4The performance of endoscopists in the EUS test sets
      AUC95% CISensitivity (%)Precision (%)Specificity (%)Accuracy (%)F1 scoreG mean
      EL
       Model0.7390.600-0.87893.7578.9552.9479.590.860.70
      Reader 10.8460.738-0.95375.0096.0094.1281.630.840.84
      Reader 20.7970.673-0.92084.3877.1452.9473.470.810.67
      Reader 30.7820.654-0.91078.1378.1358.8271.430.780.68
      Reader 40.7130.568-0.85881.2574.2947.0669.390.780.62
      Reader 50.6650.511-0.82093.7571.4329.4171.430.810.53
      Reader 60.6020.438-0.76681.2568.4229.4163.270.740.49
      EC
       Model0.7240.567-0.88152.9481.8193.7579.590.640.70
      Reader 10.8460.719-0.97294.1266.6775.0081.630.780.84
      Reader 20.7900.646-0.93352.9464.2984.3873.470.580.67
      Reader 30.7900.646-0.93358.8258.8278.1371.430.590.68
      Reader 40.7170.559-0.87547.0657.1481.2569.390.520.62
      Reader 50.6650.500-0.83129.4171.4393.7571.430.420.53
      Reader 60.6350.467-0.80329.4145.4581.2563.270.360.49
      AUC, Area under the curve; CI, confidence interval; EL, esophageal leiomyoma; EP, esophageal papilloma; EC, esophageal cyst.

      Ensemble models and endoscopists’ predictions in section 2 and 3

      Subsequently we evaluated whether an ensemble approach combining models and endoscopists’ predictions could improve the overall accuracy of differentiation. Focusing first on the task of identifying subtypes of protruded lesions, the sensitivity/specificity significantly increased compared with the best-performing endoscopists (readers 4 and 6) (Fig. 6). Interestingly, for identifying the EL subtype, the combined model and endoscopists’ prediction yielded a sensitivity boost of 5.79% and a specificity boost of 9.26%. For subtypes EC and EP, the improvement became less significant in specificity (Table 5).
      Table 5The performance of the convolutional neural network and endoscopists in the white-light endoscopy test sets
      Sensitivity (%)Precision (%)Specificity (%)Accuracy (%)F1 scoreG mean
      EL
       DL + reader 110093.0690.7495.870.960.95
       DL + reader 298.5188.0083.3391.740.930.91
       DL + reader 310091.7888.8995.040.960.94
       DL + reader 410093.0690.7495.870.960.95
       DL + reader 510085.9079.6390.910.920.89
       DL + reader 610093.0690.7495.870.960.95
      EP
       DL + reader 190.6390.6396.6395.040.910.94
       DL + reader 281.2596.3098.8894.210.880.90
       DL + reader 396.8810010099.170.980.98
       DL + reader 496.8896.8898.8898.350.970.98
       DL + reader 584.3887.1095.5192.560.860.90
       DL + reader 693.7596.7798.8897.520.950.96
      EC
       DL + reader 172.7394.1298.9994.210.820.85
       DL + reader 277.2789.4797.9894.210.830.87
       DL + reader 377.2710010095.870.870.88
       DL + reader 477.2710010095.870.870.88
       DL + reader 550.0091.6798.9990.080.650.70
       DL + reader 677.2794.4498.9995.040.850.87
      EL, Esophageal leiomyoma; DL, deep learning; EP, esophageal papilloma; EC, esophageal cyst.
      A similar analysis was performed using EUS images (Table 6). The overall accuracy increased from 80.17% to 90.91% (for EL) and outperformed endoscopists’ interpretations alone (Table 6). As shown in Figure 6D to E, combined model and endoscopists’ predictions yielded increased diagnostic ability for both classes.
      Table 6The performance of the convolutional neural network and endoscopists in EUS test sets
      Sensitivity (%)Precision (%)Specificity (%)Accuracy (%)F1 scoreG mean
      EL
       DL + reader 196.8896.8894.1295.920.970.95
       DL + reader 2100.0086.4970.5989.800.930.84
       DL + reader 396.8886.1170.5987.760.910.83
       DL + reader 496.8886.1170.5987.760.910.83
       DL + reader 596.8883.7864.7185.710.900.79
       DL + reader 696.8883.7864.7185.710.900.79
      EC
       DL + reader 194.1294.1296.8895.920.940.95
       DL + reader 270.59100.00100.0089.800.830.84
       DL + reader 370.5992.3196.8887.760.800.83
       DL + reader 470.5992.3196.8887.760.800.83
       DL + reader 564.7191.6796.8885.710.760.79
       DL + reader 664.7191.6796.8885.710.760.79
      EL, Esophageal leiomyoma; DL, deep learning; EC, esophageal cyst.

      Discussion

      In the present study, we retrospectively reviewed endoscopic images of esophageal benign protuberances and developed multiple CNN models on clinical data routinely obtained during endoscopic examinations. A total of 17,256 WLE images and 3226 EUS images were randomized to train and test the proposed models for specific classification tasks. The diagnostic accuracy in differentiating subtypes of protruded lesions has achieved the desired performance, outperforming most of the endoscopists. In addition, our preliminary results when models and endoscopists predictions were combined showed higher diagnostic accuracy compared with the endoscopists alone.
      Current AI algorithms have been trained on varying imaging modalities derived from advanced endoscopic techniques, such as chromoendoscopy, narrow-band imaging, i-Scan, and endocytoscopy.
      • Mori Y.
      • Kudo S.E.
      • Misawa M.
      • et al.
      Real-time use of artificial intelligence in identification of diminutive polyps during colonoscopy: a prospective study.
      • Takemura Y.
      • Yoshida S.
      • Tanaka S.
      • et al.
      Computer-aided system for predicting the histology of colorectal tumors by using narrow-band imaging magnifying colonoscopy (with video).
      • Kominami Y.
      • Yoshida S.
      • Tanaka S.
      • et al.
      Computer-aided diagnosis of colorectal polyp histology by using a real-time image recognition system and narrow-band imaging magnifying colonoscopy.
      These studies primarily focused on the detection of colorectal polyps or GI cancer and have achieved high diagnostic accuracy. A previous study by Wu et al
      • Wu
      • Zhou W.
      • Wan X.
      • et al.
      A deep neural network improves endoscopic detection of early gastric cancer without blind spots.
      reported a deep convolutional neural network-based system that accurately detected early gastric cancer and recognized gastric locations using gastroscopic videos. Hirasawa et al
      • Hirasawa T.
      • Aoyama K.
      • Tanimoto T.
      • et al.
      Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images.
      also established a CNN system for detection of gastric cancer on WLE images. However, early identification and diagnosis of benign esophageal protruded lesions has not been fully studied, which has significant clinical value, in particular in differentiating malignancy and disease assessment in the early stages. Therefore, in this study we aimed to investigate the application of CNN models in the differential diagnosis of upper GI protrusions (benign esophageal protruded lesions) on image modalities of WLE and EUS. To mimic the clinical workflow for diagnosis and confirmation of esophageal protuberance, we first performed the task of identifying esophageal benign lesions in normal patients and those with esophagitis using WLE images, because it was the initial modality of choice for patients suspected to have an esophageal protuberance. Once the existence of protruded lesions was confirmed, the next task was to classify the specific subtypes involved, such as EL, EC, or EP, as targeted in our study. Finally, for differential diagnosis between EL and EC due to the shared characteristics of manifestations on WLE, the CNN model was implemented and trained using EUS images. The results showed that the proposed CNN model successfully identified EL and EC with a high AUC of 0.897 and 0.868, and outperformed most of the endoscopists, including the experts. Interestingly, the greatest AUC found in detection of EP (AUC, 0.907) did not lead to a superior performance over all endoscopists (except for reader 5). The reason for these findings may be caused by the distinct patterns of EP that are often characterized as white-pink, wart-like exophytic projections on WLE images,
      • Wong M.W.
      • Bair M.J.
      • Shih S.C.
      • et al.
      Using typical endoscopic features to diagnose esophageal squamous papilloma.
      which were visually distinguishable by skilled endoscopists. In addition, the Grit survey conducted in the study showed that the participating endoscopists tended to take longer reading times than with routine clinical examinations, which often require rapid image interpretation and diagnosis.
      Distinguishing EC from EL is of great clinical value during endoscopic examinations, and EUS is the primary imaging modality of choice when evaluating EC.
      • Liu R.
      • Adler D.G.
      Duplication cysts: diagnosis, management, and the role of endoscopic ultrasound.
      ,
      • Chan M.
      • Zavala S.R.
      Esophageal cyst. Treasure Island.
      Instead of performing a biopsy or aspiration as for other subtypes for verification, EUS imaging allowed effective characterization of lesion echogenicity in EC, especially when a regular mucosal surface was observed.
      • Soares R.
      • Gasparaitis A.
      • Waxman I.
      • et al.
      Esophageal duplication cyst.
      As shown in our results, most of the skilled endoscopists had relatively low sensitivity and specificity in recognizing these lesions, whereas our proposed CNN model exhibited better accuracy although further improvements are necessary. Our trained CNN models used mature deep learning algorithms from a previously available imageNet test, which yielded high performance outcomes. Also, we managed to apply CNN models with various image modalities to complete tasks encountered in different clinical scenarios. In addition, the specific loss function was used to obtain optimized outcomes in multicategory classifications.
      With the increasing interest in AI algorithms and relevant applications in GI endoscopy, we were inspired to explore the possibility of deep learning in the identification of benign lesions to assist in our decision making and benefit the patients. The high performance of the proposed models in our study demonstrated the capacity for accurate classification of lesions subtypes, which could be helpful for prevention of under- or overdiagnosis by endoscopists across all experience levels. However, the primary purpose of a “clever” AI was to augment rather than to replace clinician services.
      • Byrne M.F.
      Artificial intelligence and the future of endoscopy: should we be quietly excited?.
      Our preliminary attempts to combine model and endoscopists’ predictions could lead to improved overall accuracy based on WLE or EUS images. Nevertheless, a prospective study accounting for the complex interaction between AI and human reading is needed to address absolute improvement in patient outcome.
      There were some limitations in the present study. First, the classification of esophageal benign protuberances used only WLE and EUS images instead of endoscopic videos. Continuous efforts to develop a real-time diagnostic CNN are needed to satisfy real clinical scenarios. Second, besides the subtypes of benign protruded lesions studied here, other types such as esophageal hemangioma, esophageal lipoma, esophageal stromal tumor, and esophageal leukoplakia have not been evaluated in the current study due to insufficient data collection. Further studies should be carried out under these conditions. Third, the accuracy of classification using EUS needs further improvement; the current data for patients are available for both WLE and EUC images. We aimed to combine both modalities into a hybrid model. Last, because it is a single-center retrospective study, the performance of the CNN algorithm should be further validated in endoscopy equipment and multicenter studies.
      In conclusion, we demonstrated that our preliminary CNN models achieved consistent diagnostic accuracy in the identification and classification of esophageal protruded lesions. A particularly promising result using EUS images was demonstrated; the ability to discriminate EL and EC outperformed most of the endoscopists. Moreover, preliminary results for the combined model and endoscopists’ predictions underscored the potential to assist endoscopists with improved diagnostic accuracy.

      Acknowledgments

      This work was supported by grants from the key project for Social Development in Jiangsu Province of China (no. BE2020784), the National Science and Technology Major Project of China (no. 2018), National Natural Science Foundation of China (no. 81770561 and no. 81970499) and the Medical Innovation Team of Jiangsu Province of China (no. CXTDA2017033).

      Appendix 1

      Supplementary Methods

      Training details

      We developed a classification model using a single image as input, and the output was the probability of each given class. The workflow and the convolutional neural network (CNN) architecture are shown in Figure 2. Several attempts have been made to identify the appropriate CNN models to achieved satisfactory classification performance with acceptable processing speed and efficacy. Eventually, we adopted the MobiLeNetv3 large
      • Hoda K.M.
      • Rodriguez S.A.
      • Faigel D.O.
      EUS-guided sampling of suspected GI stromal tumors.
      as the primary approach to classify each category in white-light endoscopy (WLE) and EUS images. Briefly, MobiLeNetv3 integrated several well-established CNN structures and convolution methods, such as depth-wise separable convolution proposed by MobiLeNetv1,
      • Baysal B.
      • Masri O.A.
      • Eloubeidi M.A.
      • et al.
      The role of EUS and EUS-guided FNA in the management of subepithelial lesions of the esophagus: a large, single-center experience.
      Squeeze-and-Excite block,
      • Mortensen M.B.
      • Fristrup C.
      • Holm F.S.
      • et al.
      Prospective evaluation of patient tolerability, satisfaction with patient information, and complications in endoscopic ultrasonography.
      and residual block proposed by ResNet.
      • Buscarini E.
      • Stasi M.D.
      • Rossi S.
      • et al.
      Endosonographic diagnosis of submucosal upper gastrointestinal tract lesions and large fold gastropathies by catheter ultrasound probe.
      The quantity of selected model parameters as 5,506,152 compared with 10,000,000 parameters in other CNN models, which could prevent the model overfitting to some extent. Advances also reflect that more images placed into the GPU results in good training stability. The initial state of the proposed models was transferred from ImageNet, a large annotated database used to train computer vision models.
      • Chak A.
      • Canto M.
      • Stevens P.D.
      • et al.
      Clinical applications of a new through-the-scope ultrasound probe: prospective comparison with an ultrasound endoscope.
      Different loss functions were used in the study, such as cross entropy loss, weighted cross entropy loss, and sigmoid loss. Eventually, we adopted weighted sigmoid loss for subtype differentiation, whereas cross entropy loss and sigmoid loss were applied in the binary classification of EUS images and major type differentiation in section 1 because similar accuracy was obtained. An all loss function was applied with class weight that equaled the inverse ratio of the amount of images in the training datasets. The definition of weighted cross entropy loss is explained in equation (1) and weighted sigmoid loss is explained in equation (2).
      Weighted Cross Entropy Loss=iwilogpi,labeli
      (1)


      Weighted sigmoid loss=ipipilabeli+wi(spftrelu(p)+relu(p))
      (2)


      where i refers to the class identifier, p indicates the prediction of the model, w is the weight of the corresponding class, and label represented the ground truth value.
      We applied similar hyperparameters to 3 training tasks. Batch size was set to 16. Weighted decay was set to 0.000001 and the initial learning rate was 0.01. We chose Adam (beta1 = 0.9, beta2 = 0.99) as optimizer. The training progress ended when the loss value decreased no larger than 0.01 for 5 epochs.

      Testing details and patient-level diagnosis

      The proposed CNN model gave class probabilities of a single image one at a time. Image-level predictions of a patient were merged together to generate a patient-level prediction. For each class, maximum probabilities were picked out as the final prediction. The work flow is shown in Figure 2. We then applied the proposed network to test the datasets mentioned earlier, ie, 1196 normal, 1244 esophagitis, and 1113 esophageal protruded lesions in section 1. Model training and testing were done using Mxnet (version1.6.0) and CUDA (version10.0). The GPU was 4 NVIDIA GeForce RTX 2070.
      Supplementary Table 1Grade information for the patients with esophagitis according to the Los Angeles classification system
      Esophagitis gradeNumber
      A117
      B79
      C4
      D0

      References

        • Hoda K.M.
        • Rodriguez S.A.
        • Faigel D.O.
        EUS-guided sampling of suspected GI stromal tumors.
        Gastrointest Endosc. 2009; 69: 1218-1223
        • Baysal B.
        • Masri O.A.
        • Eloubeidi M.A.
        • et al.
        The role of EUS and EUS-guided FNA in the management of subepithelial lesions of the esophagus: a large, single-center experience.
        Endosc Ultrasound. 2017; 6: 308-316
        • Mortensen M.B.
        • Fristrup C.
        • Holm F.S.
        • et al.
        Prospective evaluation of patient tolerability, satisfaction with patient information, and complications in endoscopic ultrasonography.
        Endoscopy. 2005; 37: 146-153
        • Buscarini E.
        • Stasi M.D.
        • Rossi S.
        • et al.
        Endosonographic diagnosis of submucosal upper gastrointestinal tract lesions and large fold gastropathies by catheter ultrasound probe.
        Gastrointest Endosc. 1999; 49: 184-191
        • Chak A.
        • Canto M.
        • Stevens P.D.
        • et al.
        Clinical applications of a new through-the-scope ultrasound probe: prospective comparison with an ultrasound endoscope.
        Gastrointest Endosc. 1997; 45: 291-295
        • Van Dam J.
        • Brady P.G.
        • Freeman M.
        • et al.
        Guidelines for training in electronic ultrasound: guidelines for clinical application. From the ASGE. American Society for Gastrointestinal Endoscopy.
        Gastrointest Endosc. 1999; 49: 829-833
        • Park C.
        • Lee S.
        • Oh M.
        • et al.
        The analysis of endoscopic ultrasonographic finding for esophageal subepithelial lesion.
        Endosc Ultrasound. 2014; 3: S5
        • Jeon H.W.
        • Choi M.G.
        • Lim C.H.
        • et al.
        Intraoperative esophagoscopy provides accuracy and safety in video-assisted thoracoscopic enucleation of benign esophageal submucosal tumors.
        Dis Esophagus. 2015; 28: 437-441
        • Liu R.
        • Adler D.G.
        Duplication cysts: diagnosis, management, and the role of endoscopic ultrasound.
        Endosc Ultrasound. 2014; 3: 152-160
        • Mendelson E.B.
        Artificial intelligence in breast imaging: potentials and limitations.
        AJR Am J Roentgenol. 2019; 212: 293-299
        • Shkolyar E.
        • Jia X.
        • Chang T.C.
        • et al.
        augmented bladder tumor detection using deep learning.
        Eur Urol. 2019; 76: 714-718
        • Willemink M.J.
        • Noel P.B.
        The evolution of image reconstruction for CT-from filtered back projection to artificial intelligence.
        Eur Radiol. 2019; 29: 2185-2195
        • Zhu Y.
        • Wang Q.C.
        • Xu M.D.
        • et al.
        Application of convolutional neural network in the diagnosis of the invasion depth of gastric cancer based on conventional endoscopy.
        Gastrointest Endosc. 2019; 89: 806-815.e1
        • Zhou J.
        • Wu L.
        • Wan X.
        • et al.
        A novel artificial intelligence system for the assessment of bowel preparation (with video).
        Gastrointest Endosc. 2020; 91: 428-435 e422
        • Aziz M.
        • Fatima R.
        • Dong C.
        • et al.
        The impact of deep convoluted neural network based artificial intelligence on colonoscopy outcomes: a systematic review with meta-analysis.
        J Gastroenterol Hepatol. 2020; 35: 1676-1683
        • Wu L.
        • Zhou W.
        • Wan X.
        • et al.
        A deep neural network improves endoscopic detection of early gastric cancer without blind spots.
        Endoscopy. 2019; 51: 522-531
        • Jing Y.
        • Bian Y.
        • Hu Z.
        • et al.
        Deep learning for drug design: an artificial intelligence paradigm for drug discovery in the big data era.
        AAPS J. 2018; 20: 58
        • Zhao L.
        • Jia K.
        Multiscale CNNs for brain tumor segmentation and diagnosis.
        Comput Math Methods Med. 2016; 2016: 8356294
        • Jisu H.
        • Bo-Yong P.
        • Hyunjin P.
        Convolutional neural network classifier for distinguishing Barrett's esophagus and neoplasia endomicroscopy images.
        Conf Proc IEEE Eng Med Biol Soc. 2017; 2017: 2892-2895
        • Saito H.
        • Aoki T.
        • Aoyama K.
        • et al.
        Automatic detection and classification of protruding lesions in wireless capsule endoscopy images based on a deep convolutional neural network.
        Gastrointest Endosc. 2020; 92: 144-151.e1
        • Urban G.
        • Tripathi P.
        • Alkayali T.
        • et al.
        Deep learning localizes and identifies polyps in real time with 96% accuracy in screening colonoscopy.
        Gastroenterology. 2018; 155: 1069-1078.e8
        • Howard A.
        • Sandler M.
        • Chu G.
        • et al.
        Searching for mobilenetv3.
        The IEEE International Conference on Computer Vision (ICCV). 2019; : 1314-1324
        • Cortes C.
        • Mohri M.
        Confidence intervals for the area under the ROC curve.
        Advances in Neural Information Processing Systems. 2005; 17
        • Selvaraju R.R.
        • Cogswell M.
        • Das A.
        • et al.
        Grad-CAM: visual explanations from deep networks via gradient-based localization.
        The IEEE International Conference on Computer Vision (ICCV). 2017; : 618-626
        • Duckworth A.L.
        • Quinn P.D.
        Development and validation of the short grit scale (grit-s).
        J Pers Assess. 2009; 91: 166-174
        • Lundell L.R.
        • Dent J.
        • Bennett J.R.
        • et al.
        Endoscopic assessment of oesophagitis: clinical and functional correlates and further validation of the Los Angeles classification.
        Gut. 1999; 45: 172-180
        • Yoon H.J.
        • Kim J.H.
        Lesion-based convolutional neural network in diagnosis of early gastric cancer.
        Clin Endosc. 2020; 53: 127-131
        • Mori Y.
        • Kudo S.E.
        • Misawa M.
        • et al.
        Real-time use of artificial intelligence in identification of diminutive polyps during colonoscopy: a prospective study.
        Ann Intern Med. 2018; 169: 357-366
        • Takemura Y.
        • Yoshida S.
        • Tanaka S.
        • et al.
        Computer-aided system for predicting the histology of colorectal tumors by using narrow-band imaging magnifying colonoscopy (with video).
        Gastrointest Endosc. 2012; 75: 179-185
        • Kominami Y.
        • Yoshida S.
        • Tanaka S.
        • et al.
        Computer-aided diagnosis of colorectal polyp histology by using a real-time image recognition system and narrow-band imaging magnifying colonoscopy.
        Gastrointest Endosc. 2016; 83: 643-649
        • Wu
        • Zhou W.
        • Wan X.
        • et al.
        A deep neural network improves endoscopic detection of early gastric cancer without blind spots.
        Endoscopy. 2019; 51: 522-531
        • Hirasawa T.
        • Aoyama K.
        • Tanimoto T.
        • et al.
        Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images.
        Gastric Cancer. 2018; 21: 653-660
        • Wong M.W.
        • Bair M.J.
        • Shih S.C.
        • et al.
        Using typical endoscopic features to diagnose esophageal squamous papilloma.
        World J Gastroenterol. 2016; 22: 2349-2356
        • Chan M.
        • Zavala S.R.
        Esophageal cyst. Treasure Island.
        StatPearls, FL2020
        • Soares R.
        • Gasparaitis A.
        • Waxman I.
        • et al.
        Esophageal duplication cyst.
        Dis Esophagus. 2011; 24: E21-E22
        • Byrne M.F.
        Artificial intelligence and the future of endoscopy: should we be quietly excited?.
        Endoscopy. 2019; 51: 511-512

      Linked Article

      • Artificial intelligence: finding the intersection of predictive modeling and clinical utility
        Gastrointestinal EndoscopyVol. 93Issue 6
        • Preview
          Artificial intelligence (AI) refers to the ability of computers to perform tasks normally reserved for human intelligence. AI is a broad concept and encompasses machine learning in which computers use data to create a binary predictive algorithm: deep learning that enhances machine learning by creating algorithms that select and weigh different variables to best predict an outcome. Convoluted neural networks are also created with interconnected “neurons” for pattern recognition, thereby weighing predictive features and learning from the data to predict outcomes.
        • Full-Text
        • PDF