Computer-aided diagnosis of esophageal cancer and neoplasms in endoscopic images: a systematic review and meta-analysis of diagnostic test accuracy

  • Chang Seok Bang
    Correspondence
    Reprint requests: Chang Seok Bang, MD, PhD, Department of Internal Medicine, Hallym University College of Medicine, Sakju-ro 77, Chuncheon, Gangwon-do, 24253, Korea.
    Affiliations
    Department of Internal Medicine, Hallym University College of Medicine, Chuncheon, Korea

    Institute for Liver and Digestive Diseases, Hallym University, Chuncheon, Korea

    Institute of New Frontier Research, Hallym University College of Medicine, Chuncheon, Korea

    Division of Big Data and Artificial Intelligence, Chuncheon Sacred Heart Hospital, Chuncheon, Korea
    Search for articles by this author
  • Jae Jun Lee
    Affiliations
    Institute of New Frontier Research, Hallym University College of Medicine, Chuncheon, Korea

    Division of Big Data and Artificial Intelligence, Chuncheon Sacred Heart Hospital, Chuncheon, Korea

    Department of Anesthesiology and Pain Medicine, Hallym University College of Medicine, Chuncheon, Korea
    Search for articles by this author
  • Gwang Ho Baik
    Affiliations
    Department of Internal Medicine, Hallym University College of Medicine, Chuncheon, Korea

    Institute for Liver and Digestive Diseases, Hallym University, Chuncheon, Korea
    Search for articles by this author
Open AccessPublished:December 04, 2020DOI:https://doi.org/10.1016/j.gie.2020.11.025

      Background and Aims

      Diagnosis of esophageal cancer or precursor lesions by endoscopic imaging depends on endoscopist expertise and is inevitably subject to interobserver variability. Studies on computer-aided diagnosis (CAD) using deep learning or machine learning are on the increase. However, studies with small sample sizes are limited by inadequate statistical strength. Here, we used a meta-analysis to evaluate the diagnostic test accuracy (DTA) of CAD algorithms of esophageal cancers or neoplasms using endoscopic images.

      Methods

      Core databases were searched for studies based on endoscopic imaging using CAD algorithms for the diagnosis of esophageal cancer or neoplasms and presenting data on diagnostic performance, and a systematic review and DTA meta-analysis were performed.

      Results

      Overall, 21 and 19 studies were included in the systematic review and DTA meta-analysis, respectively. The pooled area under the curve, sensitivity, specificity, and diagnostic odds ratio of CAD algorithms for the diagnosis of esophageal cancer or neoplasms in the image-based analysis were 0.97 (95% confidence interval [CI], 0.95-0.99), 0.94 (95% CI, 0.89-0.96), 0.88 (95% CI, 0.76-0.94), and 108 (95% CI, 43-273), respectively. Meta-regression showed no heterogeneity, and no publication bias was detected. The pooled area under the curve, sensitivity, specificity, and diagnostic odds ratio of CAD algorithms for the diagnosis of esophageal cancer invasion depth were 0.96 (95% CI, 0.86-0.99), 0.90 (95% CI, 0.88-0.92), 0.88 (95% CI, 0.83-0.91), and 138 (95% CI, 12-1569), respectively.

      Conclusions

      CAD algorithms showed high accuracy for the automatic endoscopic diagnosis of esophageal cancer and neoplasms. The limitation of a lack in performance in external validation and clinical applications should be overcome.

      Abbreviations:

      AUC (area under the curve), CAD (computer-aided diagnosis), CI (confidence interval), CNN (convolutional neural network), DOR (diagnostic odds ratio), DTA (diagnostic test accuracy), ESCC (esophageal squamous cell carcinoma), EAC (esophageal adenocarcinoma), ESCN (esophageal squamous cell neoplasia), HSROC (hierarchical summary receiver operating characteristic), IEE (image-enhanced endoscopy), NBI (narrow-band imaging), NLR (negative likelihood ratio), NPV (negative predictive value), PIVI (Preservation and Incorporation of Valuable endoscopic Innovations), PLR (positive likelihood ratio), SROC (summary receiver operating characteristic), SVM (support vector machine), WLI (white-light imaging)
      Esophageal cancer is the seventh most common cancer by incidence and the sixth most fatal cancer type worldwide.
      • Bray F.
      • Ferlay J.
      • Soerjomataram I.
      • et al.
      Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries.
      Esophageal squamous cell carcinoma (ESCC) is the most predominant histologic type globally. However, the proportion of esophageal adenocarcinoma (EAC) has been increasing due to the increased prevalence of gastroesophageal reflux disease (with Barrett’s esophagus) and obesity, especially in developed countries.
      • Bray F.
      • Ferlay J.
      • Soerjomataram I.
      • et al.
      Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries.
      ,
      • Klingelhofer D.
      • Zhu Y.
      • Braun M.
      • et al.
      A world map of esophagus cancer research: a critical accounting.
      The overall incidence of esophageal cancer is believed to be decreasing worldwide as the incidence of ESCC has decreased.
      • Wang Q.L.
      • Xie S.H.
      • Wahlin K.
      • et al.
      Global time trends in the incidence of esophageal squamous cell carcinoma.
      However, according to a recent comprehensive analysis, the global burden from esophageal cancer is still expected to increase.
      • Malhotra G.K.
      • Yanala U.
      • Ravipati A.
      • et al.
      Global trends in esophageal cancer.
      The management of esophageal cancer is challenging because most of these cancers are detected at a relatively advanced stage in elderly patients.
      • Evans J.A.
      • Early D.S.
      • Chandraskhara V.
      • et al.
      The role of endoscopy in the assessment and treatment of esophageal cancer.
      Therefore, early detection of esophageal cancer or any precursor lesion (such as Barrett’s neoplasia) is important. Currently, however, there is no established screening method for esophageal cancer, and endoscopic biopsy for incidentally detected lesions during upper endoscopic examination mainly results in the diagnosis of esophageal cancer. Endoscopic surveillance of patients with Barrett’s esophagus has been recommended; however, this initiative has also shown disappointing results.
      • Codipilly D.C.
      • Chandar A.K.
      • Singh S.
      • et al.
      The effect of endoscopic surveillance in patients with Barrett's esophagus: a systematic review and meta-analysis.
      Advanced esophageal cancer may exhibit definite protrusion into the esophageal lumen, ulceration, or infiltration in endoscopic images. However, superficial esophageal cancers limited to the mucosa or submucosa of the esophageal wall frequently exhibit only subtle changes of the surrounding mucosa, such as faint hyperemia, erosion, mucosal thickening, nodularity, white mucosal patches, or changes in the microvasculature.
      National Health Commission of the People's Republic of China
      Chinese guidelines for diagnosis and treatment of esophageal carcinoma 2018 (English version).
      Meticulous inspection, detection, and discrimination of such lesions are important endoscopist skills.
      • Cho B.J.
      • Bang C.S.
      • Park S.W.
      • et al.
      Automated classification of gastric neoplasms in endoscopic images using a convolutional neural network.
      However, the detection of these subtle changes in endoscopic images depends on the experience of the endoscopist and is inevitably subject to inter-rater variability.
      • Cho B.J.
      • Bang C.S.
      • Park S.W.
      • et al.
      Automated classification of gastric neoplasms in endoscopic images using a convolutional neural network.
      Chromoendoscopy using iodine or image-enhanced endoscopy (IEE), such as narrow-band imaging (NBI) with or without magnification, can be applied to the diagnosis of esophageal cancer. However, these are not routinely available, and substantial training is required to be able to carry out IEE.
      Superficial esophageal cancers or neoplasms may be curatively resected by endoscopic submucosal dissection.
      • Hamada K.
      • Kawano K.
      • Yamauchi A.
      • et al.
      Efficacy of endoscopic submucosal dissection of esophageal neoplasms under general anesthesia.
      Therefore, prediction of invasion depth by endoscopic imaging is also important for the endoscopist to decide on the treatment modality (endoscopic resection vs surgical resection). Endoscopic ultrasonography has been recommended for the discrimination between mucosa-confined lesions and submucosa-invading lesions.
      • Evans J.A.
      • Early D.S.
      • Chandraskhara V.
      • et al.
      The role of endoscopy in the assessment and treatment of esophageal cancer.
      ,
      National Health Commission of the People's Republic of China
      Chinese guidelines for diagnosis and treatment of esophageal carcinoma 2018 (English version).
      However, there is only limited evidence for the application of endoscopic ultrasonography for the accurate discrimination of the invasion depth of esophageal cancers.
      • Kitagawa Y.
      • Uno T.
      • Oyama T.
      • et al.
      Esophageal cancer practice guidelines 2017 edited by the Japan Esophageal Society: part 1.
      Studies based on computer-aided diagnosis (CAD) using deep learning or machine learning to determine the accuracy of artificial intelligence-based algorithms are becoming increasingly available.
      • Cho B.J.
      • Bang C.S.
      artificial intelligence for the determination of a management strategy for diminutive colorectal polyps: hype, hope, or help.
      Most of these algorithms were established through an automatic feature selection and self-learning from large amounts of endoscopic images and have been applied to the detection, classification, and segmentation (delineation) of esophageal cancers and neoplasms. However, studies carried out to evaluate the diagnostic performance of CAD algorithms have been characterized by small sample sizes and are correspondingly limited by inadequate statistical strength. Combining the results from all studies relevant to this topic might help increase the statistical strength and more accurately elucidate the performance of CAD algorithms. Moreover, no diagnostic test accuracy (DTA) meta-analysis of CAD algorithms for the diagnosis of esophageal cancer or neoplasms using endoscopic images has been published.
      This study aimed to evaluate the DTA of CAD algorithms used for detecting esophageal cancer or neoplasms in endoscopic images.

      Methods

      This study was registered in Prospero (registration number, ID 175159), protocol version 2, and IRB was not required.

      Criteria for the study selection

      The publications included in this systematic review met the following inclusion and exclusion criteria: inclusion criteria were (1) endoscopic images of esophageal cancer or neoplasms (case group) and endoscopic images without esophageal cancer or neoplasms (control group); however, in studies classifying the invasion depth of esophageal cancer or delineation of the lesion, the control group could be neoplasms located at a certain depth, or there might be no control group; (2) application of CAD algorithms (for the diagnosis, classification of invasion depth, or delineation of esophageal cancer or neoplasm); (3) presentation of the diagnostic performance of CAD algorithms, including area under the curve (AUC), sensitivity, specificity, and positive likelihood ratio (PLR), negative likelihood ratio (NLR), diagnostic odds ratio (DOR), or accuracy, which enabled estimation of true positive, false positive, false negative, and true negative values for the diagnosis of esophageal cancer or neoplasms based on endoscopic images; (4) prospective or retrospective study design; (5) adult participants; and (6) studies written in English. Exclusion criteria were (1) narrative reviews; (2) studies with incomplete data; (3) systematic review/meta-analyses; and (4) comments, proceedings, or study protocols. Articles meeting at least one of the exclusion criteria were excluded from this systematic review.

      Assessment of methodological quality

      The methodological quality of the final articles was assessed by 2 authors (C.S.B. and J.J.L) using the Quality Assessment of Diagnostic Accuracy Studies second version tool.
      • Whiting P.F.
      • Rutjes A.W.
      • Westwood M.E.
      • et al.
      QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies.
      This tool comprises 4 domains, including “patient selection,” “index test,” “reference standard,” and “flow and timing,” and the first 3 parts also have an “applicability” assessment. Each part was evaluated as either high risk, low risk, or unclear risk of bias by 2 authors.
      • Whiting P.F.
      • Rutjes A.W.
      • Westwood M.E.
      • et al.
      QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies.
      Authors adopted standard DTA meta-analysis using the bivariate method and hierarchical summary receiver operating characteristic method. STATA software version 15.1 (College Station, Tex, USA), including the packages METANDI and MIDAS and Meta-DiSc 1.4 (XI Cochrane Colloquium, Barcelona, Spain) were used for the statistical analyses. The detailed methodology is described in the Appendix 1 and Supplementary Table 1 (available online at www.giejournal.org).
      • McInnes M.D.F.
      • Moher D.
      • Thombs B.D.
      • et al.
      Preferred Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies: the PRISMA-DTA statement.
      • Reitsma J.B.
      • Glas A.S.
      • Rutjes A.W.
      • et al.
      Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews.
      • Rutter C.M.
      • Gatsonis C.A.
      A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations.
      • Harbord R.M.
      • Whiting P.
      Metandi: meta-analysis of diagnostic accuracy using hierarchical logistic regression.

      Results

      The detailed process of study selection is described in the Supplementary material and Supplementary Figure 1 (available online at www.giejournal.org).

      Clinical features observed in the studies

      The studies identified for this analysis established and explored the diagnostic value of CAD algorithms for (1) the diagnosis (automatic detection and classification) of esophageal cancer or neoplasms (n = 17),
      • de Groof A.J.
      • Struyvenberg M.R.
      • Fockens K.N.
      • et al.
      Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video).
      • Guo L.
      • Xiao X.
      • Wu C.
      • et al.
      Real-time automated diagnosis of precancerous lesions and early esophageal squamous cell carcinoma using a deep learning model (with videos).
      • Garcia-Peraza-Herrera L.C.
      • Everson M.
      • Lovat L.
      • et al.
      Intrapapillary capillary loop classification in magnification endoscopy: open dataset and baseline methodology.
      • Hashimoto R.
      • Requa J.
      • Dao T.
      • et al.
      Artificial intelligence using convolutional neural networks for real-time detection of early esophageal neoplasia in Barrett's esophagus (with video).
      • de Groof A.J.
      • Struyvenberg M.R.
      • van der Putten J.
      • et al.
      Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking.
      • Everson M.
      • Herrera L.
      • Li W.
      • et al.
      Artificial intelligence for the real-time classification of intrapapillary capillary loop patterns in the endoscopic diagnosis of early oesophageal squamous cell carcinoma: a proof-of-concept study.
      • Cai S.L.
      • Li B.
      • Tan W.M.
      • et al.
      Using a deep learning system in endoscopy for screening of early esophageal squamous cell carcinoma (with video).
      • Horie Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
      • Liu D.Y.
      • Gan T.
      • Rao N.N.
      • et al.
      Identification of lesion images from gastrointestinal endoscope based on feature extraction of combinational methods with and without learning process.
      • van der Sommen F.
      • Zinger S.
      • Curvers W.L.
      • et al.
      Computer-aided detection of early neoplastic lesions in Barrett's esophagus.
      • Ohmori M.
      • Ishihara R.
      • Aoyama K.
      • et al.
      Endoscopic detection and differentiation of esophageal lesions using a deep neural network.
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Real-time use of artificial intelligence in the evaluation of cancer in Barrett's oesophagus.
      • Zhao Y.Y.
      • Xue D.X.
      • Wang Y.L.
      • et al.
      Computer-assisted diagnosis of early esophageal squamous cell carcinoma using narrow-band imaging magnifying endoscopy.
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Computer-aided diagnosis using deep learning in the evaluation of early oesophageal adenocarcinoma.
      • de Groof J.
      • van der Sommen F.
      • van der Putten J.
      • et al.
      The Argos project: the development of a computer-aided detection system to improve detection of Barrett's neoplasia on white light endoscopy.
      • Sehgal V.
      • Rosenfeld A.
      • Graham D.
      • et al.
      Machine learning creates a simple endoscopic classification system that improves dysplasia detection in Barrett's oesophagus amongst non-expert endoscopists.
      • van der Sommen F.
      • Zinger S.
      • Schoon E.J.
      • et al.
      Supportive automatic annotation of early esophageal cancer using local gabor and color features.
      (2) the prediction of invasion depth (n = 3),
      • Horie Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
      ,
      • Tokai Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Application of artificial intelligence using convolutional neural networks in determining the invasion depth of esophageal squamous cell carcinoma.
      ,
      • Nakagawa K.
      • Ishihara R.
      • Aoyama K.
      • et al.
      Classification for invasion depth of esophageal squamous cell carcinoma using a deep neural network compared with experienced endoscopists.
      (3) the detection of esophageal cancer (n = 1),
      • Ghatwary N.
      • Zolgharni M.
      • Ye X.
      Early esophageal adenocarcinoma detection using deep learning methods.
      and (4) the segmentation of esophageal cancers (n = 1).
      • Liu D.
      • Rao N.
      • Mei X.
      • et al.
      Annotating early esophageal cancers based on two saliency levels of gastroscopic images.
      Many of the studies evaluated the diagnostic performance using a test dataset (internal validation). The study by de Groof et al
      • de Groof A.J.
      • Struyvenberg M.R.
      • van der Putten J.
      • et al.
      Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking.
      presented both external and internal validation performance data.
      Studies evaluating CAD algorithms for the diagnosis of esophageal cancers or neoplasms could be categorized based on the number of images included (image-based analysis)
      • de Groof A.J.
      • Struyvenberg M.R.
      • Fockens K.N.
      • et al.
      Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video).
      • Guo L.
      • Xiao X.
      • Wu C.
      • et al.
      Real-time automated diagnosis of precancerous lesions and early esophageal squamous cell carcinoma using a deep learning model (with videos).
      • Garcia-Peraza-Herrera L.C.
      • Everson M.
      • Lovat L.
      • et al.
      Intrapapillary capillary loop classification in magnification endoscopy: open dataset and baseline methodology.
      • Hashimoto R.
      • Requa J.
      • Dao T.
      • et al.
      Artificial intelligence using convolutional neural networks for real-time detection of early esophageal neoplasia in Barrett's esophagus (with video).
      • de Groof A.J.
      • Struyvenberg M.R.
      • van der Putten J.
      • et al.
      Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking.
      • Everson M.
      • Herrera L.
      • Li W.
      • et al.
      Artificial intelligence for the real-time classification of intrapapillary capillary loop patterns in the endoscopic diagnosis of early oesophageal squamous cell carcinoma: a proof-of-concept study.
      • Cai S.L.
      • Li B.
      • Tan W.M.
      • et al.
      Using a deep learning system in endoscopy for screening of early esophageal squamous cell carcinoma (with video).
      • Horie Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
      • Liu D.Y.
      • Gan T.
      • Rao N.N.
      • et al.
      Identification of lesion images from gastrointestinal endoscope based on feature extraction of combinational methods with and without learning process.
      • van der Sommen F.
      • Zinger S.
      • Curvers W.L.
      • et al.
      Computer-aided detection of early neoplastic lesions in Barrett's esophagus.
      and the number of patients enrolled (patient-based analysis; analysis with multiple images).
      • de Groof A.J.
      • Struyvenberg M.R.
      • Fockens K.N.
      • et al.
      Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video).
      ,
      • de Groof A.J.
      • Struyvenberg M.R.
      • van der Putten J.
      • et al.
      Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking.
      ,
      • Ohmori M.
      • Ishihara R.
      • Aoyama K.
      • et al.
      Endoscopic detection and differentiation of esophageal lesions using a deep neural network.
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Real-time use of artificial intelligence in the evaluation of cancer in Barrett's oesophagus.
      • Zhao Y.Y.
      • Xue D.X.
      • Wang Y.L.
      • et al.
      Computer-assisted diagnosis of early esophageal squamous cell carcinoma using narrow-band imaging magnifying endoscopy.
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Computer-aided diagnosis using deep learning in the evaluation of early oesophageal adenocarcinoma.
      • de Groof J.
      • van der Sommen F.
      • van der Putten J.
      • et al.
      The Argos project: the development of a computer-aided detection system to improve detection of Barrett's neoplasia on white light endoscopy.
      • Sehgal V.
      • Rosenfeld A.
      • Graham D.
      • et al.
      Machine learning creates a simple endoscopic classification system that improves dysplasia detection in Barrett's oesophagus amongst non-expert endoscopists.
      • van der Sommen F.
      • Zinger S.
      • Schoon E.J.
      • et al.
      Supportive automatic annotation of early esophageal cancer using local gabor and color features.
      Two articles
      • de Groof A.J.
      • Struyvenberg M.R.
      • Fockens K.N.
      • et al.
      Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video).
      ,
      • de Groof A.J.
      • Struyvenberg M.R.
      • van der Putten J.
      • et al.
      Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking.
      presented data from both image-based and patient-based analyses.
      Among the 10 image-based studies
      • de Groof A.J.
      • Struyvenberg M.R.
      • Fockens K.N.
      • et al.
      Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video).
      • Guo L.
      • Xiao X.
      • Wu C.
      • et al.
      Real-time automated diagnosis of precancerous lesions and early esophageal squamous cell carcinoma using a deep learning model (with videos).
      • Garcia-Peraza-Herrera L.C.
      • Everson M.
      • Lovat L.
      • et al.
      Intrapapillary capillary loop classification in magnification endoscopy: open dataset and baseline methodology.
      • Hashimoto R.
      • Requa J.
      • Dao T.
      • et al.
      Artificial intelligence using convolutional neural networks for real-time detection of early esophageal neoplasia in Barrett's esophagus (with video).
      • de Groof A.J.
      • Struyvenberg M.R.
      • van der Putten J.
      • et al.
      Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking.
      • Everson M.
      • Herrera L.
      • Li W.
      • et al.
      Artificial intelligence for the real-time classification of intrapapillary capillary loop patterns in the endoscopic diagnosis of early oesophageal squamous cell carcinoma: a proof-of-concept study.
      • Cai S.L.
      • Li B.
      • Tan W.M.
      • et al.
      Using a deep learning system in endoscopy for screening of early esophageal squamous cell carcinoma (with video).
      • Horie Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
      • Liu D.Y.
      • Gan T.
      • Rao N.N.
      • et al.
      Identification of lesion images from gastrointestinal endoscope based on feature extraction of combinational methods with and without learning process.
      • van der Sommen F.
      • Zinger S.
      • Curvers W.L.
      • et al.
      Computer-aided detection of early neoplastic lesions in Barrett's esophagus.
      for the diagnosis of esophageal cancer or neoplasms, a total of 77,521 images were identified (42,668 cases vs 34,853 controls). Specifically, 42,024 esophageal squamous cell neoplasia (ESCN), 447 Barrett’s neoplasia, and 197 esophageal cancer images were included in the case group. Five studies
      • Garcia-Peraza-Herrera L.C.
      • Everson M.
      • Lovat L.
      • et al.
      Intrapapillary capillary loop classification in magnification endoscopy: open dataset and baseline methodology.
      ,
      • Everson M.
      • Herrera L.
      • Li W.
      • et al.
      Artificial intelligence for the real-time classification of intrapapillary capillary loop patterns in the endoscopic diagnosis of early oesophageal squamous cell carcinoma: a proof-of-concept study.
      • Cai S.L.
      • Li B.
      • Tan W.M.
      • et al.
      Using a deep learning system in endoscopy for screening of early esophageal squamous cell carcinoma (with video).
      • Horie Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
      • Liu D.Y.
      • Gan T.
      • Rao N.N.
      • et al.
      Identification of lesion images from gastrointestinal endoscope based on feature extraction of combinational methods with and without learning process.
      used endoscopic images representing Asian populations, 4 studies
      • de Groof A.J.
      • Struyvenberg M.R.
      • Fockens K.N.
      • et al.
      Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video).
      ,
      • Hashimoto R.
      • Requa J.
      • Dao T.
      • et al.
      Artificial intelligence using convolutional neural networks for real-time detection of early esophageal neoplasia in Barrett's esophagus (with video).
      ,
      • de Groof A.J.
      • Struyvenberg M.R.
      • van der Putten J.
      • et al.
      Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking.
      ,
      • van der Sommen F.
      • Zinger S.
      • Curvers W.L.
      • et al.
      Computer-aided detection of early neoplastic lesions in Barrett's esophagus.
      used endoscopic images from Western populations, and 1 study
      • Guo L.
      • Xiao X.
      • Wu C.
      • et al.
      Real-time automated diagnosis of precancerous lesions and early esophageal squamous cell carcinoma using a deep learning model (with videos).
      used multinational image data. The type of CAD algorithm was the convolutional neural network (CNN) in most studies,
      • de Groof A.J.
      • Struyvenberg M.R.
      • Fockens K.N.
      • et al.
      Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video).
      • Guo L.
      • Xiao X.
      • Wu C.
      • et al.
      Real-time automated diagnosis of precancerous lesions and early esophageal squamous cell carcinoma using a deep learning model (with videos).
      • Garcia-Peraza-Herrera L.C.
      • Everson M.
      • Lovat L.
      • et al.
      Intrapapillary capillary loop classification in magnification endoscopy: open dataset and baseline methodology.
      • Hashimoto R.
      • Requa J.
      • Dao T.
      • et al.
      Artificial intelligence using convolutional neural networks for real-time detection of early esophageal neoplasia in Barrett's esophagus (with video).
      • de Groof A.J.
      • Struyvenberg M.R.
      • van der Putten J.
      • et al.
      Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking.
      • Everson M.
      • Herrera L.
      • Li W.
      • et al.
      Artificial intelligence for the real-time classification of intrapapillary capillary loop patterns in the endoscopic diagnosis of early oesophageal squamous cell carcinoma: a proof-of-concept study.
      • Cai S.L.
      • Li B.
      • Tan W.M.
      • et al.
      Using a deep learning system in endoscopy for screening of early esophageal squamous cell carcinoma (with video).
      • Horie Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
      except for 2,
      • Liu D.Y.
      • Gan T.
      • Rao N.N.
      • et al.
      Identification of lesion images from gastrointestinal endoscope based on feature extraction of combinational methods with and without learning process.
      ,
      • van der Sommen F.
      • Zinger S.
      • Curvers W.L.
      • et al.
      Computer-aided detection of early neoplastic lesions in Barrett's esophagus.
      which used a support vector machine (SVM). White-light imaging (WLI) is currently the standard method for the inspection of endoscopic lesions, and most of the studies
      • de Groof A.J.
      • Struyvenberg M.R.
      • Fockens K.N.
      • et al.
      Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video).
      ,
      • Hashimoto R.
      • Requa J.
      • Dao T.
      • et al.
      Artificial intelligence using convolutional neural networks for real-time detection of early esophageal neoplasia in Barrett's esophagus (with video).
      ,
      • de Groof A.J.
      • Struyvenberg M.R.
      • van der Putten J.
      • et al.
      Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking.
      ,
      • Cai S.L.
      • Li B.
      • Tan W.M.
      • et al.
      Using a deep learning system in endoscopy for screening of early esophageal squamous cell carcinoma (with video).
      • Horie Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
      • Liu D.Y.
      • Gan T.
      • Rao N.N.
      • et al.
      Identification of lesion images from gastrointestinal endoscope based on feature extraction of combinational methods with and without learning process.
      • van der Sommen F.
      • Zinger S.
      • Curvers W.L.
      • et al.
      Computer-aided detection of early neoplastic lesions in Barrett's esophagus.
      used WLI for the establishment of the CAD algorithm. However, 4 studies
      • Guo L.
      • Xiao X.
      • Wu C.
      • et al.
      Real-time automated diagnosis of precancerous lesions and early esophageal squamous cell carcinoma using a deep learning model (with videos).
      ,
      • Garcia-Peraza-Herrera L.C.
      • Everson M.
      • Lovat L.
      • et al.
      Intrapapillary capillary loop classification in magnification endoscopy: open dataset and baseline methodology.
      ,
      • Everson M.
      • Herrera L.
      • Li W.
      • et al.
      Artificial intelligence for the real-time classification of intrapapillary capillary loop patterns in the endoscopic diagnosis of early oesophageal squamous cell carcinoma: a proof-of-concept study.
      ,
      • Horie Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
      included NBI images (with or without magnification). The cases could be categorized into Barrett’s neoplasia (including EAC),
      • de Groof A.J.
      • Struyvenberg M.R.
      • Fockens K.N.
      • et al.
      Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video).
      ,
      • Hashimoto R.
      • Requa J.
      • Dao T.
      • et al.
      Artificial intelligence using convolutional neural networks for real-time detection of early esophageal neoplasia in Barrett's esophagus (with video).
      ,
      • de Groof A.J.
      • Struyvenberg M.R.
      • van der Putten J.
      • et al.
      Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking.
      ,
      • van der Sommen F.
      • Zinger S.
      • Curvers W.L.
      • et al.
      Computer-aided detection of early neoplastic lesions in Barrett's esophagus.
      ESCN (including ESCC),
      • Guo L.
      • Xiao X.
      • Wu C.
      • et al.
      Real-time automated diagnosis of precancerous lesions and early esophageal squamous cell carcinoma using a deep learning model (with videos).
      ,
      • Garcia-Peraza-Herrera L.C.
      • Everson M.
      • Lovat L.
      • et al.
      Intrapapillary capillary loop classification in magnification endoscopy: open dataset and baseline methodology.
      ,
      • Everson M.
      • Herrera L.
      • Li W.
      • et al.
      Artificial intelligence for the real-time classification of intrapapillary capillary loop patterns in the endoscopic diagnosis of early oesophageal squamous cell carcinoma: a proof-of-concept study.
      ,
      • Cai S.L.
      • Li B.
      • Tan W.M.
      • et al.
      Using a deep learning system in endoscopy for screening of early esophageal squamous cell carcinoma (with video).
      and esophageal cancer.
      • Horie Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
      ,
      • Liu D.Y.
      • Gan T.
      • Rao N.N.
      • et al.
      Identification of lesion images from gastrointestinal endoscope based on feature extraction of combinational methods with and without learning process.
      Comparison of the diagnostic performance of CAD algorithms compared with that of endoscopists was conducted only in the study by Cai et al
      • Cai S.L.
      • Li B.
      • Tan W.M.
      • et al.
      Using a deep learning system in endoscopy for screening of early esophageal squamous cell carcinoma (with video).
      (Table 1).
      Table 1Clinical characteristics of the studies (computer-aided diagnosis of esophageal cancers or neoplasm in image-based analysis)
      Study/yearStudy format/nationality (data)Type of CAD algorithmType of endoscopic imageType of controlsTotal number of imagesNumber of cases in test datasetNumber of controls in test datasetTPFPFNTNPerformance of endoscopists (TP/FP/FN/TN)
      de Groof et al (1) (2020)
      • de Groof A.J.
      • Struyvenberg M.R.
      • Fockens K.N.
      • et al.
      Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video).
      Prospective/EuropeCNNWLINondysplastic BE14433 Barrett’s neoplasias111 nondysplastic BEs2515896NA
      Guo et al (2020)
      • Guo L.
      • Xiao X.
      • Wu C.
      • et al.
      Real-time automated diagnosis of precancerous lesions and early esophageal squamous cell carcinoma using a deep learning model (with videos).
      Retrospective/multinational dataCNNNBINoncancer66711480 precancerous and ESCCs (ESCN)5191 noncancers1451258294933NA
      Garcia-Peraza-Herrera et al (2020)
      • Garcia-Peraza-Herrera L.C.
      • Everson M.
      • Lovat L.
      • et al.
      Intrapapillary capillary loop classification in magnification endoscopy: open dataset and baseline methodology.
      Retrospective/AsiaCNNME-NBINormal IPCL6774039662 abnormal IPCLs (ESCN)28078 normal IPCLs37,1632134249925,944NA
      Hashimoto et al (2020)
      • Hashimoto R.
      • Requa J.
      • Dao T.
      • et al.
      Artificial intelligence using convolutional neural networks for real-time detection of early esophageal neoplasia in Barrett's esophagus (with video).
      Retrospective/U.S.CNNWLINondysplastic BE448225 Barrett’s neoplasias223 nondysplastic BEs217138220NA
      de Groof AJ et al (2) (2020)
      • de Groof A.J.
      • Struyvenberg M.R.
      • van der Putten J.
      • et al.
      Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking.
      Prospective/multinational data (Europe)CNNWLINondysplastic BE297129 Barrett’s neoplasias168 nondysplastic BEs1131916149NA
      Everson et al (2019)
      • Everson M.
      • Herrera L.
      • Li W.
      • et al.
      Artificial intelligence for the real-time classification of intrapapillary capillary loop patterns in the endoscopic diagnosis of early oesophageal squamous cell carcinoma: a proof-of-concept study.
      Retrospective/AsiaCNNME-NBINormal IPCL1437791 abnormal IPCLs (ESCN)646 normal IPCLs7102081626NA
      Cai et al (2019)
      • Cai S.L.
      • Li B.
      • Tan W.M.
      • et al.
      Using a deep learning system in endoscopy for screening of early esophageal squamous cell carcinoma (with video).
      Retrospective/AsiaCNNWLINormal image18791 ESCN96 normal images891428280/8/13/86 (senior endoscopists)
      Horie et al (2019)
      • Horie Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
      Retrospective/AsiaCNNWLI with NBINoncancer9747 esophageal cancers50 noncancers464218NA
      WLI3835915
      NBI4128522
      Liu et al (2016)
      • Liu D.Y.
      • Gan T.
      • Rao N.N.
      • et al.
      Identification of lesion images from gastrointestinal endoscope based on feature extraction of combinational methods with and without learning process.
      Retrospective/AsiaSVMWLINormal image400150 early esophageal cancers250 normal images1402710223NA
      van der Sommen et al (1) (2016)
      • van der Sommen F.
      • Zinger S.
      • Curvers W.L.
      • et al.
      Computer-aided detection of early neoplastic lesions in Barrett's esophagus.
      Retrospective/EuropeSVMWLINondysplastic BE10060 Barrett’s neoplasia40 nondysplastic BEs5071033NA
      CAD, Computer-aided diagnosis; TP, true positive; FP, false positive; FN, false negative; TN, true negative; CNN, convolutional neural network; WLI, white-light imaging; BE, Barrett’s esophagus; NA, not applicable; NBI, narrow-band imaging; ESCC, esophageal squamous cell carcinoma; ESCN, early squamous cell neoplasia; ME, magnification endoscopy; IPCL, intrapapillary capillary loop classification; SVM, support vector machine.
      For the 9 patient-based studies
      • de Groof A.J.
      • Struyvenberg M.R.
      • Fockens K.N.
      • et al.
      Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video).
      ,
      • de Groof A.J.
      • Struyvenberg M.R.
      • van der Putten J.
      • et al.
      Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking.
      ,
      • Ohmori M.
      • Ishihara R.
      • Aoyama K.
      • et al.
      Endoscopic detection and differentiation of esophageal lesions using a deep neural network.
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Real-time use of artificial intelligence in the evaluation of cancer in Barrett's oesophagus.
      • Zhao Y.Y.
      • Xue D.X.
      • Wang Y.L.
      • et al.
      Computer-assisted diagnosis of early esophageal squamous cell carcinoma using narrow-band imaging magnifying endoscopy.
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Computer-aided diagnosis using deep learning in the evaluation of early oesophageal adenocarcinoma.
      • de Groof J.
      • van der Sommen F.
      • van der Putten J.
      • et al.
      The Argos project: the development of a computer-aided detection system to improve detection of Barrett's neoplasia on white light endoscopy.
      • Sehgal V.
      • Rosenfeld A.
      • Graham D.
      • et al.
      Machine learning creates a simple endoscopic classification system that improves dysplasia detection in Barrett's oesophagus amongst non-expert endoscopists.
      • van der Sommen F.
      • Zinger S.
      • Schoon E.J.
      • et al.
      Supportive automatic annotation of early esophageal cancer using local gabor and color features.
      evaluating CAD algorithms in the diagnosis of esophageal cancers or neoplasms, a total of 2102 patients were identified (1525 cases vs 577 controls). More specifically, 1228 patients with ESCC, 196 patients with Barrett’s neoplasia, and 101 patients with EAC were included in the case group. Seven studies
      • de Groof A.J.
      • Struyvenberg M.R.
      • Fockens K.N.
      • et al.
      Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video).
      ,
      • de Groof A.J.
      • Struyvenberg M.R.
      • van der Putten J.
      • et al.
      Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking.
      ,
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Real-time use of artificial intelligence in the evaluation of cancer in Barrett's oesophagus.
      ,
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Computer-aided diagnosis using deep learning in the evaluation of early oesophageal adenocarcinoma.
      • de Groof J.
      • van der Sommen F.
      • van der Putten J.
      • et al.
      The Argos project: the development of a computer-aided detection system to improve detection of Barrett's neoplasia on white light endoscopy.
      • Sehgal V.
      • Rosenfeld A.
      • Graham D.
      • et al.
      Machine learning creates a simple endoscopic classification system that improves dysplasia detection in Barrett's oesophagus amongst non-expert endoscopists.
      • van der Sommen F.
      • Zinger S.
      • Schoon E.J.
      • et al.
      Supportive automatic annotation of early esophageal cancer using local gabor and color features.
      used endoscopic images from Western individuals, and 2 studies
      • Ohmori M.
      • Ishihara R.
      • Aoyama K.
      • et al.
      Endoscopic detection and differentiation of esophageal lesions using a deep neural network.
      ,
      • Zhao Y.Y.
      • Xue D.X.
      • Wang Y.L.
      • et al.
      Computer-assisted diagnosis of early esophageal squamous cell carcinoma using narrow-band imaging magnifying endoscopy.
      used images representing Asian individuals. The type of CAD algorithm was CNN in most studies,
      • de Groof A.J.
      • Struyvenberg M.R.
      • Fockens K.N.
      • et al.
      Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video).
      ,
      • de Groof A.J.
      • Struyvenberg M.R.
      • van der Putten J.
      • et al.
      Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking.
      ,
      • Ohmori M.
      • Ishihara R.
      • Aoyama K.
      • et al.
      Endoscopic detection and differentiation of esophageal lesions using a deep neural network.
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Real-time use of artificial intelligence in the evaluation of cancer in Barrett's oesophagus.
      • Zhao Y.Y.
      • Xue D.X.
      • Wang Y.L.
      • et al.
      Computer-assisted diagnosis of early esophageal squamous cell carcinoma using narrow-band imaging magnifying endoscopy.
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Computer-aided diagnosis using deep learning in the evaluation of early oesophageal adenocarcinoma.
      except for 3 studies, which used an SVM
      • de Groof J.
      • van der Sommen F.
      • van der Putten J.
      • et al.
      The Argos project: the development of a computer-aided detection system to improve detection of Barrett's neoplasia on white light endoscopy.
      ,
      • van der Sommen F.
      • Zinger S.
      • Schoon E.J.
      • et al.
      Supportive automatic annotation of early esophageal cancer using local gabor and color features.
      or decision tree algorithm.
      • Sehgal V.
      • Rosenfeld A.
      • Graham D.
      • et al.
      Machine learning creates a simple endoscopic classification system that improves dysplasia detection in Barrett's oesophagus amongst non-expert endoscopists.
      Most of the studies
      • de Groof A.J.
      • Struyvenberg M.R.
      • Fockens K.N.
      • et al.
      Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video).
      ,
      • de Groof A.J.
      • Struyvenberg M.R.
      • van der Putten J.
      • et al.
      Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking.
      ,
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Real-time use of artificial intelligence in the evaluation of cancer in Barrett's oesophagus.
      ,
      • de Groof J.
      • van der Sommen F.
      • van der Putten J.
      • et al.
      The Argos project: the development of a computer-aided detection system to improve detection of Barrett's neoplasia on white light endoscopy.
      • Sehgal V.
      • Rosenfeld A.
      • Graham D.
      • et al.
      Machine learning creates a simple endoscopic classification system that improves dysplasia detection in Barrett's oesophagus amongst non-expert endoscopists.
      • van der Sommen F.
      • Zinger S.
      • Schoon E.J.
      • et al.
      Supportive automatic annotation of early esophageal cancer using local gabor and color features.
      used WLI for the establishment of the CAD algorithm. However, 3 studies
      • Ohmori M.
      • Ishihara R.
      • Aoyama K.
      • et al.
      Endoscopic detection and differentiation of esophageal lesions using a deep neural network.
      ,
      • Zhao Y.Y.
      • Xue D.X.
      • Wang Y.L.
      • et al.
      Computer-assisted diagnosis of early esophageal squamous cell carcinoma using narrow-band imaging magnifying endoscopy.
      ,
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Computer-aided diagnosis using deep learning in the evaluation of early oesophageal adenocarcinoma.
      included NBI or blue-light imaging (with or without magnification). The cases could be categorized into Barrett’s neoplasia (including EAC),
      • de Groof A.J.
      • Struyvenberg M.R.
      • Fockens K.N.
      • et al.
      Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video).
      ,
      • de Groof A.J.
      • Struyvenberg M.R.
      • van der Putten J.
      • et al.
      Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking.
      ,
      • de Groof J.
      • van der Sommen F.
      • van der Putten J.
      • et al.
      The Argos project: the development of a computer-aided detection system to improve detection of Barrett's neoplasia on white light endoscopy.
      ,
      • Sehgal V.
      • Rosenfeld A.
      • Graham D.
      • et al.
      Machine learning creates a simple endoscopic classification system that improves dysplasia detection in Barrett's oesophagus amongst non-expert endoscopists.
      EAC,
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Real-time use of artificial intelligence in the evaluation of cancer in Barrett's oesophagus.
      ,
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Computer-aided diagnosis using deep learning in the evaluation of early oesophageal adenocarcinoma.
      ,
      • van der Sommen F.
      • Zinger S.
      • Schoon E.J.
      • et al.
      Supportive automatic annotation of early esophageal cancer using local gabor and color features.
      and ESCC.
      • Ohmori M.
      • Ishihara R.
      • Aoyama K.
      • et al.
      Endoscopic detection and differentiation of esophageal lesions using a deep neural network.
      ,
      • Zhao Y.Y.
      • Xue D.X.
      • Wang Y.L.
      • et al.
      Computer-assisted diagnosis of early esophageal squamous cell carcinoma using narrow-band imaging magnifying endoscopy.
      A comparison of the diagnostic performance of the CAD algorithm and that of endoscopists was performed in 3 studies
      • Ohmori M.
      • Ishihara R.
      • Aoyama K.
      • et al.
      Endoscopic detection and differentiation of esophageal lesions using a deep neural network.
      ,
      • Zhao Y.Y.
      • Xue D.X.
      • Wang Y.L.
      • et al.
      Computer-assisted diagnosis of early esophageal squamous cell carcinoma using narrow-band imaging magnifying endoscopy.
      ,
      • Sehgal V.
      • Rosenfeld A.
      • Graham D.
      • et al.
      Machine learning creates a simple endoscopic classification system that improves dysplasia detection in Barrett's oesophagus amongst non-expert endoscopists.
      (Supplementary Table 2, available online at www.giejournal.org).
      In terms of predicting the invasion depth of esophageal cancers, 3 studies
      • Horie Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
      ,
      • Tokai Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Application of artificial intelligence using convolutional neural networks in determining the invasion depth of esophageal squamous cell carcinoma.
      ,
      • Nakagawa K.
      • Ishihara R.
      • Aoyama K.
      • et al.
      Classification for invasion depth of esophageal squamous cell carcinoma using a deep neural network compared with experienced endoscopists.
      evaluated the diagnostic performance of CAD algorithms. A total of 1361 images were included (1103 cases vs 258 controls). Specifically, 1193 ESCC and 168 esophageal cancer images, irrespective of ESCC or EAC, were included. All 3 studies included endoscopic images reflecting Asian populations, and CNN was commonly adopted as the background algorithm. Two studies
      • Tokai Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Application of artificial intelligence using convolutional neural networks in determining the invasion depth of esophageal squamous cell carcinoma.
      ,
      • Nakagawa K.
      • Ishihara R.
      • Aoyama K.
      • et al.
      Classification for invasion depth of esophageal squamous cell carcinoma using a deep neural network compared with experienced endoscopists.
      used WLI images to establish the algorithm; however, the study by Horie et al
      • Horie Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
      used NBI images. The standard for differentiating invasion depth was “sm1” in 2 studies (“confined to mucosa or sm1” vs “invaded deeper than sm1”).
      • Tokai Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Application of artificial intelligence using convolutional neural networks in determining the invasion depth of esophageal squamous cell carcinoma.
      ,
      • Nakagawa K.
      • Ishihara R.
      • Aoyama K.
      • et al.
      Classification for invasion depth of esophageal squamous cell carcinoma using a deep neural network compared with experienced endoscopists.
      However, the study by Horie et al
      • Horie Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
      adopted the standard of mucosa versus submucosa (“confined to mucosa” vs “invaded deeper than submucosa”). A comparison of the diagnostic performance between CAD algorithms and that of endoscopists was performed in 2 studies
      • Tokai Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Application of artificial intelligence using convolutional neural networks in determining the invasion depth of esophageal squamous cell carcinoma.
      ,
      • Nakagawa K.
      • Ishihara R.
      • Aoyama K.
      • et al.
      Classification for invasion depth of esophageal squamous cell carcinoma using a deep neural network compared with experienced endoscopists.
      (Supplementary Table 3, available online at www.giejournal.org).
      For the detection and segmentation of esophageal cancers by endoscopic imaging, Ghatwary et al
      • Ghatwary N.
      • Zolgharni M.
      • Ye X.
      Early esophageal adenocarcinoma detection using deep learning methods.
      established automated detection algorithms using regional-based CNN and a single-shot multibox detector, and Liu et al
      • Liu D.
      • Rao N.
      • Mei X.
      • et al.
      Annotating early esophageal cancers based on two saliency levels of gastroscopic images.
      established a segmentation algorithm using the Simple Linear Iterative Clustering technique (Supplementary Table 4, available online at www.giejournal.org).
      The identified modifiers were assessed as potential sources of heterogeneity by subgroup analysis and meta-regression. Detailed clinical features of the studies included are presented in Table 1 and Supplementary Table 2, Supplementary Table 3, Supplementary Table 4. (available online at www.giejournal.org).
      The detailed results of the quality assessment of study methodology are described in Figure 1, the Appendix 1, and Supplementary Figure 2 (available online at www.giejournal.org).
      Figure thumbnail gr1
      Figure 1Summary graph of quality in methodology.

      DTA meta-analysis of CAD algorithms for esophageal cancer or neoplasms

      Because only one study was identified as pertaining to cancer detection and segmentation based on a CAD algorithm, DTA meta-analysis was performed along with the studies pertaining to the diagnosis of esophageal cancers and neoplasms.
      Among the 10 studies
      • de Groof A.J.
      • Struyvenberg M.R.
      • Fockens K.N.
      • et al.
      Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video).
      • Guo L.
      • Xiao X.
      • Wu C.
      • et al.
      Real-time automated diagnosis of precancerous lesions and early esophageal squamous cell carcinoma using a deep learning model (with videos).
      • Garcia-Peraza-Herrera L.C.
      • Everson M.
      • Lovat L.
      • et al.
      Intrapapillary capillary loop classification in magnification endoscopy: open dataset and baseline methodology.
      • Hashimoto R.
      • Requa J.
      • Dao T.
      • et al.
      Artificial intelligence using convolutional neural networks for real-time detection of early esophageal neoplasia in Barrett's esophagus (with video).
      • de Groof A.J.
      • Struyvenberg M.R.
      • van der Putten J.
      • et al.
      Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking.
      • Everson M.
      • Herrera L.
      • Li W.
      • et al.
      Artificial intelligence for the real-time classification of intrapapillary capillary loop patterns in the endoscopic diagnosis of early oesophageal squamous cell carcinoma: a proof-of-concept study.
      • Cai S.L.
      • Li B.
      • Tan W.M.
      • et al.
      Using a deep learning system in endoscopy for screening of early esophageal squamous cell carcinoma (with video).
      • Horie Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
      • Liu D.Y.
      • Gan T.
      • Rao N.N.
      • et al.
      Identification of lesion images from gastrointestinal endoscope based on feature extraction of combinational methods with and without learning process.
      • van der Sommen F.
      • Zinger S.
      • Curvers W.L.
      • et al.
      Computer-aided detection of early neoplastic lesions in Barrett's esophagus.
      for the diagnosis of esophageal cancers or neoplasms (image-based), the AUC, sensitivity, specificity, PLR, NLR, and DOR of CAD algorithms for the diagnosis of esophageal cancers and neoplasms were 0.97 (95% confidence interval [CI], 0.95-0.99), 0.94 (95% CI, 0.89-0.96), 0.88 (95% CI, 0.76-0.94), 7.7 (95% CI, 3.8-15.7), 0.07 (95% CI, 0.04-0.12), and 108 (95% CI, 43-273), respectively (Table 2).
      Table 2Summary of diagnostic test accuracy meta-analysis and subgroup analysis for the diagnosis of esophageal cancers or neoplasms of the studies with image-based analysis
      SubgroupNumber of studies includedSensitivity (95% CI)Specificity (95% CI)PLRNLRDORAUC
      Value of meta-analysis in all the studies included for the diagnosis of100.94 (0.89-0.96)0.88 (0.76-0.94)70.7 (30.8-150.7)00.07 (00.04-00.12)108 (43-273)0.97 (0.95-0.99)
      Nationality of data
      Asian50.96 (0.91-0.98)0.84 (0.54-0.96)60.0 (10.7-210.1)0.05 (0.03-0.09)118 (52-266)0.97 (0.95-0.98)
      Western40.89 (0.78-0.94)0.89 (0.84-0.93)80.4 (50.1-130.7)00.13 (00.06-00.27)66 (20-213)0.94 (0.92-0.96)
      Multinational1NullNullNullNullNullNull
      Format of study
      Retrospective90.95 (0.91-0.97)0.88 (0.75-0.95)70.9 (30.6-170.4)0.06 (0.04-0.10)127 (48-332)0.97 (0.95-0.98)
      Prospective1NullNullNullNullNullNull
      Type of artificial intelligence
      CNN80.95 (0.90-0.97)0.88 (0.73-0.95)80.0 (30.3-190.2)0.06 (0.03-0.12)128 (44-378)0.97 (0.95-0.98)
      SVM2NullNullNullNullNullNull
      Type of endoscopic image
      WLI image70.90 (0.84-0.95)0.84 (0.70-0.92)50.6 (20.8-110.3)0.11 (0.06-0.22)49 (14-170)0.94 (0.92-0.96)
      NBI image40.94 (0.88-0.97)0.90 (0.69-0.97)90.2 (20.6-320.5)0.07 (0.03-0.14)137 (23-805)0.97 (0.95-0.98)
      Type of cases
      Barrett’s neoplasias (including EACs)40.89 (0.78-0.94)0.89 (0.84-0.93)80.4 (50.1-130.7)0.13 (0.06-0.27)66 (20-213)0.94 (0.92-0.96)
      ESCNs (including ESCC)40.96 (0.91-0.98)0.94 (0.89-0.96)150.3 (90.1-250.5)0.04 (0.02-0.10)341 (163-712)0.98 (0.97-0.99)
      Esophageal cancers (irrespective of EACs or ESCCs)2NullNullNullNullNullNull
      Methodological quality of the studies
      High quality70.95 (0.91-0.97)0.85 (0.66-0.94)60.5 (20.6-160.1)0.06 (0.03-0.11)114 (33-392)0.97 (0.95-0.98)
      Low quality3NullNullNullNullNullNull
      CI, Confidence interval; PLR, positive likelihood ratio; NLR, negative likelihood ratio; DOR, diagnostic odds ratio; AUC, area under the curve; CNN, convolutional neural network; SVM, support vector machine; WLI, wight-light imaging; NBI, narrow-band imaging; EAC, esophageal adenocarcinoma; ESCN, esophageal squamous cell neoplasia; ESCC, esophageal squamous cell carcinoma.
      For the 9 patient-based studies
      • de Groof A.J.
      • Struyvenberg M.R.
      • Fockens K.N.
      • et al.
      Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video).
      ,
      • de Groof A.J.
      • Struyvenberg M.R.
      • van der Putten J.
      • et al.
      Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking.
      ,
      • Ohmori M.
      • Ishihara R.
      • Aoyama K.
      • et al.
      Endoscopic detection and differentiation of esophageal lesions using a deep neural network.
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Real-time use of artificial intelligence in the evaluation of cancer in Barrett's oesophagus.
      • Zhao Y.Y.
      • Xue D.X.
      • Wang Y.L.
      • et al.
      Computer-assisted diagnosis of early esophageal squamous cell carcinoma using narrow-band imaging magnifying endoscopy.
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Computer-aided diagnosis using deep learning in the evaluation of early oesophageal adenocarcinoma.
      • de Groof J.
      • van der Sommen F.
      • van der Putten J.
      • et al.
      The Argos project: the development of a computer-aided detection system to improve detection of Barrett's neoplasia on white light endoscopy.
      • Sehgal V.
      • Rosenfeld A.
      • Graham D.
      • et al.
      Machine learning creates a simple endoscopic classification system that improves dysplasia detection in Barrett's oesophagus amongst non-expert endoscopists.
      • van der Sommen F.
      • Zinger S.
      • Schoon E.J.
      • et al.
      Supportive automatic annotation of early esophageal cancer using local gabor and color features.
      evaluating CAD algorithms for the diagnosis of esophageal cancer or neoplasms, the AUC, sensitivity, specificity, PLR, NLR, and DOR of CAD algorithms for the diagnosis of esophageal cancers and neoplasms were 0.94 (95% CI, 0.91-0.96), 0.93 (95% CI, 0.86-0.96), 0.85 (95% CI, 0.78-0.89), 6.0 (95% CI, 4.3-8.4), 0.09 (95% CI, 0.05-0.16), and 69 (95% CI, 35-137), respectively (Supplementary Table 5, available online at www.giejournal.org).
      In terms of prediction of the invasion depth of esophageal cancer, 3 studies
      • Horie Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
      ,
      • Tokai Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Application of artificial intelligence using convolutional neural networks in determining the invasion depth of esophageal squamous cell carcinoma.
      ,
      • Nakagawa K.
      • Ishihara R.
      • Aoyama K.
      • et al.
      Classification for invasion depth of esophageal squamous cell carcinoma using a deep neural network compared with experienced endoscopists.
      evaluated the diagnostic performance of CAD algorithms. The total AUC, sensitivity, specificity, PLR, NLR, and DOR of CAD algorithms for the diagnosis of esophageal cancers or neoplasms were 0.96 (95% CI, 0.86-0.99), 0.90 (0.88-0.92), 0.88 (0.83-0.91), 9.1 (1.6-53.5), 0.10 (0.04-0.23), and 138 (12-1569), respectively (Supplementary Table 6, available online at www.giejournal.org).

      Assessment of heterogeneity with meta-regression and subgroup analysis

      First, the summary receiver operating characteristic (SROC) curve was generated for the diagnosis of esophageal cancers or neoplasms with endoscopic images, and the shape of the curve was observed to be symmetric (Fig. 2). The authors observed a negative correlation coefficient between logit-transformed sensitivity and specificity (r = −0.15) and an asymmetric β parameter, with a nonsignificant P value (.24), implicating no heterogeneity among the studies. Second, a coupled forest plot of sensitivity and specificity was generated (Fig. 3). Compared with the enrolled studies, the study by Horie et al
      • Horie Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
      showed lower specificity; however, the authors could not find significant methodological pitfalls in this study. Third, meta-regression using modifiers identified in the systematic review was conducted, and no source of heterogeneity could be identified [nationality (P = .61), published year (P = .60), type of case (P = .10), total number of images (P = .84), study format (P = .17), methodological quality (P = .21), type of endoscopic image (P = .44), and type of CAD algorithm (P = .61)] (Supplementary Fig. 3, available online at www.giejournal.org). Finally, a subgroup analysis based on the potential modifiers was performed; no significant changes in the diagnostic performance according to the modifiers were observed (Table 2).
      Figure thumbnail gr2
      Figure 2Summary receiver operating characteristic (SROC) curve with 95% confidence region and prediction region of computer-aided diagnosis algorithms for the diagnosis of esophageal cancers or neoplasms in endoscopic images (image-based analysis). AUC, Area under the curve; SENS, sensitivity; SPEC, specificity.
      Figure thumbnail gr3
      Figure 3Coupled forest plots of sensitivity and specificity of computer-aided diagnosis algorithms for the diagnosis of esophageal cancers or neoplasms in endoscopic images (image-based analysis). CI, Confidence interval.
      With regard to the diagnosis of esophageal cancer or neoplasms using patient-based analysis, the shape of the SROC curve was symmetric (Supplementary Fig. 4, available online at www.giejournal.org). The coupled forest plot of sensitivity and specificity showed no outliers (Supplementary Fig. 5, available online at www.giejournal.org). Meta-regression revealed that nationality was the source of heterogeneity (P = .04) (published year [P = .31], type of case [P = .50], total number of images [P = .06], study format [P = .96], methodological quality [P = .26], type of endoscopic image [P = .35], and type of CAD algorithm [P = .11]) (Supplementary Fig. 5, available online at www.giejournal.org). However, there were no significant changes in the diagnostic performance according to the modifiers, especially the nationality (Supplementary Table 6, available online at www.giejournal.org).
      For the prediction of invasion depth of esophageal cancer in the image-based analysis, an SROC curve and a coupled forest plot of sensitivity and specificity by the Moses-Shapiro-Littenberg method were generated (Supplementary Figs. 7 and 8, available online at www.giejournal.org). Because only 3 studies
      • Horie Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
      ,
      • Tokai Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Application of artificial intelligence using convolutional neural networks in determining the invasion depth of esophageal squamous cell carcinoma.
      ,
      • Nakagawa K.
      • Ishihara R.
      • Aoyama K.
      • et al.
      Classification for invasion depth of esophageal squamous cell carcinoma using a deep neural network compared with experienced endoscopists.
      were included in the analysis, meta-regression or subgroup analysis was not performed.

      Evaluation of publication bias

      Deeks’ funnel plot of studies for the diagnosis of esophageal cancer or neoplasms in the image-based analysis exhibited a symmetrical shape with respect to the regression line (Fig. 4), and the asymmetry test showed no evidence of publication bias (P = .96). However, Deeks’ funnel plot asymmetry test for the studies in the patient-based analysis showed a P value of .03, indicating publication bias (Supplementary Fig. 9, available online at www.giejournal.org).
      Figure thumbnail gr4
      Figure 4Deek’s funnel plot of computer-aided diagnosis algorithms for the diagnosis of esophageal cancers or neoplasms in endoscopic images (image-based analysis). ESS, Effect sample size.

      Discussion

      The current study showed strong evidence of high accuracy of CAD algorithms for the diagnosis of esophageal cancer or neoplasms according to the DTA standard, implicating the feasibility of the CAD algorithm in clinical practice.
      • Lee Y.H.
      Overview of the process of conducting meta-analyses of the diagnostic test accuracy.
      • Deeks J.J.
      Systematic reviews in health care: systematic reviews of evaluations of diagnostic and screening tests.
      • Okeh U.
      • Okoro C.N.
      Evaluating measures of indicators of diagnostic test performance: fundamental meanings and formulars.
      Although esophageal cancer is one of the fastest-growing cancers, current guidelines do not recommend routine primary screening for esophageal cancer in the general popoulation.
      National Health Commission of the People's Republic of China
      Chinese guidelines for diagnosis and treatment of esophageal carcinoma 2018 (English version).
      ,
      • Kitagawa Y.
      • Uno T.
      • Oyama T.
      • et al.
      Esophageal cancer practice guidelines 2017 edited by the Japan Esophageal Society: part 1.
      Endoscopic screening or surveillance of high-risk patients, such as those with Barrett’s esophagus or precancerous lesions, has been commonly recommended.
      National Health Commission of the People's Republic of China
      Chinese guidelines for diagnosis and treatment of esophageal carcinoma 2018 (English version).
      ,
      • di Pietro M.
      • Fitzgerald R.C.
      Revised British Society of Gastroenterology recommendation on the diagnosis and management of Barrett's oesophagus with low-grade dysplasia.
      • Qumseya B.
      • Sultan S.
      • Bain P.
      • et al.
      ASGE guideline on screening and surveillance of Barrett's esophagus.
      • Saftoiu A.
      • Hassan C.
      • Areia M.
      • et al.
      Role of gastrointestinal endoscopy in the screening of digestive tract cancers in Europe: European Society of Gastrointestinal Endoscopy (ESGE) Position Statement.
      • Shaheen N.J.
      • Falk G.W.
      • Iyer P.G.
      • et al.
      ACG Clinical Guideline: diagnosis and management of Barrett's esophagus.
      However, a recent systematic review concluded that the evidence for screening for EAC and precancerous lesions is unclear, making it difficult to conclude whether or not populations with risk factors should be screened.
      • Hamel C.
      • Ahmadzai N.
      • Beck A.
      • et al.
      Screening for esophageal adenocarcinoma and precancerous conditions (dysplasia and Barrett's esophagus) in patients with chronic gastroesophageal reflux disease with or without other risk factors: two systematic reviews and one overview of reviews to inform a guideline of the Canadian Task Force on Preventive Health Care (CTFPHC).
      Therefore, incidental detection during endoscopic examination is still the mainstay for the diagnosis of esophageal cancer or neoplasms, and endoscopists have to be careful not to miss these lesions in the esophagus during routine procedures. However, a meticulous inspection of the esophagus is challenging during the endoscopic examination. Remnant saliva or mucus, normal peristaltic movements, and physiologic narrowing make it difficult to inspect the entire esophageal mucosa. Endoscopic inspection of the esophagus is also affected by the patient's heartbeat and breathing, and the upper esophagus can only be observed securely when the endoscope is slowly withdrawn.
      An increasing number of studies with CAD algorithms have been published, and these algorithms help provide additional identification and automated diagnosis of the lesions as digital assistants. In addition to esophageal lesions, CAD models have shown high performance in the detection of upper GI lesions. Therefore, it would be helpful throughout the upper GI endoscopy procedure.
      • Lui T.K.L.
      • Tsui V.W.M.
      • Leung W.K.
      Accuracy of artificial intelligence-assisted detection of upper GI lesions: a systematic review and meta-analysis.
      Although we are not aware of how endoscopists would react to a putative diagnosis made by CAD algorithms, providing robust answers by artificial intelligence irrespective of the fatigue level of endoscopists would be helpful to increase the likelihood of identifying critical lesions during endoscopic examination. Optical endoscopic diagnosis using IEE has received a great deal of attention and has the potential to replace chromoendoscopy. IEE aims to take a closer look at or scrutinize the subtle morphology of what the human eye cannot recognize. However, the interpretation is affected by the experience of endoscopists, and insufficient sensitivity has been observed when inexperienced endoscopists adopted IEE.
      • Horie Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
      ,
      • Ishihara R.
      • Takeuchi Y.
      • Chatani R.
      • et al.
      Prospective evaluation of narrow-band imaging endoscopy for screening of esophageal squamous mucosal high-grade neoplasia in experienced and less experienced endoscopists.
      Substantial training time and intra- and interobserver variability are the major drawbacks, and a definitive diagnosis still needs an endoscopic biopsy, which is an invasive procedure. The application of a highly accurate CAD algorithm in an endoscopic examination may reduce the need for unnecessary biopsies in a substantial proportion of patients.
      The American Society for Gastrointestinal Endoscopy recommended Preservation and Incorporation of Valuable endoscopic Innovations (PIVI) performance thresholds for an IEE with targeted biopsies (optical diagnosis), requiring a per patient sensitivity of >90%, a negative predictive value (NPV) of >98%, and specificity of >80% for detecting high-grade dysplasia or early EAC.
      • Sharma P.
      • Savides T.J.
      • Canto M.I.
      • et al.
      The American Society for Gastrointestinal Endoscopy PIVI (Preservation and Incorporation of Valuable Endoscopic Innovations) on imaging in Barrett's Esophagus.
      The pooled estimates of sensitivity and specificity of CAD algorithms in our study satisfied these criteria, irrespective of the type of analysis (image-based or patient-based). The NPV was not a pooled estimated value in our meta-analysis, and this value depends not only on sensitivity and specificity but also on the prevalence of the disease. Pooled NPV by integrating conditional prevalence with respect to a previous distribution (considering the heterogeneity in prevalence) can be calculated by a probability-modifying plot, and Supplementary Figures 10 and 11 (available online at www.giejournal.org) showed NPVs of 0.92 and 0.91, respectively. Although the pooled NPV of CAD algorithms did not reach the 98% PIVI threshold, some of the highly accurate CAD algorithms included in our meta-analysis are expected to meet this threshold. Considering the fact that there are currently no IEE tools perfectly meeting the PIVI threshold, further studies on CAD algorithms are expected in this regard.
      An additional finding of this study is the robustness of the diagnostic performance of the CAD algorithms, irrespective of the modifiers. The diagnostic performance with WLI showed values comparable with those of NBI. The estimated performance was not affected by the histologic type of cases, such as Barrett’s neoplasia, ESCN, ESCC, or EAC, indicating that the potential benefit is not limited to any such specific clinical situations. The type of CAD model (CNN vs SVM) was not a significant modifier. Considering the high sensitivity and specificity of CAD algorithms, this technology is expected to be integrated as a screening tool of esophageal cancers or neoplasms during endoscopic examination.
      Despite the strong diagnostic evidence accounted for above, several inevitable limitations were identified in this DTA meta-analysis. First, we found evidence of publication bias in the patient-based analysis. This bias is known to be increasing with the advancement of some technologies because nonsignificant results are not always published.
      • Shields P.G.
      Publication bias is a scientific problem with adverse ethical outcomes: the case for a section for null results.
      Recently, however, there has been a growing tendency in some journals to publish null results. Therefore, the accumulation of more data with patient-based analyses will further elucidate the value of this result. Second, there were only 3 studies with data on the prediction of the invasion depth. The bivariate and HSROC methods are advanced statistical techniques that have overcome the limitations of the Moses-Shapiro-Littenberg method (which does not consider any heterogeneity between studies). However, these did not apply to the analysis of the invasion depth due to the limited number of studies included. The final goal of CAD algorithms would not be limited to automated diagnosis but would also include the judgment of the invasion depth. Therefore, further studies are expected to enable us to draw more firm conclusions. Third, there was only one study focusing on pure detection and segmentation of esophageal cancers. Considering that most of the studies analyzed by the current DTA meta-analysis adopted automated detection and classification algorithms, pure detection and segmentation would already be technically possible. Fourth, there was an insufficient number of studies for comparison of the relative diagnostic performance of endoscopists and CAD algorithms. Because many studies have been accumulating on this topic,
      • Yang Y.J.
      • Bang C.S.
      Application of artificial intelligence in gastroenterology.
      studies on the doctor-CAD model collaboration might be more useful based on the future perspectives than comparing diagnostic performance. Fifth, the studies are heterogenous with different types of CADs (CNN vs SVM), types of endoscopic images (WLI vs NBI with/without magnification), types of controls (normal, nondysplastic Barrett's esophagus, normal intrapapillary capillary loop), and study designs (retrospective vs prospective). Given the current published literature, authors had to combine several outcomes of interest (detection of Barrett's neoplasia, ESCN, ESCC, or EAC) into one primary outcome.
      The performance of CAD algorithms depends on the quality of the baseline training data. However, we do not have qualified quality indicators for the baseline data. High-quality representative data reflecting real clinical practice should be collected to avoid spectrum bias (data imbalance) or overfitting (modeling error, which occurs when a certain learning model is excessively tailored to the training dataset, and predictions are not well generalized to new datasets) of CAD algorithms.
      • Cho B.J.
      • Bang C.S.
      artificial intelligence for the determination of a management strategy for diminutive colorectal polyps: hype, hope, or help.
      Additional studies focusing on CAD algorithm development are expected to be published shortly. However, the research focus should be changed to providing external validation-oriented performance or suggesting a clinical application benefit for future perspectives.
      In conclusion, CAD algorithms showed high accuracy for automatic endoscopic diagnosis of esophageal cancers and neoplasms. The limitation of a lack of performance in external validation or clinical application studies should be overcome.

      Acknowledgments

      Funding for this research was provided by the Bio & Medical Technology Development Program of the National Research Foundation (NRF) and by the Korean government, Ministry of Science and ICT ( MSIT ) (grant number NRF2017M3A9E8033253).

      Appendix 1

      Methodological considerations

      This systematic review with DTA meta-analysis was performed in accordance with the statement of the Preferred Reporting Items for a Systematic Review and Meta-analysis of DTA Studies.
      • McInnes M.D.F.
      • Moher D.
      • Thombs B.D.
      • et al.
      Preferred Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies: the PRISMA-DTA statement.
      The study protocol was registered at the International Prospective Register of Systematic Reviews database before initiation of the systematic review (CRD42020175159). Approval from the institutional review board of the Chuncheon Sacred Heart Hospital was exempted because only anonymized data were collected from the publications included in the review.

      Literature search

      Two authors (C.S.B. and J.J.L) independently performed a web-based core database search (MEDLINE-PubMed, Embase, and the Cochrane Library) using common search formulas (from inception to April 2020). Any duplicate articles were excluded. The titles and abstracts of all identified articles were reviewed, and irrelevant articles were excluded. Full-text reviews were subsequently carried out to determine whether the pre-established inclusion criteria were satisfied in the publications identified. References were also reviewed to identify any additional relevant studies. Disagreements between the authors were resolved by discussion or consultation with a third author (G.H.B.). The search formulas used to identify the relevant articles are detailed in Supplementary Table 1.

      Data extraction, primary outcomes, and additional analyses

      Two authors (C.S.B. and J.J.L.) independently extracted the data from each study and cross-checked the collected data. In cases where data were unclear, the corresponding author of the study was contacted by e-mail to obtain insight into the original dataset. A descriptive synthesis was developed by a systematic review process, and diagnostic test accuracy (DTA) meta-analysis was performed if the studies were sufficiently homogeneous.
      The primary outcomes of the DTA meta-analysis were true positive (TP), false positive (FP), false negative (FN), and true negative (TN) values in each study. For the computer-aided diagnosis (CAD) of esophageal cancer or neoplasms in endoscopic images, the definition of the primary outcome was as follows: TP, patients with a positive finding by a CAD algorithm who have esophageal cancers or neoplasms as shown by endoscopic imaging; FP, patients with a positive finding by a CAD algorithm who do not have esophageal cancers or neoplasms based on endoscopic images; FN, patients with a negative finding by a CAD algorithm who have esophageal cancers or neoplasms as shown by endoscopic images; and TN, patients with a negative finding by a CAD algorithm who do not have esophageal cancers or neoplasms based on endoscopic imaging.
      For the CAD of cancer invasion depth, a positive finding was defined as any lesion confined to the mucosa or with an invasion depth of less than 200 μm (sm1), and a negative finding was defined as any lesion with an invasion depth deeper than the submucosa or sm1 in esophageal cancers. Based on these definitions, TP, FP, FN, and TN values were calculated for each study.
      For additional analysis, such as subgroup analysis or meta-regression, the authors extracted the following variables from each study: the geographic origin of the data, year of publication, number of total images or patients included, study format, type of endoscopic images, type of CAD algorithm, and type of diagnosis, such as Barrett’s neoplasia, esophageal squamous cell neoplasia (ESCN), esophageal squamous cell carcinoma (ESCC), or esophageal adenocarcinoma (EAC).

      Statistics

      The bivariate method
      • Reitsma J.B.
      • Glas A.S.
      • Rutjes A.W.
      • et al.
      Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews.
      and hierarchical summary receiver operating characteristic (HSROC) method
      • Rutter C.M.
      • Gatsonis C.A.
      A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations.
      were adopted for the DTA meta-analysis. A forest plot of sensitivity and specificity and a summary receiver operating characteristic (SROC) curve were generated by the bivariate method and the HSROC method, respectively. The level of heterogeneity across the articles was determined by the correlation coefficient between logit-transformed sensitivity and specificity by the bivariate method and the asymmetry parameter β, where β = 0 corresponds to a symmetric ROC curve, in which the diagnostic odds ratio does not vary along the curve according to the HSROC method. A positive correlation coefficient and a β with a significant probability (P) value (P < .05) indicate heterogeneity between the studies.
      • Rutter C.M.
      • Gatsonis C.A.
      A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations.
      ,
      • Harbord R.M.
      • Whiting P.
      Metandi: meta-analysis of diagnostic accuracy using hierarchical logistic regression.
      Visual examination of the SROC curve was also performed to identify heterogeneity. Subgroup analysis by univariate meta-regression using the modifiers identified during the systematic review was also performed to identify the reasons for heterogeneity. STATA software version 15.1 (College Station, Tex, USA), including the packages METANDI and MIDAS, was used for the DTA meta-analysis. The METANDI and MIDAS packages require the inclusion of a minimum of 4 studies for DTA meta-analysis. Therefore, if less than 4 studies were included in the subgroup analysis, the Moses-Shapiro-Littenberg method, as implemented in Meta-DiSc 1.4 (XI Cochrane Colloquium, Barcelona, Spain) was used. Publication bias was evaluated using Deeks’ funnel plot asymmetry test.

      Study inclusion

      A total of 981 publications were identified by searching core databases. Two studies were additionally identified by manual screening of bibliographies. After excluding 306 duplicate studies, 545 additional articles were excluded after reviewing titles and abstracts. Full-text versions of the remaining 132 articles were obtained and thoroughly reviewed based on the inclusion and exclusion criteria listed above. Among these, 111 articles were excluded from the final enrollment for the following reasons: incomplete data (n = 97), narrative review (n = 8), comment (n = 2), proceedings (n = 2), systematic review/meta-analysis (n = 1), and study protocol (n = 1). Finally, 21 studies
      • de Groof A.J.
      • Struyvenberg M.R.
      • Fockens K.N.
      • et al.
      Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video).
      • Guo L.
      • Xiao X.
      • Wu C.
      • et al.
      Real-time automated diagnosis of precancerous lesions and early esophageal squamous cell carcinoma using a deep learning model (with videos).
      • Garcia-Peraza-Herrera L.C.
      • Everson M.
      • Lovat L.
      • et al.
      Intrapapillary capillary loop classification in magnification endoscopy: open dataset and baseline methodology.
      • Hashimoto R.
      • Requa J.
      • Dao T.
      • et al.
      Artificial intelligence using convolutional neural networks for real-time detection of early esophageal neoplasia in Barrett's esophagus (with video).
      • de Groof A.J.
      • Struyvenberg M.R.
      • van der Putten J.
      • et al.
      Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking.
      • Everson M.
      • Herrera L.
      • Li W.
      • et al.
      Artificial intelligence for the real-time classification of intrapapillary capillary loop patterns in the endoscopic diagnosis of early oesophageal squamous cell carcinoma: a proof-of-concept study.
      • Cai S.L.
      • Li B.
      • Tan W.M.
      • et al.
      Using a deep learning system in endoscopy for screening of early esophageal squamous cell carcinoma (with video).
      • Horie Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
      • Liu D.Y.
      • Gan T.
      • Rao N.N.
      • et al.
      Identification of lesion images from gastrointestinal endoscope based on feature extraction of combinational methods with and without learning process.
      • van der Sommen F.
      • Zinger S.
      • Curvers W.L.
      • et al.
      Computer-aided detection of early neoplastic lesions in Barrett's esophagus.
      • Ohmori M.
      • Ishihara R.
      • Aoyama K.
      • et al.
      Endoscopic detection and differentiation of esophageal lesions using a deep neural network.
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Real-time use of artificial intelligence in the evaluation of cancer in Barrett's oesophagus.
      • Zhao Y.Y.
      • Xue D.X.
      • Wang Y.L.
      • et al.
      Computer-assisted diagnosis of early esophageal squamous cell carcinoma using narrow-band imaging magnifying endoscopy.
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Computer-aided diagnosis using deep learning in the evaluation of early oesophageal adenocarcinoma.
      • de Groof J.
      • van der Sommen F.
      • van der Putten J.
      • et al.
      The Argos project: the development of a computer-aided detection system to improve detection of Barrett's neoplasia on white light endoscopy.
      • Sehgal V.
      • Rosenfeld A.
      • Graham D.
      • et al.
      Machine learning creates a simple endoscopic classification system that improves dysplasia detection in Barrett's oesophagus amongst non-expert endoscopists.
      • van der Sommen F.
      • Zinger S.
      • Schoon E.J.
      • et al.
      Supportive automatic annotation of early esophageal cancer using local gabor and color features.
      • Tokai Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Application of artificial intelligence using convolutional neural networks in determining the invasion depth of esophageal squamous cell carcinoma.
      • Nakagawa K.
      • Ishihara R.
      • Aoyama K.
      • et al.
      Classification for invasion depth of esophageal squamous cell carcinoma using a deep neural network compared with experienced endoscopists.
      • Ghatwary N.
      • Zolgharni M.
      • Ye X.
      Early esophageal adenocarcinoma detection using deep learning methods.
      • Liu D.
      • Rao N.
      • Mei X.
      • et al.
      Annotating early esophageal cancers based on two saliency levels of gastroscopic images.
      were included in the systematic review. A flowchart of the selection process is shown in Supplementary Figure 1.

      Quality assessment of study methodology

      As the CAD algorithms learn from data and are meant to infer correct conclusions, the quality of the baseline data is important. Theoretically, the images included in each study should reflect real clinical practice. However, because some lesions are rare or abnormal, data imbalance is the main barrier to the learning of CAD algorithms. Most studies
      • Guo L.
      • Xiao X.
      • Wu C.
      • et al.
      Real-time automated diagnosis of precancerous lesions and early esophageal squamous cell carcinoma using a deep learning model (with videos).
      • Garcia-Peraza-Herrera L.C.
      • Everson M.
      • Lovat L.
      • et al.
      Intrapapillary capillary loop classification in magnification endoscopy: open dataset and baseline methodology.
      • Hashimoto R.
      • Requa J.
      • Dao T.
      • et al.
      Artificial intelligence using convolutional neural networks for real-time detection of early esophageal neoplasia in Barrett's esophagus (with video).
      • de Groof A.J.
      • Struyvenberg M.R.
      • van der Putten J.
      • et al.
      Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking.
      ,
      • Cai S.L.
      • Li B.
      • Tan W.M.
      • et al.
      Using a deep learning system in endoscopy for screening of early esophageal squamous cell carcinoma (with video).
      ,
      • Horie Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
      ,
      • van der Sommen F.
      • Zinger S.
      • Curvers W.L.
      • et al.
      Computer-aided detection of early neoplastic lesions in Barrett's esophagus.
      ,
      • Ohmori M.
      • Ishihara R.
      • Aoyama K.
      • et al.
      Endoscopic detection and differentiation of esophageal lesions using a deep neural network.
      ,
      • Zhao Y.Y.
      • Xue D.X.
      • Wang Y.L.
      • et al.
      Computer-assisted diagnosis of early esophageal squamous cell carcinoma using narrow-band imaging magnifying endoscopy.
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Computer-aided diagnosis using deep learning in the evaluation of early oesophageal adenocarcinoma.
      • de Groof J.
      • van der Sommen F.
      • van der Putten J.
      • et al.
      The Argos project: the development of a computer-aided detection system to improve detection of Barrett's neoplasia on white light endoscopy.
      ,
      • Tokai Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Application of artificial intelligence using convolutional neural networks in determining the invasion depth of esophageal squamous cell carcinoma.
      • Nakagawa K.
      • Ishihara R.
      • Aoyama K.
      • et al.
      Classification for invasion depth of esophageal squamous cell carcinoma using a deep neural network compared with experienced endoscopists.
      • Ghatwary N.
      • Zolgharni M.
      • Ye X.
      Early esophageal adenocarcinoma detection using deep learning methods.
      included in the systematic review tried to mitigate this pitfall using an adaptation of specific inclusion and exclusion criteria for the enrollment of endoscopic images. However, 7 studies
      • de Groof A.J.
      • Struyvenberg M.R.
      • Fockens K.N.
      • et al.
      Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video).
      ,
      • Everson M.
      • Herrera L.
      • Li W.
      • et al.
      Artificial intelligence for the real-time classification of intrapapillary capillary loop patterns in the endoscopic diagnosis of early oesophageal squamous cell carcinoma: a proof-of-concept study.
      ,
      • Liu D.Y.
      • Gan T.
      • Rao N.N.
      • et al.
      Identification of lesion images from gastrointestinal endoscope based on feature extraction of combinational methods with and without learning process.
      ,
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Real-time use of artificial intelligence in the evaluation of cancer in Barrett's oesophagus.
      ,
      • Sehgal V.
      • Rosenfeld A.
      • Graham D.
      • et al.
      Machine learning creates a simple endoscopic classification system that improves dysplasia detection in Barrett's oesophagus amongst non-expert endoscopists.
      ,
      • van der Sommen F.
      • Zinger S.
      • Schoon E.J.
      • et al.
      Supportive automatic annotation of early esophageal cancer using local gabor and color features.
      ,
      • Liu D.
      • Rao N.
      • Mei X.
      • et al.
      Annotating early esophageal cancers based on two saliency levels of gastroscopic images.
      did not include a detailed description of the patient enrollment standard. Therefore, these studies
      • de Groof A.J.
      • Struyvenberg M.R.
      • Fockens K.N.
      • et al.
      Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video).
      ,
      • Everson M.
      • Herrera L.
      • Li W.
      • et al.
      Artificial intelligence for the real-time classification of intrapapillary capillary loop patterns in the endoscopic diagnosis of early oesophageal squamous cell carcinoma: a proof-of-concept study.
      ,
      • Liu D.Y.
      • Gan T.
      • Rao N.N.
      • et al.
      Identification of lesion images from gastrointestinal endoscope based on feature extraction of combinational methods with and without learning process.
      ,
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Real-time use of artificial intelligence in the evaluation of cancer in Barrett's oesophagus.
      ,
      • Sehgal V.
      • Rosenfeld A.
      • Graham D.
      • et al.
      Machine learning creates a simple endoscopic classification system that improves dysplasia detection in Barrett's oesophagus amongst non-expert endoscopists.
      ,
      • van der Sommen F.
      • Zinger S.
      • Schoon E.J.
      • et al.
      Supportive automatic annotation of early esophageal cancer using local gabor and color features.
      ,
      • Liu D.
      • Rao N.
      • Mei X.
      • et al.
      Annotating early esophageal cancers based on two saliency levels of gastroscopic images.
      were rated as “unclear risk” in the “patient selection” domain (Fig. 1 and Supplementary Fig. 2). This binary classification of “low risk” and “unclear risk” in the “patient selection” domain was adopted as a modifier in the subgroup or meta-regression analysis.
      Figure thumbnail fx1
      Supplementary Figure 1Flowchart of the search process.
      Figure thumbnail fx2
      Supplementary Figure 2Summary table of quality in methodology.
      Figure thumbnail fx3
      Supplementary Figure 3Univariable meta-regression plot of computer-aided diagnosis algorithms for the diagnosis of esophageal cancers or neoplasms in endoscopic images (image-based analysis). CI, Confidence interval.
      Figure thumbnail fx4
      Supplementary Figure 4Summary receiver operating characteristic (SROC) curve with 95% confidence region and prediction region of computer-aided diagnosis algorithms for the diagnosis of esophageal cancers or neoplasms in endoscopic images (patient-based analysis). AUC, Area under the curve; SENS, sensitivity; SPEC, specificity.
      Figure thumbnail fx5
      Supplementary Figure 5Coupled forest plots of sensitivity and specificity of computer-aided diagnosis algorithms for the diagnosis of esophageal cancers or neoplasms in endoscopic images (patient-based analysis). CI, Confidence interval.
      Figure thumbnail fx6
      Supplementary Figure 6Univariable meta-regression plot of computer-aided diagnosis algorithms for the diagnosis of esophageal cancers or neoplasms in endoscopic images (patient-based analysis). ESS, Effect sample size.
      Figure thumbnail fx7
      Supplementary Figure 7Summary receiver operating characteristic curve (SROC) with 95% confidence region and prediction region of computer-aided diagnosis algorithms for the prediction of invasion depth in esophageal cancers (image-based analysis). AUC, Area under the curve; SE, standard error.
      Figure thumbnail fx8
      Supplementary Figure 8Coupled forest plots of sensitivity and specificity of computer-aided diagnosis algorithms for the prediction of invasion depth in esophageal cancers (image-based analysis). CI, Confidence interval.
      Figure thumbnail fx9
      Supplementary Figure 9Deek’s funnel plot of computer-aided diagnosis algorithms for the diagnosis of esophageal cancers or neoplasms in endoscopic images (patient-based analysis). ESS, Effect sample size.
      Figure thumbnail fx10
      Supplementary Figure 10Probability-modifying plot of computer-aided diagnosis algorithms for the diagnosis of esophageal cancers or neoplasms in endoscopic images (image-based analysis). LR, Likelihood ratio; NPV, negative predictive value; PPV, positive predictive value.
      Figure thumbnail fx11
      Supplementary Figure 11Probability-modifying plot of computer-aided diagnosis algorithms for the diagnosis of esophageal cancers or neoplasms in endoscopic images (patient-based analysis). LR, Likelihood ratio; NPV, negative predictive value; PPV, positive predictive value.
      Supplementary Table 1Search strategy to find the relevant articles
      Database: MEDLINE (through PubMed)
      #1 "artificial intelligence"[tiab] OR "AI"[tiab] OR "deep learning"[tiab] OR "machine learning"[tiab] OR "computer"[tiab] OR "neural network"[tiab] OR "CNN"[tiab] OR "automatic"[tiab] OR "automated"[tiab]: 457179
      #2 "esophageal cancer"[tiab] OR "esophageal neoplasia"[tiab] OR “esophageal neoplasms”[Mesh] OR "Barrett’s neoplasia"[tiab] OR "Barrett’s neoplasm"[tiab] OR "Barrett’s esophagus"[tiab] OR "esophageal lesion"[tiab] OR "esophageal squamous cell carcinoma"[tiab] OR "esophageal adenocarcinoma"[tiab]: 59957
      #3 #1 AND #2: 426
      #4 #3 AND English[Lang]: 373
      Database: Embase
      #1 'artificial intelligence':ab,ti,kw OR 'AI':ab,ti,kw OR 'deep learning':ab,ti,kw OR 'machine learing':ab,ti,kw OR 'computer':ab,ti,kw OR 'neural network':ab,ti,kw OR 'CNN':ab,ti,kw OR 'automatic':ab,ti,kw OR 'automated': 590441
      #2 'esophageal cancer':ab,ti,kw OR 'esophageal neoplasia':ab,ti,kw OR 'Barrett neoplasia':ab,ti,kw OR 'Barrett neoplasms':ab,ti,kw OR 'Barrett esophagus':ab,ti,kw OR 'esophageal lesion':ab,ti,kw OR 'esophageal squamous cell carcinoma':ab,ti,kw OR 'esophageal adenocarcinoma':ab,ti,kw OR 'esophageal neoplasms'/exp: 90893
      #3 #1 AND #2: 778
      #4 #3 AND ([article]/lim OR [article in press]/lim OR [review]/lim) AND [English]/lim: 428
      Database: Cochrane Library
      #1 artificial intelligence:ab,ti,kw or AI:ab,ti,kw or deep learning:ab,ti,kw or machine learning:ab,ti,kw or computer:ab,ti,kw or neural network:ab,ti,kw or CNN:ab,ti,kw or automatic:ab,ti,kw or automated:ab,ti,kw: 52211
      #2 MeSH descriptor: [esophageal neoplasms] explode all trees: 1530
      #3 esophageal cancer:ab,ti,kw or esophageal neoplasia:ab,ti,kw or Barrett’s neoplasia:ab,ti,kw or Barrett’s neoplasms:ab,ti,kw or Barrett’s esophagus:ab,ti,kw or esophageal lesion:ab,ti,kw or esophageal squamous cell carcinoma:ab,ti,kw or esophageal adenocarcinoma:ab,ti,kw: 4338
      #3 #2 or #3: 4541
      #4 #1 and #4: 180
      Supplementary Table 2Clinical characteristics of the studies (computer-aided diagnosis of esophageal cancers or neoplasm in patient-based analysis)
      Study/yearStudy format/nationality (data)Type of CAD algorithmType of endoscopic imageType of controlsTotal number of patientsNumber of cases in test datasetNumber of controls in test datasetTPFPFNTNPerformance of endoscopists (TP/FP/FN/TN)
      de Groof et al (1) (2020)
      • de Groof A.J.
      • Struyvenberg M.R.
      • Fockens K.N.
      • et al.
      Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video).
      Prospective/EuropeCNNWLINondysplastic BE2010 Barrett’s neoplasias10 nondysplastic BEs9317NA
      Ohmori et al (2020)
      • Ohmori M.
      • Ishihara R.
      • Aoyama K.
      • et al.
      Endoscopic detection and differentiation of esophageal lesions using a deep neural network.
      Retrospective/AsiaCNNNon-ME detection, ME diagnosisNoncancer or normal10252 superficial ESCC50511613443/13/9/37
      WLI472056345/27/7/56
      NBI/BLI523105248/26/4/57
      ME-NBI/BLI512212843/15/9/35
      Ebigbo et al (1) (2020)
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Real-time use of artificial intelligence in the evaluation of cancer in Barrett's oesophagus.
      Retrospective/EuropeCNNWLINondysplastic BE6236 early EACs26 nondysplastic BE300626NA
      de Groof A et al (2) (2020)
      • de Groof A.J.
      • Struyvenberg M.R.
      • van der Putten J.
      • et al.
      Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking.
      Prospective/multinational data (Europe)CNNWLINondysplastic BE297129 Barrett’s neoplasias168 nondysplastic BEs1131916149NA
      Zhao et al (2019)
      • Zhao Y.Y.
      • Xue D.X.
      • Wang Y.L.
      • et al.
      Computer-assisted diagnosis of early esophageal squamous cell carcinoma using narrow-band imaging magnifying endoscopy.
      Retrospective/AsiaCNNME-NBINoncancerous IPCL13831176 IPCLs (early ESCC)207 noncancerous IPCLs1023331531741064/33/112/174 (senior endoscopists)
      Ebigbo et al (2) (2019)
      • Ebigbo A.
      • Mendel R.
      • Probst A.
      • et al.
      Computer-aided diagnosis using deep learning in the evaluation of early oesophageal adenocarcinoma.
      Retrospective/EuropeCNNWLI with NBINondysplastic BE7433 early EACs41 nondysplastic BE325136NA
      de Groof et al (3) (2019)
      • de Groof J.
      • van der Sommen F.
      • van der Putten J.
      • et al.
      The Argos project: the development of a computer-aided detection system to improve detection of Barrett's neoplasia on white light endoscopy.
      Prospective/EuropeSVMWLINondysplastic BE6040 Barrett’s neoplasias20 nondysplastic BE383217NA
      Sehgal et al (2018)
      • Sehgal V.
      • Rosenfeld A.
      • Graham D.
      • et al.
      Machine learning creates a simple endoscopic classification system that improves dysplasia detection in Barrett's oesophagus amongst non-expert endoscopists.
      Retrospective/EuropeDecision tree algorithmWLINondysplastic BE image4017 Barrett’s neoplasias23 nondysplastic BE16312015/3/2/20 (expert)
      van der Sommen et al (2) (2014)
      • van der Sommen F.
      • Zinger S.
      • Schoon E.J.
      • et al.
      Supportive automatic annotation of early esophageal cancer using local gabor and color features.
      Retrospective/EuropeSVMWLINondysplastic BE6432 early EAC32 nondysplastic BEs324028NA
      CAD, Computer-aided diagnosis; TP, true positive; FP, false positive; FN, false negative; TN, true negative; CNN, convolutional neural network; WLI, white-light imaging; BE, Barrett’s esophagus; NA, not applicable; ME, magnification endoscopy; ESCC, esophageal squamous cell carcinoma; NBI, narrow-band imaging; BLI, blue-light imaging; EAC, esophageal adenocarcinoma; IPCL, intrapapillary capillary loop classification; SVM, support vector machine;
      Supplementary Table 3Clinical characteristics of the included studies (computer-aided diagnosis of invasion depth of esophageal cancers in image-based analysis)
      Study/yearStudy format/nationality (data)Type of CAD algorithmType of endoscopic imageStandard of depth classificationTotal number of imagesNumber of cases in test datasetNumber of controls in test datasetTPFPFNTNPerformance of endoscopists (TP/FP/FN/TN)
      Tokai et al (2020)
      • Tokai Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Application of artificial intelligence using convolutional neural networks in determining the invasion depth of esophageal squamous cell carcinoma.
      Retrospective/AsiaCNNWLIpSM1279189 ESCC confined to EP-SM190 ESCC invaded SM2159243066149/34/40/56
      Nakagawa et al (2020)
      • Nakagawa K.
      • Ishihara R.
      • Aoyama K.
      • et al.
      Classification for invasion depth of esophageal squamous cell carcinoma using a deep neural network compared with experienced endoscopists.
      Retrospective/AsiaCNNWLI (ME or non-ME)pSM1914771 ESCC confined to EP-SM1143 ESCC invaded SM2 or SM3695676137704/15/80/1115 (experienced endoscopists)
      Horie et al (2019)
      • Horie Y.
      • Yoshio T.
      • Aoyama K.
      • et al.
      Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
      Retrospective/AsiaCNNWLI with NBIT1a, T1b vs T2-4168143 superficial esophageal cancers25 advanced esophageal cancers1422123NA
      WLI8975 superficial esophageal cancers14 advanced esophageal cancers750014
      NBI7968 superficial esophageal cancers11 advanced esophageal cancers67219
      CAD, Computer-aided diagnosis; TP, true positive; FP, false positive; FN, false negative; TN, true negative; CNN, convolutional neural network; WLI, white-light imaging; SM, submucosa; ESCC, esophageal squamous cell carcinoma; ME, magnification endoscopy; NBI, narrow-band imaging; NA, not applicable.
      Supplementary Table 4Clinical characteristics of the studies (computer-aided detection in image-based analysis and segmentation of esophageal cancers in patient-based analysis)
      Study/yearStudy format/nationality (data)Type of CAD algorithmType of endoscopic imageType of controlsTotal numberNumber of cases in test datasetNumber of controls in test datasetTPFPFNTNPerformance of endoscopists (TP/FP/FN/TN)
      Ghatwary et al (2019)
      • Ghatwary N.
      • Zolgharni M.
      • Ye X.
      Early esophageal adenocarcinoma detection using deep learning methods.
      Retrospective/public dataR-CNN, Fast R-CNN, Faster R-CNN, SSD for detection taskWLINoncancerous Barrett’s esophagus100 images49 early EAC51 noncancerous Barrett’s esophagus474247NA
      Liu et al (2018)
      • Liu D.
      • Rao N.
      • Mei X.
      • et al.
      Annotating early esophageal cancers based on two saliency levels of gastroscopic images.
      Retrospective/AsiaSimple Linear Iterative Clustering; SLIC for segmentation taskWLIImages without esophageal cancers871434 patients with early esophageal cancers231 patients without esophageal cancer322100112131NA
      CAD, Computer-aided diagnosis; TP, true positive; FP, false positive; FN, false negative; TN, true negative; R-CNN, regional-based convolutional neural network; SSD, single-shot multibox detector; WLI, white-light imaging; EAC, esophageal adenocarcinoma; NA, not applicable.
      Supplementary Table 5Summary of diagnostic test accuracy meta-analysis and subgroup analysis for the diagnosis of esophageal cancers or neoplasms of the studies with patient-based analysis
      SubgroupNumber of studies includedSensitivity (95% CI)Specificity (95% CI)PLRNLRDORAUC
      Value of meta-analysis in all the studies included90.93 (0.86-0.96)0.85 (0.78-0.89)6.0 (4.3-8.4)0.09 (0.05-0.16)69 (35-137)0.94 (0.91-0.96)
      Nationality of data
      Western70.93 (0.86-0.97)0.88 (0.83-0.91)7.7 (5.5-10.7)0.08 (0.04-0.16)96 (44-205)0.93 (0.91-0.95)
      Asia2NullNullNullNullNullNull
      Format of study
      Retrospective60.94 (0.85-0.98)0.86 (0.75-0.92)6.6 (3.8-11.4)0.07 (0.03-0.18)97 (33-287)0.95 (0.93-0.97)
      Prospective3NullNullNullNullNullNull
      Type of artificial intelligence
      CNN60.90 (0.80-0.95)0.84 (0.75-0.90)5.6 (3.7-8.7)0.12 (0.07-0.23)46 (22-93)0.94 (0.91-0.95)
      SVM or decision tree algorithm3NullNullNullNullNullNull
      Type of endoscopic image
      WLI image70.91 (0.86-0.94)0.86 (0.79-0.91)6.4 (4.2-9.6)0.11 (0.07-0.16)60 (35-101)0.95 (0.92-0.96)
      NBI image3NullNullNullNullNullNull
      Type of cases
      Barrett’s neoplasias (including EACs)40.90 (0.85-0.93)0.87 (0.82-0.91)7.1 (5.0-10.1)0.12 (0.08-0.18)61 (33-112)0.95 (0.92-0.96)
      ESCCs/EACs2/3NullNullNullNullNullNull
      Methodological quality of the studies
      High quality50.92 (0.84-0.97)0.82 (0.75-0.88)5.2 (3.8-7.2)0.09 (0.04-0.19)57 (27-119)0.92 (0.90-0.94)
      Low quality40.94 (0.76-0.96)0.90 (0.76-0.96)9.6 (3.8-24.1)0.07 (0.02-0.20)146 (40-538)0.97 (0.95-0.98)
      CI, Confidence interval; PLR, positive likelihood ratio; NLR, negative likelihood ratio; DOR, diagnostic odds ratio; AUC, area under the curve; CNN, convolutional neural network; SVM, support vector machine; WLI, wight-light imaging; NBI, narrow-band imaging; EAC, esophageal adenocarcinoma; ESCC, esophageal squamous cell carcinoma.
      Supplementary Table 6Summary of diagnostic test accuracy meta-analysis and subgroup analysis for the prediction of invasion depth in esophageal cancers of the included studies with patient-based analysis
      SubgroupNumber of studies includedSensitivity (95% CI)Specificity (95% CI)PLRNLRDORAUC
      Value of meta-analysis in all the studies included30.90 (0.88-0.92)0.88 (0.83-0.91)9.1 (1.6-53.5)0.10 (0.04-0.23)138 (12-1569)0.96 (0.86-0.99)
      CI, Confidence interval; PLR, positive likelihood ratio; NLR, negative likelihood ratio; DOR, diagnostic odds ratio; AUC, area under the curve.

      References

        • Bray F.
        • Ferlay J.
        • Soerjomataram I.
        • et al.
        Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries.
        CA Cancer J Clin. 2018; 68: 394-424
        • Klingelhofer D.
        • Zhu Y.
        • Braun M.
        • et al.
        A world map of esophagus cancer research: a critical accounting.
        J Transl Med. 2019; 17: 150
        • Wang Q.L.
        • Xie S.H.
        • Wahlin K.
        • et al.
        Global time trends in the incidence of esophageal squamous cell carcinoma.
        Clin Epidemiol. 2018; 10: 717-728
        • Malhotra G.K.
        • Yanala U.
        • Ravipati A.
        • et al.
        Global trends in esophageal cancer.
        J Surg Oncol. 2017; 115: 564-579
        • Evans J.A.
        • Early D.S.
        • Chandraskhara V.
        • et al.
        The role of endoscopy in the assessment and treatment of esophageal cancer.
        Gastrointest Endosc. 2013; 77: 328-334
        • Codipilly D.C.
        • Chandar A.K.
        • Singh S.
        • et al.
        The effect of endoscopic surveillance in patients with Barrett's esophagus: a systematic review and meta-analysis.
        Gastroenterology. 2018; 154: 2068-2086.e5
        • National Health Commission of the People's Republic of China
        Chinese guidelines for diagnosis and treatment of esophageal carcinoma 2018 (English version).
        Chin J Cancer Res. 2019; 31: 223-258
        • Cho B.J.
        • Bang C.S.
        • Park S.W.
        • et al.
        Automated classification of gastric neoplasms in endoscopic images using a convolutional neural network.
        Endoscopy. 2019; 51: 1121-1129
        • Hamada K.
        • Kawano K.
        • Yamauchi A.
        • et al.
        Efficacy of endoscopic submucosal dissection of esophageal neoplasms under general anesthesia.
        Clin Endosc. 2019; 52: 252-257
        • Kitagawa Y.
        • Uno T.
        • Oyama T.
        • et al.
        Esophageal cancer practice guidelines 2017 edited by the Japan Esophageal Society: part 1.
        Esophagus. 2019; 16: 1-24
        • Cho B.J.
        • Bang C.S.
        artificial intelligence for the determination of a management strategy for diminutive colorectal polyps: hype, hope, or help.
        Am J Gastroenterol. 2020; 115: 70-72
        • Whiting P.F.
        • Rutjes A.W.
        • Westwood M.E.
        • et al.
        QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies.
        Ann Intern Med. 2011; 155: 529-536
        • McInnes M.D.F.
        • Moher D.
        • Thombs B.D.
        • et al.
        Preferred Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies: the PRISMA-DTA statement.
        JAMA. 2018; 319: 388-396
        • Reitsma J.B.
        • Glas A.S.
        • Rutjes A.W.
        • et al.
        Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews.
        J Clin Epidemiol. 2005; 58: 982-990
        • Rutter C.M.
        • Gatsonis C.A.
        A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations.
        Stat Med. 2001; 20: 2865-2884
        • Harbord R.M.
        • Whiting P.
        Metandi: meta-analysis of diagnostic accuracy using hierarchical logistic regression.
        Stata J. 2009; 9: 211-229
        • de Groof A.J.
        • Struyvenberg M.R.
        • Fockens K.N.
        • et al.
        Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video).
        Gastrointest Endosc. 2020; 91: 1242-1250
        • Guo L.
        • Xiao X.
        • Wu C.
        • et al.
        Real-time automated diagnosis of precancerous lesions and early esophageal squamous cell carcinoma using a deep learning model (with videos).
        Gastrointest Endosc. 2020; 91: 41-51
        • Garcia-Peraza-Herrera L.C.
        • Everson M.
        • Lovat L.
        • et al.
        Intrapapillary capillary loop classification in magnification endoscopy: open dataset and baseline methodology.
        Int J Comput Assist Radiol Surg. 2020; 15: 651-659
        • Hashimoto R.
        • Requa J.
        • Dao T.
        • et al.
        Artificial intelligence using convolutional neural networks for real-time detection of early esophageal neoplasia in Barrett's esophagus (with video).
        Gastrointest Endosc. 2020; 91: 1264-1271.e1
        • de Groof A.J.
        • Struyvenberg M.R.
        • van der Putten J.
        • et al.
        Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking.
        Gastroenterology. 2020; 158: 915-929.e4
        • Everson M.
        • Herrera L.
        • Li W.
        • et al.
        Artificial intelligence for the real-time classification of intrapapillary capillary loop patterns in the endoscopic diagnosis of early oesophageal squamous cell carcinoma: a proof-of-concept study.
        United European Gastroenterol J. 2019; 7: 297-306
        • Cai S.L.
        • Li B.
        • Tan W.M.
        • et al.
        Using a deep learning system in endoscopy for screening of early esophageal squamous cell carcinoma (with video).
        Gastrointest Endosc. 2019; 90: 745-753.e2
        • Horie Y.
        • Yoshio T.
        • Aoyama K.
        • et al.
        Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.
        Gastrointest Endosc. 2019; 89: 25-32
        • Liu D.Y.
        • Gan T.
        • Rao N.N.
        • et al.
        Identification of lesion images from gastrointestinal endoscope based on feature extraction of combinational methods with and without learning process.
        Med Image Anal. 2016; 32: 281-294
        • van der Sommen F.
        • Zinger S.
        • Curvers W.L.
        • et al.
        Computer-aided detection of early neoplastic lesions in Barrett's esophagus.
        Endoscopy. 2016; 48: 617-624
        • Ohmori M.
        • Ishihara R.
        • Aoyama K.
        • et al.
        Endoscopic detection and differentiation of esophageal lesions using a deep neural network.
        Gastrointest Endosc. 2020; 91: 301-309.e1
        • Ebigbo A.
        • Mendel R.
        • Probst A.
        • et al.
        Real-time use of artificial intelligence in the evaluation of cancer in Barrett's oesophagus.
        Gut. 2020; 69: 615-616
        • Zhao Y.Y.
        • Xue D.X.
        • Wang Y.L.
        • et al.
        Computer-assisted diagnosis of early esophageal squamous cell carcinoma using narrow-band imaging magnifying endoscopy.
        Endoscopy. 2019; 51: 333-341
        • Ebigbo A.
        • Mendel R.
        • Probst A.
        • et al.
        Computer-aided diagnosis using deep learning in the evaluation of early oesophageal adenocarcinoma.
        Gut. 2019; 68: 1143-1145
        • de Groof J.
        • van der Sommen F.
        • van der Putten J.
        • et al.
        The Argos project: the development of a computer-aided detection system to improve detection of Barrett's neoplasia on white light endoscopy.
        United European Gastroenterol J. 2019; 7: 538-547
        • Sehgal V.
        • Rosenfeld A.
        • Graham D.
        • et al.
        Machine learning creates a simple endoscopic classification system that improves dysplasia detection in Barrett's oesophagus amongst non-expert endoscopists.
        Gastroenterol Res Pract. 2018; 2018: 1872437
        • van der Sommen F.
        • Zinger S.
        • Schoon E.J.
        • et al.
        Supportive automatic annotation of early esophageal cancer using local gabor and color features.
        Neurocomputing. 2014; 144: 92-106
        • Tokai Y.
        • Yoshio T.
        • Aoyama K.
        • et al.
        Application of artificial intelligence using convolutional neural networks in determining the invasion depth of esophageal squamous cell carcinoma.
        Esophagus. 2020; 17: 250-256
        • Nakagawa K.
        • Ishihara R.
        • Aoyama K.
        • et al.
        Classification for invasion depth of esophageal squamous cell carcinoma using a deep neural network compared with experienced endoscopists.
        Gastrointest Endosc. 2019; 90: 407-414
        • Ghatwary N.
        • Zolgharni M.
        • Ye X.
        Early esophageal adenocarcinoma detection using deep learning methods.
        Int J Comput Assist Radiol Surg. 2019; 14: 611-621
        • Liu D.
        • Rao N.
        • Mei X.
        • et al.
        Annotating early esophageal cancers based on two saliency levels of gastroscopic images.
        J Med Syst. 2018; 42: 237
        • Lee Y.H.
        Overview of the process of conducting meta-analyses of the diagnostic test accuracy.
        J Rheum Dis. 2018; 25: 3-10
        • Deeks J.J.
        Systematic reviews in health care: systematic reviews of evaluations of diagnostic and screening tests.
        BMJ. 2001; 323: 157-162
        • Okeh U.
        • Okoro C.N.
        Evaluating measures of indicators of diagnostic test performance: fundamental meanings and formulars.
        J Biomet Biostat. 2012; 3: 2
        • di Pietro M.
        • Fitzgerald R.C.
        Revised British Society of Gastroenterology recommendation on the diagnosis and management of Barrett's oesophagus with low-grade dysplasia.
        Gut. 2018; 67: 392-393
        • Qumseya B.
        • Sultan S.
        • Bain P.
        • et al.
        ASGE guideline on screening and surveillance of Barrett's esophagus.
        Gastrointest Endosc. 2019; 90: 335-359.e2
        • Saftoiu A.
        • Hassan C.
        • Areia M.
        • et al.
        Role of gastrointestinal endoscopy in the screening of digestive tract cancers in Europe: European Society of Gastrointestinal Endoscopy (ESGE) Position Statement.
        Endoscopy. 2020; 52: 293-304
        • Shaheen N.J.
        • Falk G.W.
        • Iyer P.G.
        • et al.
        ACG Clinical Guideline: diagnosis and management of Barrett's esophagus.
        Am J Gastroenterol. 2016; 111 (quiz 51): 30-50
        • Hamel C.
        • Ahmadzai N.
        • Beck A.
        • et al.
        Screening for esophageal adenocarcinoma and precancerous conditions (dysplasia and Barrett's esophagus) in patients with chronic gastroesophageal reflux disease with or without other risk factors: two systematic reviews and one overview of reviews to inform a guideline of the Canadian Task Force on Preventive Health Care (CTFPHC).
        Syst Rev. 2020; 9: 20
        • Lui T.K.L.
        • Tsui V.W.M.
        • Leung W.K.
        Accuracy of artificial intelligence-assisted detection of upper GI lesions: a systematic review and meta-analysis.
        Gastrointest Endosc. 2020; 92: 821-830.e9
        • Ishihara R.
        • Takeuchi Y.
        • Chatani R.
        • et al.
        Prospective evaluation of narrow-band imaging endoscopy for screening of esophageal squamous mucosal high-grade neoplasia in experienced and less experienced endoscopists.
        Dis Esophagus. 2010; 23: 480-486
        • Sharma P.
        • Savides T.J.
        • Canto M.I.
        • et al.
        The American Society for Gastrointestinal Endoscopy PIVI (Preservation and Incorporation of Valuable Endoscopic Innovations) on imaging in Barrett's Esophagus.
        Gastrointest Endosc. 2012; 76: 252-254