Advertisement

Quality assessment for systematic reviews and meta-analyses of cohort studies

  • Bashar J. Qumseya
    Correspondence
    Reprint requests: Bashar Qumseya, MD, MPH, Division of Gastroenterology, Hepatology and Nutrition, University of Florida, PO Box 100214, 1329 SW 16th St, Ste 5251, Gainesville, FL 32610-0214.
    Affiliations
    Division of Gastroenterology, Hepatology and Nutrition, University of Florida, Gainesville, Florida, USA
    Search for articles by this author
Published:October 14, 2020DOI:https://doi.org/10.1016/j.gie.2020.10.007

      Background and Aims

      There is a growing need for valid, efficient, and easy scoring scales to rate the quality of cohort studies. We aimed to develop and validate a quality assessment score to be used for cohort studies.

      Methods

      We followed a rigorous process to establish content, face, and construct validity. Most questions were scored at 0 or 1. Inter-rater reliability and test–retest reliability were assessed using the Spearman correlation coefficient (rs) and Cohen’s κ statistic. Internal consistency was measured using the Kuder-Richardson formula 20 (KR20).

      Results

      The final tool consists of 9 questions with a maximum score of 10. The inter-rater reliability was high with the Spearman correlation coefficient (rs = .66). Agreement for inclusion was 90%. Test–retest reliability was high. For rater 1, rs = .91, κ = .38 for scores, and κ = 1 for inclusion. For rater 2, r = .94, 80% agreement for scores, and 100% agreement for inclusion. Internal consistency was reasonable based on 2 studies: KR20 = .21 and KR20 = .65. The novel scale rated highest in efficiency, understandably, ease of use, and ease of interpretation when compared with 3 other scales.

      Conclusions

      This novel scale has favorable performance characteristics, is efficient to conduct, and is easy to interpret and will be very helpful for physicians and researchers conducting systematic reviews and meta-analyses.

      Abbreviations:

      BE (Barrett’s esophagus), KR20 (Kuder-Richardson formula 20), OR (odds ratio)
      In recent years, clinicians have come to recognize that medical practices need to be based on systematic review of all available evidence.
      • Johnston A.
      • Kelly S.E.
      • Hsieh S.C.
      • et al.
      Systematic reviews of clinical practice guidelines: a methodological guide.
      In 2011, the Institute of Medicine, now the National Academy of Medicine, defined guidelines as “recommendations intended to optimize patient care that are informed by a systematic review of evidence.”(p. 25)
      Institute of Medicine Committee on Standards for Developing Trustworthy Clinical Practice Guidelines.
      This evidence-based approach has become a critical aspect of the practice of medicine. The process is transparent, based on rigorous methodology, and reproducible. Adopting evidence-based guidelines has changed the way physicians, medical societies, and guideline panels use the medical literature to make clinical decisions.
      • Guyatt G.H.
      • Oxman A.D.
      • Vist G.E.
      • et al.
      GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.
      Performing systematic reviews and meta-analyses are key components of all medical guidelines at this point.
      • Gopalakrishnan S.
      • Ganeshkumar P.
      Systematic reviews and meta-analysis: understanding the best evidence in primary healthcare.
      An integral part of conducting a systematic review is to rate the quality of individual articles.
      • Sahni M.
      • Goel A.
      • Pande P.
      • et al.
      Multiple gastrointestinal cancers in a single patient—a rare clinical entity.
      For this purpose, many scoring systems have been developed and validated.
      • Clark H.D.
      • Wells G.A.
      • Huet C.
      • et al.
      Assessing the quality of randomized trials: reliability of the Jadad scale.
      ,
      • Savovic J.
      • Weeks L.
      • Sterne J.A.
      • et al.
      Evaluation of the Cochrane Collaboration's tool for assessing the risk of bias in randomized trials: focus groups, online survey, proposed recommendations and their implementation.
      However, the use of such quality scores has been fraught with several shortcomings. First, a large gap remains in the area of scoring cohort studies. These include prospective or retrospective studies, which were not designed as randomized trials. Although many scoring systems exist for randomized trails,
      • Olivo S.A.
      • Macedo L.G.
      • Gadotti I.C.
      • et al.
      Scales to assess the quality of randomized controlled trials: a systematic review.
      only a few scores have been developed for cohort studies.
      • Downs S.H.
      • Black N.
      The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions.
      • Hartling L.
      • Milne A.
      • Hamm M.P.
      • et al.
      Testing the Newcastle-Ottawa scale showed low reliability between individual reviewers.
      • Sterne J.A.
      • Hernan M.A.
      • Reeves B.C.
      • et al.
      ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.
      Second, previous scales can be complex to use, score, and interpret. Last, many scoring tools did little to address specific concerns relating to the process of meta-analysis. We believe the ideal quality assessment scale for cohort studies should be valid, efficient, and easy to use and interpret. Therefore, the aim of this study was to develop and validate a quality assessment score to be used for cohort studies in medical literature.

      Methods

      Pilot version

      The initial version of the study was conceptualized based on our experience in conducting systematic reviews and meta-analyses and guideline development.
      • Qumseya B.
      • Gendy S.
      • Wallace A.
      • et al.
      Prevalence of Barrett's esophagus in obese patients undergoing pre-bariatric surgery evaluation: a systematic review and meta-analysis.
      • Qumseya B.J.
      • Bukannan A.
      • Gendy S.
      • et al.
      Systematic review and meta-analysis of prevalence and risk factors for Barrett's esophagus.
      • Aghaie Meybodi M.
      • Qumseya B.J.
      • Shakoor D.
      • et al.
      Efficacy and feasibility of G-POEM in management of patients with refractory gastroparesis: a systematic review and meta-analysis.
      • Qumseya B.J.
      • Bartel M.J.
      • Gendy S.
      • et al.
      High rate of over-staging of Barrett's neoplasia with endoscopic ultrasound: systemic review and meta-analysis.
      • Keswani R.N.
      • Qumseya B.J.
      • O'Dwyer L.C.
      • et al.
      Association between endoscopist and center endoscopic retrograde cholangiopancreatography volume with procedure success and adverse outcomes: a systematic review and meta-analysis.
      • Qumseya B.J.
      • Wani S.
      • Gendy S.
      • et al.
      Disease progression in Barrett's low-grade dysplasia with radiofrequency ablation compared with surveillance: systematic review and meta-analysis.
      • Qumseya B.J.
      • Wani S.
      • Desai M.
      • et al.
      Adverse events after radiofrequency ablation in patients with barrett's esophagus: a systematic review and meta-analysis.
      • Qumseya B.J.
      • Wang H.
      • Badie N.
      • et al.
      Advanced imaging technologies increase detection of dysplasia and neoplasia in patients with Barrett's esophagus: a meta-analysis and systematic review.
      • Kothari S.T.
      • Huang R.J.
      • Shaukat A.
      • et al.
      ASGE review of adverse events in colonoscopy.
      • Qumseya B.
      • Sultan S.
      • et al.
      ASGE Standards of Practice Committee
      ASGE guideline on screening and surveillance of Barrett's esophagus.
      • Wani S.
      • Qumseya B.
      • et al.
      Standards of Practice Committee
      Endoscopic eradication therapy for patients with Barrett's esophagus-associated dysplasia and intramucosal cancer.
      The pilot version was developed based on common concerns regarding study quality as encountered by physicians and raters who review such literature. The pilot tool consisted of 10 questions. The final tool consisted of 9 questions. The tool design allowed the rater to identify the study being rated (author, year), identify the reviewer (by initials), state the final score (out of 10), and use that score to qualify the study as high, moderate, or poor quality. The maximum score was 10. Studies scoring less than 6 were considered low quality, 6 to 7 moderate quality, and 8 to 10 high quality (Figure 1).
      Figure thumbnail gr1
      Figure 1Quality assessment for meta-analysis scoring items (Qumseya scale).
      The final tool rates studies based on 4 domains:
      • 1.
        Study design: This consisted of 2 questions assessing study design. Prospective studies and multicenter studies received higher points on the scale.
      • 2.
        Outcomes and their measurements: This consisted of 3 questions. The first question assessed whether the primary outcome of the study was the same as the primary outcome of the systematic review. The second question assessed validity of the primary outcome, and the third question assessed whether procedures and interventions were described clearly.
      • 3.
        Included patients and their characteristics: This consisted of 3 questions regarding patients included in the study, the proportion not included or who refused to be included, and a description of patient characteristics.
      • 4.
        Identifying outliers: This consisted of 1 question assessing whether the study under review could represent an outlier. This is the only question that may require answering after the meta-analysis was already conducted.
      Most questions were scored at 0 or 1. Only 1 question had a score of 2 for multicenter studies and 1 for single-center studies.

      Pilot tool validation

      We used our previously described process
      • Qumseya B.J.
      • Tayem Y.I.
      • Dasa O.Y.
      • et al.
      Barriers to colorectal cancer screening in Palestine: a national study in a medically underserved population.
      to establish face, content, and construct validity.

      First step

      An expert physician and methodologist created the initial tool. Two physicians with expertise in systematic reviews and meta-analyses were asked to review the tool and comment on the wording, content, and appropriateness of each item in the tool. The tool was modified based on the initial input.

      Second step

      We identified 5 prospective cohort studies
      • Genco A.
      • Soricelli E.
      • Casella G.
      • et al.
      Gastroesophageal reflux disease and Barrett's esophagus after laparoscopic sleeve gastrectomy: a possible, underestimated long-term complication.
      • Tai C.M.
      • Huang C.K.
      • Lee Y.C.
      • et al.
      Increase in gastroesophageal reflux disease symptoms and erosive esophagitis 1 year after laparoscopic sleeve gastrectomy among obese adults.
      • Braghetto I.
      • Korn O.
      • Csendes A.
      • et al.
      Laparoscopic treatment of obese patients with gastroesophageal reflux disease and Barrett's esophagus: a prospective study.
      • Braghetto I.
      • Csendes A.
      Prevalence of Barrett's esophagus in bariatric patients undergoing sleeve gastrectomy.
      • Soricelli E.
      • Casella G.
      • Baglio G.
      • et al.
      Lack of correlation between gastroesophageal reflux disease symptoms and esophageal lesions after sleeve gastrectomy.
      and 5 retrospective cohort studies.
      • Lim C.H.
      • Lee P.C.
      • Lim E.
      • et al.
      Correlation between symptomatic gastro-esophageal reflux disease (GERD) and erosive esophagitis (EE) post-vertical sleeve gastrectomy (VSG).
      • Houghton S.G.
      • Romero Y.
      • Sarr M.G.
      Effect of Roux-en-Y gastric bypass in obese patients with Barrett's esophagus: attempts to eliminate duodenogastric reflux.
      • Sebastianelli L.
      • Benois M.
      • Vanbiervliet G.
      • et al.
      Systematic endoscopy 5 years after sleeve gastrectomy results in a high rate of Barrett’s esophagus: results of a multicenter study.
      • Felsenreich D.M.
      • Kefurt R.
      • Schermann M.
      • et al.
      Reflux, sleeve dilation, and Barrett’s esophagus after laparoscopic sleeve gastrectomy: long-term follow-up.
      • Andrew B.
      • Alley J.B.
      • Aguilar C.E.
      • et al.
      Barrett’s esophagus before and after Roux-en-Y gastric bypass for severe obesity.
      These studies were randomly identified based on a literature search to answer the following clinical question: What is the prevalence of Barrett’s esophagus (BE) in patients you have bariatric surgery? We asked 2 independent reviewers to use the checklist and rate each study. They were then given a debriefing questionnaire to comment on the ease of use, understandability, and clarity of each item in the checklist. We used feedback from the debriefing questionnaires to make any necessary modifications.

      Third step

      Three additional independent reviewers, who were not part of the initial process, were asked to rate 5 prospective and 5 retrospective cohort studies using the modified tool. For each questions we assessed inter-rater agreement. Cohen’s κ correlation coefficient was used when applicable. If any question scored ≤60% agreement, or κ was <.3, the question was flagged. Raters were asked to comment on the flagged questions, and modifications were made or the question was removed from the tool.

      Fourth step

      The final tool was given to 2 independent raters who were asked to rate the same 5 prospective and 5 retrospective cohort studies. These raters were not involved in any of the previous validation steps. Inter-rater reliability was calculated using the κ correlation coefficient. This was done for each question. Three weeks later, the same raters were asked to rate the same articles without going back to their previous notes. We used these responses to assess test–retest reliability.

      Comparison with existing scales

      After completing the validation process and to compare our novel scale with existing scales, we used an existing systematic review
      • Facciorusso A.
      • Del Prete V.
      • Antonino M.
      • et al.
      Diagnostic yield of EUS-guided through-the-needle biopsy in pancreatic cysts: a meta-analysis.
      of 11 cohort studies published in 2020 in Gastrointestinal Endoscopy. Three studies were abstracts. Of the remaining 8 studies, 5 studies
      • Kothari S.T.
      • Huang R.J.
      • Shaukat A.
      • et al.
      ASGE review of adverse events in colonoscopy.
      ,
      • Basar O.
      • Yuksel O.
      • Yang D.J.
      • et al.
      Feasibility and safety of microforceps biopsy in the diagnosis of pancreatic cysts.
      • Crino S.F.
      • Bernardoni L.
      • Brozzi L.
      • et al.
      Association between macroscopically visible tissue samples and diagnostic accuracy of EUS-guided through-the-needle microforceps biopsy sampling of pancreatic cystic lesions.
      • Kovacevic B.
      • Karstensen J.G.
      • Havre R.F.
      • et al.
      Initial experience with EUS-guided microbiopsy forceps in diagnosing pancreatic cystic lesions: a multicenter feasibility study (with video).
      • Samarasena J.
      • Yu A.
      • Lee D.
      • et al.
      EUS-guided through-the-needle biopsy for pancreatic cystic lesions.
      were randomly selected to be rated. These included 2 prospective and 3 retrospective studies assessing the diagnostic yield of EUS-guided through-the-needle biopsy sampling of pancreatic cysts.
      We chose an independent reviewer who was given 4 scales to use: Newcastle-Ottawa,
      • Hartling L.
      • Milne A.
      • Hamm M.P.
      • et al.
      Testing the Newcastle-Ottawa scale showed low reliability between individual reviewers.
      Down and Black,
      • Downs S.H.
      • Black N.
      The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions.
      ROBINS-I,
      • Sterne J.A.
      • Hernan M.A.
      • Reeves B.C.
      • et al.
      ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.
      and Qumseya scales. The independent reviewer had no prior experience with any of the existing tools and was blinded to the names of the scales, goals of the study, or that 1 of the tools was a novel scale. The scales were numbered 1 through 4. The reviewer was instructed to read the 5 studies and rate the quality of each study using each of the 4 scales. The reviewer was asked to keep track of the time it took to do the rating and the number of questions that were not applicable in each scale. The reviewer was then asked to comment on the understandability, ease of use, ease of interpretation, time of use, and the overall rank of the scale.

      Statistical analysis

      Scores were reported for individual studies. Mean scores and standard deviations were reported for each rater as mean ± standard deviation. The mean scores from raters were compared using either the Student t test for paired data or the Student t test for independent groups, as appropriate, with P < .05 used for significance. Inter-rater reliability and test–retest reliability were evaluated using Cohen's κ statistic, a chance-corrected measure of agreement, and the Spearman rank correlation coefficient (rs), respectively. Spearman rs was characterized as follows: 0 to .19 as very weak, .2 to .39 as weak, .4 to .59 as moderate, .6 to .79 as strong, and .8 to 1.0 as very strong correlation. Cohen’s κ statistic was characterized as follows: >.8 as almost perfect, .61 to .80 as substantial, .41 to .60 as moderate, .21 to .40 as fair, and .01 to .20 as none to slight agreement. Cohen’s κ was reported for agreement between scores and agreement for inclusion (based on the final score). Internal consistency was measured using the Kuder-Richardson formula 20 (KR20).

      Results

      The initial scale consisted of 9 questions. Based on feedback from 2 expert physicians, a question was added to address the number of centers in the study (single vs multiple centers). Additionally, wording was changed in 3 questions (numbers 7-9 on the initial scale). This modified scale had 10 questions.
      In the second validation step, 2 raters, physicians in fellowship, were asked to rate 5 prospective and 5 retrospective studies. They had to complete a debriefing questionnaire afterward. Based on their feedback, the wording on questions 4, 6, 7, and 9 was modified.
      In the third step, the scale resulting from the second validation step was given to 3 physicians to rate 10 studies. Agreement between raters was high for questions 1 to 5 and question 10, and therefore these were included in the final scale. Agreement was fair in question 6 (60%, 70%, and 70%), and this was also included. Agreement was poor in questions 7, 8, and 9. Many raters chose “unclear” for question 9. Therefore, question 9 was removed from the final scale. Questions 7 and 8 were modified to address potential ambiguity. Agreement and κ between the 3 raters are reported in Figure 2 and Table 1. The final scale consisted of 9 questions in 4 domains: study design, outcomes and their measurements, included patients and their characteristics, and identifying outliers.
      Figure thumbnail gr2
      Figure 2Agreement among 3 raters based on the pilot tool.
      Table 1Agreement and Cohen’s κ coefficient for the 3 raters


      Question
      Agreement (%)Cohen’s κ
      Raters 1 and 2Raters 1 and 3Raters 2 and 3Raters 1 and 2Raters 1 and 3Raters 2 and 3
      Question 1809090.62.8.8
      Question 2100100100111
      Question 3709070
      Question 4100100100
      Question 58080100
      Question 6706070
      Question 7407050.10
      Question 8606370.21.35
      Question 95060400–.2
      Question 10100100100

      Basic summary

      For the final scale (Figure 1), the mean score for prospective studies was 6.8 ± .8 and for retrospective studies was 7.4 ± 1.3. The difference was not statistically significant (P = .11). Scores for each of the studies are summarized in Figure 3.
      Figure thumbnail gr3
      Figure 3Scores for each of the studies based on 2 raters done 3 weeks apart.

      Inter-rater reliability

      The mean score of the first rater was 6.8 ± 1.0 and of the second rater was 7.2 ± .9. The difference was not statistically significant (P = .168). The inter-rater reliability was high, with a Spearman rank correlation coefficient r = .66. Agreement for inclusion was 90%. One rater scored Lim et al
      • Lim C.H.
      • Lee P.C.
      • Lim E.
      • et al.
      Correlation between symptomatic gastro-esophageal reflux disease (GERD) and erosive esophagitis (EE) post-vertical sleeve gastrectomy (VSG).
      at 5, whereas the other rater scored the study at 6.

      Test–retest reliability

      Test–retest reliability was high. For rater 1, rs = .91, κ = .38 for scores and κ = 1 for inclusion. For rater 2, r = .94, 80% agreement for scores, and 100% agreement for inclusion. Agreement and correlation by question for each rater are summarized in Table 2.
      Table 2Agreement and Cohen’s κ correlation for inter-rater and intrarater agreement for each question in the final scale


      Question
      Inter-rater agreementIntrarater agreement (rater 1)Intrarater agreement (rater 2)
      Agreement (%)κAgreement (%)κAgreement (%)κ
      1100180.81001
      2100110011001
      3100110011001
      41001100NA100NA
      5100NA100NA100NA
      680.55100180.55
      770.21100190.74
      890.7410011001
      980NA80NA100NA
      NA, not applicable.

      Internal consistency

      To check for internal consistency, we calculated the KR20 using 1 retrospective study
      • Lim C.H.
      • Lee P.C.
      • Lim E.
      • et al.
      Correlation between symptomatic gastro-esophageal reflux disease (GERD) and erosive esophagitis (EE) post-vertical sleeve gastrectomy (VSG).
      and 1 prospective study.
      • Genco A.
      • Soricelli E.
      • Casella G.
      • et al.
      Gastroesophageal reflux disease and Barrett's esophagus after laparoscopic sleeve gastrectomy: a possible, underestimated long-term complication.
      For this analysis, we used ratings from the 5 raters in validation steps 3 and 4. For the prospective study by Genco et al,
      • Genco A.
      • Soricelli E.
      • Casella G.
      • et al.
      Gastroesophageal reflux disease and Barrett's esophagus after laparoscopic sleeve gastrectomy: a possible, underestimated long-term complication.
      KR20 was .21. Once removing the third domain, KR20 increased to .64, indicating high internal consistency. For the retrospective study by Lim et al,
      • Lim C.H.
      • Lee P.C.
      • Lim E.
      • et al.
      Correlation between symptomatic gastro-esophageal reflux disease (GERD) and erosive esophagitis (EE) post-vertical sleeve gastrectomy (VSG).
      KR20 was .65, also indicating high internal consistency.

      Understandability, ease of use, interpretation, and efficiency

      The meta-analysis by Facciorusso et al
      • Facciorusso A.
      • Del Prete V.
      • Antonino M.
      • et al.
      Diagnostic yield of EUS-guided through-the-needle biopsy in pancreatic cysts: a meta-analysis.
      assessed the diagnostic yield of EUS-guided through-the-needle biopsy sampling of pancreatic cyst. This study included 3 abstracts and 8 full-text studies. Of the 8 full-text studies, 5 were randomly
      • Basar O.
      • Yuksel O.
      • Yang D.J.
      • et al.
      Feasibility and safety of microforceps biopsy in the diagnosis of pancreatic cysts.
      • Crino S.F.
      • Bernardoni L.
      • Brozzi L.
      • et al.
      Association between macroscopically visible tissue samples and diagnostic accuracy of EUS-guided through-the-needle microforceps biopsy sampling of pancreatic cystic lesions.
      • Kovacevic B.
      • Karstensen J.G.
      • Havre R.F.
      • et al.
      Initial experience with EUS-guided microbiopsy forceps in diagnosing pancreatic cystic lesions: a multicenter feasibility study (with video).
      • Samarasena J.
      • Yu A.
      • Lee D.
      • et al.
      EUS-guided through-the-needle biopsy for pancreatic cystic lesions.
      • Yang D.
      • Trindade A.J.
      • Yachimski P.
      • et al.
      Histologic analysis of endoscopic ultrasound-guided through the needle microforceps biopsies accurately identifies mucinous pancreas cysts.
      selected and assessed using 4 scales by a blinded reviewer. The reviewer was blinded to the names of the scales, the goal of the study, and that 1 of the scales was novel and developed by the study author. Our scale rated best (10/10) in terms understandability, ease of use, and ease of interpretation (Table 3). Our scale was also ranked highest in preference followed by Newcastle-Ottawa, Down and Black, and ROBINS-I. Furthermore, our scale ranked best in efficiency, taking the least amount of time to fill out, averaging 2.2 ± 1.1 minutes. This was followed by Newcastle-Ottawa (2.7 ± .5 minutes), Downs and Black (6.6 ± 1.1 minutes), and ROBINS-I (19.4 ± 1.6 minutes). None of the questions in the Qumseya or Newcastle-Ottawa scales were found to be nonapplicable. Down and Black and ROBINS-I had several nonapplicable questions in each study. Further details on comparing the 4 scales are shown in Table 4.
      Table 3Understandability, ease of use, ease of interpretation, and overall score for each scale based in a blinded reviewer
      Newcastle-OttawaDown and BlackROBINS-IQumseya
      Understandability: How easy was it for you to understand the format, layout, and wording of the scale? (0-10: 0, difficult to understand; 10, easy to understand)86210
      Ease of use: How easy was this scale to apply to each of the studies? (0-10: 0, difficult ease of use; 10, easy to use)85110
      Ease of interpretation: How easy was it to classify studies based on their quality using each specific scale? (0-10: 0, difficult; 10, easy)86010
      Overall: Which scale you would use if you had the choice; in ascending order (4 is best and 1 is worst)3214
      Table 4Time it took to complete the 4 scales and the number of nonapplicable questions for 5 randomly selected studies


      Reference
      Time (min)Number of nonapplicable questions
      Newcastle-OttawaDown and BlackROBINS-IQumseyaNewcastle-OttawaDown and BlackROBINS-IQumseya
      Yang et al
      • Yang D.
      • Trindade A.J.
      • Yachimski P.
      • et al.
      Histologic analysis of endoscopic ultrasound-guided through the needle microforceps biopsies accurately identifies mucinous pancreas cysts.
      2.86.522.21.50180
      Samarasena et al
      • Samarasena J.
      • Yu A.
      • Lee D.
      • et al.
      EUS-guided through-the-needle biopsy for pancreatic cystic lesions.
      3.35.118.54.10230
      Crino et al
      • Crino S.F.
      • Bernardoni L.
      • Brozzi L.
      • et al.
      Association between macroscopically visible tissue samples and diagnostic accuracy of EUS-guided through-the-needle microforceps biopsy sampling of pancreatic cystic lesions.
      3.26.519.11.80520
      Kovacevic et al
      • Kovacevic B.
      • Karstensen J.G.
      • Havre R.F.
      • et al.
      Initial experience with EUS-guided microbiopsy forceps in diagnosing pancreatic cystic lesions: a multicenter feasibility study (with video).
      2.28.219.21.80240
      Basar et al
      • Basar O.
      • Yuksel O.
      • Yang D.J.
      • et al.
      Feasibility and safety of microforceps biopsy in the diagnosis of pancreatic cysts.
      2.26.818.22.10140
      Average ± standard deviation2.7 ± .56.6 ± 1.119.4 ± 1.62.2 ± 1.102.24.20
      The reviewer noted that the ROBINS-I was the most difficult to interpret, took the most amount of time to complete, and had several nonapplicable questions. Of the 3 scales, the Newcastle-Ottawa was the closest to our scale in terms of efficiency, understandability, and ease of use. However, interpretation of the Newcastle-Ottawa scale was more difficult than our scale because each question was given 1 or 2 stars, which was not very intuitive.

      Interscale agreement

      We found 100% agreement between our scale and the ROBINS-I: 1 study was rated high quality (corresponding to low risk of bias on ROBINS-I), and 4 studies were rated moderate quality (corresponding to moderate risk of bias on ROBINS-I) (Table 5). When a cutoff of ≥7 stars for high quality and 5 to 6 stars for moderate quality was used, all rated studies were rated as high quality on the Newcastle-Ottawa scale. However, considering agreement for inclusion, where moderate and high quality would be included in a meta-analysis, the agreement between the Newcastle-Ottawa and our scale would be 100%. Finally, the Down and Black scale resulted in low quality in most studies, largely because of nonapplicable questions. Details on the quality assessment of the 5 studies using the various scores are shown in Table 5.
      Table 5Quality score of 5 randomly selected studies using 4 scales


      Reference
      Quality score in published studyQuality score
      Newcastle-OttawaNewcastle-OttawaDowns and BlackROBINS-IQumseya
      Yang et al
      • Yang D.
      • Trindade A.J.
      • Yachimski P.
      • et al.
      Histologic analysis of endoscopic ultrasound-guided through the needle microforceps biopsies accurately identifies mucinous pancreas cysts.
      High (7)High (8)Low (16)HighHigh (9)
      Samarasena et al
      • Samarasena J.
      • Yu A.
      • Lee D.
      • et al.
      EUS-guided through-the-needle biopsy for pancreatic cystic lesions.
      Moderate (4)High (7)Low (13)ModerateModerate (6)
      Crino et al
      • Crino S.F.
      • Bernardoni L.
      • Brozzi L.
      • et al.
      Association between macroscopically visible tissue samples and diagnostic accuracy of EUS-guided through-the-needle microforceps biopsy sampling of pancreatic cystic lesions.
      High (7)High (8)Low (19)ModerateModerate (6)
      Kovacevic et al
      • Kovacevic B.
      • Karstensen J.G.
      • Havre R.F.
      • et al.
      Initial experience with EUS-guided microbiopsy forceps in diagnosing pancreatic cystic lesions: a multicenter feasibility study (with video).
      Low (3)High (8)Low (17)ModerateModerate (7)
      Basar et al
      • Basar O.
      • Yuksel O.
      • Yang D.J.
      • et al.
      Feasibility and safety of microforceps biopsy in the diagnosis of pancreatic cysts.
      High (7)High (8)Mod (21)ModerateModerate (7)

      Effect of down-rating

      We wanted to assess the effect of the post-hoc question in our scale on the results of the published meta-analysis. In this meta-analysis,
      • Facciorusso A.
      • Del Prete V.
      • Antonino M.
      • et al.
      Diagnostic yield of EUS-guided through-the-needle biopsy in pancreatic cysts: a meta-analysis.
      7 studies were included comparing the sample adequacy between microforceps biopsy sampling and standard FNA. Two studies, by Crino et al
      • Crino S.F.
      • Bernardoni L.
      • Brozzi L.
      • et al.
      Association between macroscopically visible tissue samples and diagnostic accuracy of EUS-guided through-the-needle microforceps biopsy sampling of pancreatic cystic lesions.
      and Cheesman et al,
      • Cheesman A.R.
      • Zhu H.
      • Kumta N.A.
      • et al.
      EUS-guided microforceps biopsy and needle-based confocal laser endomicroscopy significantly improve the diagnostic yield and have major impact on clinical management of pancreatic cystic lesions [abstract].
      met our criteria for down-rating because of potential outliers. The odds ratios (ORs) reported for Crino et al was 154 and for Cheesman et al was 38. The pooled OR was 4.83. Therefore, both studies have an effect estimate, which is >8 to 10 times the expected effect estimate.
      Crino et al scored 6 on our scale before down-rating. Once a point was deducted for question 9, the final score was 5 and thus would be of low quality. The study by Cheesman et al is an abstract, which also scored 5 on our scale and was rated as low quality in the published meta-analysis. In the 7 studies,
      • Basar O.
      • Yuksel O.
      • Yang D.J.
      • et al.
      Feasibility and safety of microforceps biopsy in the diagnosis of pancreatic cysts.
      ,
      • Crino S.F.
      • Bernardoni L.
      • Brozzi L.
      • et al.
      Association between macroscopically visible tissue samples and diagnostic accuracy of EUS-guided through-the-needle microforceps biopsy sampling of pancreatic cystic lesions.
      ,
      • Yang D.
      • Trindade A.J.
      • Yachimski P.
      • et al.
      Histologic analysis of endoscopic ultrasound-guided through the needle microforceps biopsies accurately identifies mucinous pancreas cysts.
      • Cheesman A.R.
      • Zhu H.
      • Kumta N.A.
      • et al.
      EUS-guided microforceps biopsy and needle-based confocal laser endomicroscopy significantly improve the diagnostic yield and have major impact on clinical management of pancreatic cystic lesions [abstract].
      • Mittal C.
      • Obuch J.C.
      • Hammad H.
      • et al.
      Technical feasibility, diagnostic yield, and safety of microforceps biopsies during EUS evaluation of pancreatic cystic lesions (with video).
      • Zhang M.L.
      • Arpin R.N.
      • Brugge W.R.
      • et al.
      Moray micro forceps biopsy improves the diagnosis of specific pancreatic cysts.
      • Wilen J.
      • Visrodia K.
      • Lan G.
      • et al.
      The feasibility and value of cyst wall biopsy using micro forceps in the diagnosis of pancreatic cysts [abstract].
      including Crino et al and Cheesman et al, the pooled OR was 4.82 (95% confidence interval, 1.65-14.31; P = .004, I2 = 83%). When Crino et al and Cheesman et al were removed based on down-rating as outliers, the OR became insignificant (OR, 2.33; 95% confidence interval, .9-6.03; P = .08, I2 = 76%) (Supplementary Fig. 1A and B, available online at www.giejournal.org). There also a slight drop in heterogeneity. Therefore, this down-rating property would have changed the results of the published meta-analysis from significant to nonsignificant. These 2 studies, identified by our scale as outliers, appear to have undue influence on the results of this meta-analysis.

      Discussion

      We report on the development and validation of a quality assessment scale for cohort studies to be used in reviewing medical literature for systematic reviews and meta-analyses. This novel scale was validated and found to have favorable performance characteristics. The scale scored best for understandability, ease of use, ease of interpretation, and efficiency when compared with 3 other scales used for cohort studies.
      This novel scale is efficient and easy to score and interpret as reported by a blinded reviewer. The scale was compared with 3 other scales in use for cohort studies. For example, the Down and Black scale, which we have used many times, is composed of 27 questions.
      • Downs S.H.
      • Black N.
      The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions.
      In our experience, conducting the scoring can be time-consuming. The scale took an average of 6.6 ± 1.1 minutes compared with only 2.2 ± 1.1 minutes for the Qumseya scale. Based on a blinded reviewer, the Down and Black scale had 1 to 5 nonapplicable questions. Many of the questions are irrelevant to cohort studies because the tool can be used for randomized studies. Therefore, when using this scale, we sometimes had to exclude some of these questions.
      • Qumseya B.
      • Gendy S.
      • Wallace A.
      • et al.
      Prevalence of Barrett's esophagus in obese patients undergoing pre-bariatric surgery evaluation: a systematic review and meta-analysis.
      This can create confusion for reviewers and readers and is not a standardized process. Additionally, the scoring results are not easy to interpret as reported in our results. This is shown by the fact that most studies were rated as low quality by Down and Black but rated differently in other scales.
      Our scale was also compared with the ROBINS-I. This is a comprehensive scale, yet it can be rather complex to follow. Our blinded reviewer rated this scale very low in terms of understandability, ease of use, and overall score. Furthermore, ROBINS-I took a very long time to fill out, averaging 19.4 ± 1.6 minutes, which is significantly higher than any of the other scales. In some meta-analyses, a rater may have to score 50 or more studies. Obviously, this would be a substantial amount of time, which many researchers may not have. Our independent rater gave ROBINS-I a score of 1 of 10 on ease of use and 0 of 10 on ease of interpretation, compared with 10 of 10 for the Qumseya scale for these categories. Interestingly, the agreement between our scale and the ROBINS-I was high. Although the sample size was small, this indicates that our more efficient and more user friendly scale compares well with ROBINS-I in quality scoring.
      We also compared our scale with the Newcastle-Ottawa scale.
      • Hartling L.
      • Milne A.
      • Hamm M.P.
      • et al.
      Testing the Newcastle-Ottawa scale showed low reliability between individual reviewers.
      This tool is shorter than the Down and Black and ROBINS-I and took less time to fill out. Yet, the rating in this tool can also be confusing because studies are awarded stars for each question. There is no numerical score or cutoff for excluding studies with poor design. On the other hand, our scale has 9 questions; completion takes less time from a busy physician’s schedule to do. Additionally, the scoring system is out of 10, which is intuitive to understand and relate to. This is analogous to scoring of any examination. A score less than 60% (6/10) means failing the test. In this case, a score <60% means the study is of poor quality. Such a score is relatable to most people and does not require much interpretation.
      In addition to the ease of use and understandability of our scale, the process of validation is multilayered. The scale underwent face, content, and construct validation by expert physicians and by using a debriefing questionnaire. The final scale had high test–retest reliability and inter-rater reliability with a Spearman rank correlation coefficient r = .66.
      As discussed above, our scale compared more favorably than existing scales in many aspects. Moreover, our scale has several innovative questions that are not considered in previous scales, for example the ability to deduct a point from a study if there is evidence that the study represents an outlier. This question can be answered from the study if there is an obvious reason for a study to be an outlier. However, we designed this question to be answered post hoc. For example, the rate of BE in prebariatric patients was found to be 10% in 1 study,
      • Gomez V.
      • Bhalla R.
      • Heckman M.G.
      • et al.
      routine screening endoscopy before bariatric surgery: Is it necessary?.
      whereas the pooled rate in 29 other studies was <1%.
      • Qumseya B.
      • Gendy S.
      • Wallace A.
      • et al.
      Prevalence of Barrett's esophagus in obese patients undergoing pre-bariatric surgery evaluation: a systematic review and meta-analysis.
      Based on this scale, raters would be able deduct a point from this study. Such post-hoc analysis, which is defined a priori, can be very helpful in rating down studies that can be potential outliers in a meta-analysis.
      In our previous example, the outlier study by Gomez et al
      • Gomez V.
      • Bhalla R.
      • Heckman M.G.
      • et al.
      routine screening endoscopy before bariatric surgery: Is it necessary?.
      was removed from the meta-analysis. The effect estimate for identifying a study as an outlier was set at 8 to 10 times the expected effect estimate. We have used this rate in a previous study
      • Qumseya B.
      • Gendy S.
      • Wallace A.
      • et al.
      Prevalence of Barrett's esophagus in obese patients undergoing pre-bariatric surgery evaluation: a systematic review and meta-analysis.
      and believe that it sets a high bar for rating down potential outlier studies. This allows the researcher to deduct points at the time of analysis in studies with unexpected effect estimates. This post hoc question is critical and can allow raters to give a lower score to a study that is clearly an outlier. This can also result in lower heterogeneity because outlier studies tend to contribute significant to between-study variation. We further tested this in a published meta-analysis and found that this down-rating feature can change the results of the study as discussed. The OR of sample adequacy went from being slightly significant to insignificant when the outliers were removed. This also resulted in a slight improvement in heterogeneity. This feature is unique to our scale and will prove to be very helpful for researchers in this field.
      Another innovation is that the scale awards more scores to prospective studies compared with retrospective studies. This is important because not all cohort studies are the same. Additionally, our scale provides 2 points for multicentered studies and 1 point for single-centered studies. Having a multicenter study does not necessarily mean a better study design. However, designing a protocol that spans multiple center is usually more rigorous and requires more collaboration across various institutions or countries. Therefore, we decided to give an extra point for this reason. Obviously, this is only 1 of many factors that are considered for study quality.
      In addition, the question on the primary outcome of the study is unique to our scale and of great importance. If the primary outcome of the study was the same as the primary outcome of the meta-analysis, the study would be awarded 1 point. However, many times the primary outcome of the review is different from that of the study. For example, if the primary outcome for a study is the prevalence of BE after bariatric surgery, one may find 2 types of studies. One study may be specifically looking at the rates of BE after surgery. Another study may be looking at the rates of esophagitis but will also report the rate of BE as a secondary outcome. We argue that if the authors are specifically looking for BE, they would have a specific definition of BE and may be seeking it at the time of endoscopy. This is not necessarily the case if BE was a secondary outcome of a study. Previous quality assessment scales for cohort studies included no such distinctions.
      Finally, this scale was designed for published studies. Using the scale in meeting abstracts is not advisable because of the lack of data in such abstracts to make a full assessment. We recommend rating meeting abstracts as low quality in any meta-analysis and planning a sensitivity analysis to compare data from published articles and meeting abstracts. Based on feedback we expect to get on our scale from researchers, we intend to review this scale in 3 years and consider further improvements.
      In conclusion, we present this validated scale to be used for quality assessment of cohort studies in the medical literature. We believe this scale has favorable performance characteristics; is more efficient to conduct, easier to use, and easier to interpret; and will be very helpful for physicians and researchers conducting systematic reviews and meta-analyses.

      Acknowledgments

      We thank Drs Sachin Wani and Michael Bartel for their review of the initial tool as expert physicians. We also acknowledge our raters, Drs Nanlong Liu, Jake Wilson, Walid Khan, Yazen Qumsiyeh, Sandeep Ponniah, Nikhil Kadle, Yaseen Prebtani, and Tony Brar.

      Appendix

      Figure thumbnail fx1
      Supplementary Figure 1Pooled odds ratio of sample adequacy between microfoceps biopsy sampling and standard FNA and (A) including Crino et al and Cheesman et al and (B) excluding Crino et al and Cheesman et al. CI, Confidence interval.

      References

        • Johnston A.
        • Kelly S.E.
        • Hsieh S.C.
        • et al.
        Systematic reviews of clinical practice guidelines: a methodological guide.
        J Clin Epidemiol. 2019; 108: 64-76
      1. Institute of Medicine Committee on Standards for Developing Trustworthy Clinical Practice Guidelines.
        in: Graham R. Mancher M. Miller Wolman D. Clinical practice guidelines we can trust. National Academies Press, Washington, DC2011
        • Guyatt G.H.
        • Oxman A.D.
        • Vist G.E.
        • et al.
        GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.
        BMJ. 2008; 336: 924-926
        • Gopalakrishnan S.
        • Ganeshkumar P.
        Systematic reviews and meta-analysis: understanding the best evidence in primary healthcare.
        J Fam Med Prim Care. 2013; 2: 9-14
        • Sahni M.
        • Goel A.
        • Pande P.
        • et al.
        Multiple gastrointestinal cancers in a single patient—a rare clinical entity.
        Indian J Surg Oncol. 2018; 9: 633-635
        • Clark H.D.
        • Wells G.A.
        • Huet C.
        • et al.
        Assessing the quality of randomized trials: reliability of the Jadad scale.
        Control Clin Trials. 1999; 20: 448-452
        • Savovic J.
        • Weeks L.
        • Sterne J.A.
        • et al.
        Evaluation of the Cochrane Collaboration's tool for assessing the risk of bias in randomized trials: focus groups, online survey, proposed recommendations and their implementation.
        Syst Rev. 2014; 3: 37
        • Olivo S.A.
        • Macedo L.G.
        • Gadotti I.C.
        • et al.
        Scales to assess the quality of randomized controlled trials: a systematic review.
        Phys Ther. 2008; 88: 156-175
        • Downs S.H.
        • Black N.
        The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions.
        J Epidemiol Community Health. 1998; 52: 377-384
        • Hartling L.
        • Milne A.
        • Hamm M.P.
        • et al.
        Testing the Newcastle-Ottawa scale showed low reliability between individual reviewers.
        J Clin Epidemiol. 2013; 66: 982-993
        • Sterne J.A.
        • Hernan M.A.
        • Reeves B.C.
        • et al.
        ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.
        BMJ. 2016; 355: i4919
        • Qumseya B.
        • Gendy S.
        • Wallace A.
        • et al.
        Prevalence of Barrett's esophagus in obese patients undergoing pre-bariatric surgery evaluation: a systematic review and meta-analysis.
        Endoscopy. 2020; 52: 537-547
        • Qumseya B.J.
        • Bukannan A.
        • Gendy S.
        • et al.
        Systematic review and meta-analysis of prevalence and risk factors for Barrett's esophagus.
        Gastrointest Endosc. 2019; 90: 707-717
        • Aghaie Meybodi M.
        • Qumseya B.J.
        • Shakoor D.
        • et al.
        Efficacy and feasibility of G-POEM in management of patients with refractory gastroparesis: a systematic review and meta-analysis.
        Endosc Int Open. 2019; 7: E322-E329
        • Qumseya B.J.
        • Bartel M.J.
        • Gendy S.
        • et al.
        High rate of over-staging of Barrett's neoplasia with endoscopic ultrasound: systemic review and meta-analysis.
        Dig Liver Dis. 2018; 50: 438-445
        • Keswani R.N.
        • Qumseya B.J.
        • O'Dwyer L.C.
        • et al.
        Association between endoscopist and center endoscopic retrograde cholangiopancreatography volume with procedure success and adverse outcomes: a systematic review and meta-analysis.
        Clin Gastroenterol Hepatol. 2017; 15: 1866-1875
        • Qumseya B.J.
        • Wani S.
        • Gendy S.
        • et al.
        Disease progression in Barrett's low-grade dysplasia with radiofrequency ablation compared with surveillance: systematic review and meta-analysis.
        Am J Gastroenterol. 2017; 112: 849-865
        • Qumseya B.J.
        • Wani S.
        • Desai M.
        • et al.
        Adverse events after radiofrequency ablation in patients with barrett's esophagus: a systematic review and meta-analysis.
        Clin Gastroenterol Hepatol. 2016; 14: 1086-1095
        • Qumseya B.J.
        • Wang H.
        • Badie N.
        • et al.
        Advanced imaging technologies increase detection of dysplasia and neoplasia in patients with Barrett's esophagus: a meta-analysis and systematic review.
        Clin Gastroenterol Hepatol. 2013; 11: 1562-1570
        • Kothari S.T.
        • Huang R.J.
        • Shaukat A.
        • et al.
        ASGE review of adverse events in colonoscopy.
        Gastrointest Endosc. 2019; 90: 863-876
        • Qumseya B.
        • Sultan S.
        • et al.
        • ASGE Standards of Practice Committee
        ASGE guideline on screening and surveillance of Barrett's esophagus.
        Gastrointest Endosc. 2019; 90: 335-359
        • Wani S.
        • Qumseya B.
        • et al.
        • Standards of Practice Committee
        Endoscopic eradication therapy for patients with Barrett's esophagus-associated dysplasia and intramucosal cancer.
        Gastrointest Endosc. 2018; 87: 907-931
        • Qumseya B.J.
        • Tayem Y.I.
        • Dasa O.Y.
        • et al.
        Barriers to colorectal cancer screening in Palestine: a national study in a medically underserved population.
        Clin Gastroenterol Hepatol. 2014; 12: 463-469
        • Genco A.
        • Soricelli E.
        • Casella G.
        • et al.
        Gastroesophageal reflux disease and Barrett's esophagus after laparoscopic sleeve gastrectomy: a possible, underestimated long-term complication.
        Surg Obes Relat Dis. 2017; 13: 568-574
        • Tai C.M.
        • Huang C.K.
        • Lee Y.C.
        • et al.
        Increase in gastroesophageal reflux disease symptoms and erosive esophagitis 1 year after laparoscopic sleeve gastrectomy among obese adults.
        Surg Endosc Other Intervent Techn. 2013; 27: 1260-1266
        • Braghetto I.
        • Korn O.
        • Csendes A.
        • et al.
        Laparoscopic treatment of obese patients with gastroesophageal reflux disease and Barrett's esophagus: a prospective study.
        Obes Surg. 2012; 22: 764-772
        • Braghetto I.
        • Csendes A.
        Prevalence of Barrett's esophagus in bariatric patients undergoing sleeve gastrectomy.
        Obes Surg. 2016; 26: 710-714
        • Soricelli E.
        • Casella G.
        • Baglio G.
        • et al.
        Lack of correlation between gastroesophageal reflux disease symptoms and esophageal lesions after sleeve gastrectomy.
        Surg Obes Relat Dis. 2018; 14: 751-756
        • Lim C.H.
        • Lee P.C.
        • Lim E.
        • et al.
        Correlation between symptomatic gastro-esophageal reflux disease (GERD) and erosive esophagitis (EE) post-vertical sleeve gastrectomy (VSG).
        Obes Surg. 2019; 29: 207-214
        • Houghton S.G.
        • Romero Y.
        • Sarr M.G.
        Effect of Roux-en-Y gastric bypass in obese patients with Barrett's esophagus: attempts to eliminate duodenogastric reflux.
        Surg Obes Relat Dis. 2008; 4: 1-4
        • Sebastianelli L.
        • Benois M.
        • Vanbiervliet G.
        • et al.
        Systematic endoscopy 5 years after sleeve gastrectomy results in a high rate of Barrett’s esophagus: results of a multicenter study.
        Obes Surg. 2019; 29: 1462-1469
        • Felsenreich D.M.
        • Kefurt R.
        • Schermann M.
        • et al.
        Reflux, sleeve dilation, and Barrett’s esophagus after laparoscopic sleeve gastrectomy: long-term follow-up.
        Obes Surg. 2017; 27: 3092-3101
        • Andrew B.
        • Alley J.B.
        • Aguilar C.E.
        • et al.
        Barrett’s esophagus before and after Roux-en-Y gastric bypass for severe obesity.
        Surg Endosc. 2018; 32: 930-936
        • Facciorusso A.
        • Del Prete V.
        • Antonino M.
        • et al.
        Diagnostic yield of EUS-guided through-the-needle biopsy in pancreatic cysts: a meta-analysis.
        Gastrointest Endosc. 2020; 92: 1-8
        • Basar O.
        • Yuksel O.
        • Yang D.J.
        • et al.
        Feasibility and safety of microforceps biopsy in the diagnosis of pancreatic cysts.
        Gastrointest Endosc. 2018; 88: 79-86
        • Crino S.F.
        • Bernardoni L.
        • Brozzi L.
        • et al.
        Association between macroscopically visible tissue samples and diagnostic accuracy of EUS-guided through-the-needle microforceps biopsy sampling of pancreatic cystic lesions.
        Gastrointest Endosc. 2019; 90: 933-943
        • Kovacevic B.
        • Karstensen J.G.
        • Havre R.F.
        • et al.
        Initial experience with EUS-guided microbiopsy forceps in diagnosing pancreatic cystic lesions: a multicenter feasibility study (with video).
        Endosc Ultrasound. 2018; 7: 383-388
        • Samarasena J.
        • Yu A.
        • Lee D.
        • et al.
        EUS-guided through-the-needle biopsy for pancreatic cystic lesions.
        VideoGIE. 2019; 4: 436-439
        • Yang D.
        • Trindade A.J.
        • Yachimski P.
        • et al.
        Histologic analysis of endoscopic ultrasound-guided through the needle microforceps biopsies accurately identifies mucinous pancreas cysts.
        Clin Gastroenterol Hepatol. 2019; 17: 1587-1596
        • Cheesman A.R.
        • Zhu H.
        • Kumta N.A.
        • et al.
        EUS-guided microforceps biopsy and needle-based confocal laser endomicroscopy significantly improve the diagnostic yield and have major impact on clinical management of pancreatic cystic lesions [abstract].
        Gastrointest Endosc. 2019; 89: AB144
        • Mittal C.
        • Obuch J.C.
        • Hammad H.
        • et al.
        Technical feasibility, diagnostic yield, and safety of microforceps biopsies during EUS evaluation of pancreatic cystic lesions (with video).
        Gastrointest Endosc. 2018; 87: 1263-1269
        • Zhang M.L.
        • Arpin R.N.
        • Brugge W.R.
        • et al.
        Moray micro forceps biopsy improves the diagnosis of specific pancreatic cysts.
        Cancer Cytopathol. 2018; 126: 414-420
        • Wilen J.
        • Visrodia K.
        • Lan G.
        • et al.
        The feasibility and value of cyst wall biopsy using micro forceps in the diagnosis of pancreatic cysts [abstract].
        Gastrointest Endosc. 2019; 89: AB605
        • Qumseya B.
        • Gendy S.
        • Wallace A.
        • et al.
        Prevalence of Barrett's esophagus in obese patients undergoing pre-bariatric surgery evaluation: a systematic review and meta-analysis.
        Endoscopy. 2020; 52: 537-547
        • Gomez V.
        • Bhalla R.
        • Heckman M.G.
        • et al.
        routine screening endoscopy before bariatric surgery: Is it necessary?.
        Bariatr Surg Pract Patient Care. 2014; 9: 143-149