Background and Aims
The accurate measurement of technical skill in ERCP is essential for endoscopic training,
quality assurance, and coaching of this procedure. Hypothesizing that technical skill
can be measured by analysis of ERCP videos, we aimed to develop and validate a video-based
ERCP skill assessment tool.
Methods
Based on review of procedural videos, the task of ERCP was deconstructed into its
basic components by an expert panel that developed an initial version of the Bethesda
ERCP Skill Assessment Tool (BESAT). Subsequently, 2 modified Delphi panels and 3 validation
exercises were conducted with the goal of iteratively refining the tool. Fully crossed
generalizability studies investigated the contributions of assessors, ERCP performance,
and technical elements to reliability.
Results
Twenty-nine technical elements were initially generated from task deconstruction.
Ultimately, after iterative refinement, the tool comprised 6 technical elements and
11 subelements. The developmental process achieved consistent improvements in the
performance characteristics of the tool with every iteration. For the most recent
version of the tool, BESAT-v4, the generalizability coefficient (a reliability index)
was .67. Most variance in BESAT scores (43.55%) was attributed to differences in endoscopists’
skill, indicating that the tool can reliably differentiate between endoscopists based
on video analysis.
Conclusions
Video-based assessment of ERCP skill appears to be feasible with a novel instrument
that demonstrates favorable validity evidence. Future steps include determining whether
the tool can discriminate between endoscopists of varying experience levels and predict
important outcomes in clinical practice.
Abbreviations:
BESAT (Bethesda ERCP Skill Assessment Tool), D (decision), G (generalizability), SVI (stent versus indomethacin [trial])To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Gastrointestinal EndoscopyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- The ERCP quality network: a pilot study of benchmarking practice and performance.Am J Med Qual. 2013; 28: 256-260
- Lower provider volume is associated with higher failure rates for endoscopic retrograde cholangiopancreatography.Med Care. 2013; 51: 1040-1047
- Association between endoscopist and center endoscopic retrograde cholangiopancreatography volume with procedure success and adverse outcomes: a systematic review and meta-analysis.Clin Gastroenterol Hepatol. 2017; 15: 1866-1875
- Risk factors for post-ERCP pancreatitis: a prospective, multicenter study.Gastrointest Endosc. 2001; 54: 425-434
- Risk factors for complications after performance of ERCP.Gastrointest Endosc. 2002; 56: 652-656
- Needle knife sphincterotomy does not increase the risk of pancreatitis in patients with difficult biliary cannulation.Clin Gastroenterol Hepatol. 2013; 11: 430-436
- Major early complications from diagnostic and therapeutic ERCP: a prospective multicenter study.Gastrointest Endosc. 1998; 48: 1-10
- Quality indicators for ERCP.Am J Gastroenterol. 2015; 110: 91-101
- Surgical skill and complication rates after bariatric surgery.N Engl J Med. 2013; 369: 1434-1442
- Grading of surgeon technical performance predicts postoperative pancreatic fistula for pancreaticoduodenectomy independent of patient-related variables.Ann Surg. 2016; 264: 482-491
- Complementing operating room teaching with video-based coaching.JAMA Surg. 2017; 152: 318-325
- Impact of video coaching on gynecologic resident laparoscopic suturing: a randomized controlled trial.J Minim Invasive Gynecol. 2017; 24: 426-431
- A prospective comparison of live and video-based assessments of colonoscopy performance.Gastrointest Endosc. 2018; 87: 766-775
- Development and validation of a video-based cold snare polypectomy assessment tool (with videos).Gastrointest Endosc. 2019; 89: 1222-1230
- Colon polypectomy report card improves polypectomy competency: results of a prospective quality improvement study (with video).Gastrointest Endosc. 2019; 89: 1212-1221
- Validity of test interpretation and use.Cognit Ther Res. 1990; 34: 380-387
- American Psychological Association, the National Council on Measurement in Education.in: Standards for educational and psychological testing. American Educational Research Association, Washington, DC2014
- Rectal indomethacin alone versus indomethacin and prophylactic pancreatic stent placement for preventing pancreatitis after ERCP: study protocol for a randomized controlled trial.Trials. 2016; 17: 120
- The Delphi technique in health sciences education research.Med Teach. 2005; 27: 639-643
Keeney S, Hasson F, McKenna H, editors. The Delphi technique. In: The Delphi technique in nursing and health research. Oxford, UK: Wiley-Blackwell; 2011, p. 1-26.
- Reliability coefficients and generalizability theory.in: Rao C.R. Sinharay S. Handbook of statistics. Elsevier, Amsterdam, Netherlands2006: 81-124
- Generalizability theory: overview.in: Wiley StatsRef: statistical reference online. John Wiley & Sons, Hoboken, NJ2014
- Generalizability theory.in: Green J.L. Camilli G. Elmore P.B. Complementary methods for research in education. 3rd ed. American Educational Research Association, Washington, DC2005: 599-612
- Generalizability theory for the perplexed: a practical introduction and guide: AMEE guide no. 68.Med Teach. 2020; 34: 960-992
- Generalizability theory: overview.in: Everitt B.S. Howell D.C. Encyclopedia of statistics in behavioral science. Wiley, Chichester, UK2005: 717-719
- Reliability: on the reproducibility of assessment data.Med Educ. 2004; 38: 1006-1012
- Development and validation of a novel method for assessing competency in polypectomy: direct observation of polypec my skills.Gastrointest Endosc. 2011; 73: 1232-1239
- Development of an objective assessment tool for total laparoscopic hysterectomy: a Delphi method among experts and evaluation on a virtual reality simulator.PLoS One. 2018; 13e0190580
- A prospective study demonstrating the reliability and validity of two procedure-specific evaluation tools to assess operative competence in laparoscopic colorectal surgery.Surg Endosc. 2012; 26: 2489-2503
- Video-based coaching in surgical education: a systematic review and meta-analysis.Surg Endosc. 2020; 34: 521-535
- A video anchored rating scale leads to high inter-rater reliability of inexperienced and expert raters in the absence of rater training.Am J Surg. 2020; 219: 221-226
- Competence in endoscopic ultrasound and endoscopic retrograde cholangiopancreatography, from training through independent practice.Gastroenterology. 2018; 155: 1483-1494
- Exploring the impact of mental workload on rater-based assessments.Adv Health Sci Educ Theory Pract. 2013; 18: 291-303
- Is a resident’s score on a video- taped objective structured assessment of technical skills affected by revealing the resident’s identity?.Am J Obstet Gynecol. 2003; 189: 688-691
- Video-based assessments: a promising step in improving polypectomy competency.Gastrointest Endosc. 2019; 89: 1231-1233
Article info
Publication history
Published online: July 30, 2020
Accepted:
July 24,
2020
Received:
May 21,
2020
Footnotes
DISCLOSURE: Dr Elmunzer, Consultant for Takeda pharmaceuticals; Dr Elta, consultant for Olympus Medical; Dr Cote, consultant for Olympus Corporation and Boston Scientific, paid speaker/teacher for Abbvie Pharmaceuticals, research support from Kangen Pharmaceuticals and Boston Scientific Corporation; Dr Willingham, research funding from Cancer Prevention Pharmaceuticals, Cook Medical, Boston Scientific, and PCI Biotech; Dr Varadarajulu, consultant for Boston Scientifc, Olympus America, Medtronic, and Creo Medical. All other authors disclosed no financial relationships. Research support for this study was provided in part by National Institutes of Health grant U01DK104833.
If you would like to chat with an author of this article, you may contact Dr Elmunzer at [email protected]
Identification
Copyright
© 2021 by the American Society for Gastrointestinal Endoscopy
ScienceDirect
Access this article on ScienceDirectLinked Article
- ERCP and video assessment: Can video judge the endoscopy star?Gastrointestinal EndoscopyVol. 93Issue 4
- PreviewERCP continues to be one of the most technically challenging and complex endoscopic procedures performed. ERCP carries higher risk and rates of adverse events than traditional endoscopy, and, as with other endoscopic procedures, effective training is critical. As with most procedures, it is believed that volume, skill level, and competency affect outcomes and adverse events; however, this has largely been speculative or based on indirect evidence. Formal assessment of ERCP competency is lacking, and traditional measures have previously relied on volume thresholds as a surrogate.
- Full-Text
- Preview
