Background and Aims
Measuring adenoma detection rates (ADRs) at the population level is challenging because
pathology reports are often reported in an unstructured format; further, there is
significant variation in reporting methods across institutions. Natural language processing
(NLP) can be used to extract relevant information from text-based records. We aimed
to develop and validate an NLP algorithm to identify colorectal adenomas that could
be used to report ADR at the population level in Ontario, Canada.
Methods
The sampling frame included pathology reports from all colonoscopies performed in
Ontario in 2015 and 2016. Two random samples of 450 and 1000 reports were selected
as the training and validation sets, respectively. Expert clinicians reviewed and
classified reports as adenoma or other. The training set was used to develop an NLP
algorithm (to identify adenomas) that was evaluated using the validation set. The
NLP algorithm test characteristics were calculated using expert review as the reference.
We used the algorithm to measure ADR for all endoscopists in Ontario in 2019.
Results
The 1450 pathology reports were derived from 62 laboratories, 266 pathologists, and
532 endoscopists. In the training set, the NLP algorithm for any adenoma had a sensitivity
of 99.60% (95% confidence interval (CI), 97.77-99.99), specificity of 99.01% (95%
CI, 96.49-99.88), positive predictive value of 99.19% (95% CI, 97.12-99.90), and F1
score of .99. Similar results were obtained for the validation set. The median ADR
was 33% (interquartile range, 26%-40%).
Conclusions
When we used a population-based sample from Ontario, our NLP algorithm was highly
accurate and was used at the system level to measure ADR.
Abbreviations:
ADR (adenoma detection rate), CI (confidence interval), CCO (Cancer Care Ontario), eMaRC (Electronic Mapping Reporting and Coding), NLP (natural language processing), NPV (negative predictive value), OHIP (Ontario Health Insurance Plan), PPV (positive predictive value)To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Gastrointestinal EndoscopyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Colonoscopy quality assurance in Ontario: systematic review and clinical practice guideline.Can J Gastroenterol Hepatol. 2014; 28: 251-274
- Adenoma detection rate and risk of colorectal cancer and death.N Engl J Med. 2014; 370: 1298-1306
- Quality indicators for colonoscopy and the risk of interval cancer.N Engl J Med. 2010; 362: 1795-1803
- Increased rate of adenoma detection associates with reduced risk of colorectal cancer and death.Gastroenterology. 2017; 153: 98-105
- European guidelines for quality assurance in colorectal cancer screening and diagnosis. First edition—quality assurance in endoscopy in colorectal cancer screening and diagnosis.Endoscopy. 2012; 44: SE88-SE108
- Quality indicators for colonoscopy.Gastrointest Endosc. 2015; 81: 31-53
- Guidelines for quality assurance in colorectal screening.2nd ed. Ireland. 2017 (Available at: https://www.screeningservice.ie/publications/BS-Guidelines-for-Quality-Assurance-in-Colorectal-Screening.pdf. Accessed August 17, 2022)
- Applying a natural language processing tool to electronic health records to assess performance on colonoscopy quality measures.Gastrointest Endosc. 2012; 75: 1-12
- Natural language processing accurately categorizes findings from colonoscopy and pathology reports.Clin Gastroenterol Hepatol. 2013; 11: 689-694
- Accurate identification of colonoscopy quality and polyp findings using natural language processing.J Clin Gastroenterol. 2019; 531: e25-e30
- A transparent and adaptable method to extract colonoscopy and pathology data using natural language processing.J Med Syst. 2020; 44: 151
- A natural language-based tool for diagnosis of serrated polyposis syndrome.Gastrointest Endosc. 2020; 92: 886-890
- Natural language processing accurately calculates adenoma and sessile serrated polyp detection rates.Dig Dis Sci. 2018; 63: 1794-1800
- Natural language processing as an alternative to manual reporting of colonoscopy quality metrics.Gastrointest Endosc. 2015; 82: 512-519
- Anatomic and advanced adenoma detection rates as quality metrics determined via natural language processing.Am J Gastroenterol. 2014; 109: 1844-1849
- Multi-center colonoscopy quality measurement utilizing natural language processing.Am J Gastroenterol. 2015; 110: 543-552
- National US healthcare system web-based colonoscopy quality report card: accurate, usable, and robust [abstract].Gastrointest Endosc. 2018; 87: AB114-5
- Developing a natural language processing application for measuring the quality of colonoscopy procedures.J Am Med Inform Assoc. 2011; 18: 150-156
- Validation of 5 key colonoscopy-related data elements from Ontario health administrative databases compared to the clinical record: a cross-sectional study.CMAJ Open. 2018; 6: E330-E338
Quality Management Partnership. Advancing quality: progress on key priorities in colonoscopy, mammography and pathology. Toronto: 2018. Available at: https://www.ccohealth.ca/sites/CCOHealth/files/assets/QMP-ProgressReport2018.pdf. Accessed November 12, 2018.
Klein WT, Havener L (eds). Standards for Cancer Registries Volume V: Pathology Laboratory Electronic Reporting, Version 4.0. Springfield, IL: North American Association of Central Cancer Registries, Inc, April 2011. Available at: https://www.naaccr.org/wp-content/uploads/2016/11/Pathology-Laboratory-Electronic-Reporting-Version-4.0-April-2011.pdf. Accessed November 12, 2018.
- Registry Plus eMarc Plus.(Available at:) (Accessed November 12, 2018)
- Natural language processing to extract medical problems from electronic clinical documents: performance evaluation.J Biomed Inform. 2006; 39: 589-599
- An introduction to power and sample size estimation the importance of power and sample size estimation for study design and analysis.Emerg Med J. 2003; 20: 453-458
- Institute for Quality Management in Healthcare.Accredited facilities. 2018; (Available at:) (Accessed November 12, 2019)
- Pathological reporting of colorectal polyps: pan-Canadian consensus guidelines.Can J Pathol. 2012; 4: 81-90
- Colorectal carcinoma: pathologic aspects.J Gastrointest Oncol. 2012; 3: 153-173
- Malignant colorectal polyps; pathological consideration (a review).Iran J Pathol. 2017; 12: 1-8
- Adenoma and malignant colorectal polyp: pathological considerations and clinical applications.EMJ Gastroenterol. 2018; 7: 92-102
- Guidelines for colonoscopy surveillance after screening and polypectomy: a consensus update by the US Multi-Society Task Force on Colorectal Cancer.Gastroenterology. 2012; 143: 844-857
- Post-polypectomy colonoscopy surveillance: European Society of Gastrointestinal Endoscopy (ESGE) guideline.Endoscopy. 2013; 45: 842-851
- Colorectal cancer surveillance after index colonoscopy: guidance from the Canadian Association of Gastroenterology.Can J Gastroenterol. 2013; 27: 224-228
Article info
Publication history
Published online: July 14, 2022
Accepted:
July 8,
2022
Received:
May 12,
2021
Footnotes
DISCLOSURE: The following authors disclosed financial relationships: J. Tinmouth, D. Swain, V. Lee, Y. Li, J. Gao: Employee of Ontario Health (Cancer Care Ontario). Research support for this study was provided by Ontario Health (Cancer Care Ontario).
Identification
Copyright
© 2023 by the American Society for Gastrointestinal Endoscopy
