Tehran University of Medical Sciences

Science Communicator Platform

Stay connected! Follow us on X network (Twitter):
Share this content! On (X network) By
Practical Evaluation of Chatgpt Performance for Radiology Report Generation Publisher Pubmed



Soleimani M1 ; Seyyedi N1 ; Ayyoubzadeh SM1, 2 ; Kalhori SRN1, 3 ; Keshavarz H4
Authors
Show Affiliations
Authors Affiliations
  1. 1. Department of Health Information Management and Medical Informatics, School of Allied Medical Sciences, Tehran University of Medical Sciences, Tehran, Iran
  2. 2. Health Information Management Research Centre, Tehran University of Medical Sciences, Tehran, Iran
  3. 3. Peter L. Reichertz Institute for Medical Informatics, TU Braunschweig and Hannover Medical School, Braunschweig, Germany
  4. 4. Faculty of Electrical and Computer Engineering, Tarbiat Modares University, Tehran, Iran

Source: Academic Radiology Published:2024


Abstract

Rationale and Objectives: The process of generating radiology reports is often time-consuming and labor-intensive, prone to incompleteness, heterogeneity, and errors. By employing natural language processing (NLP)-based techniques, this study explores the potential for enhancing the efficiency of radiology report generation through the remarkable capabilities of ChatGPT (Generative Pre-training Transformer), a prominent large language model (LLM). Materials and Methods: Using a sample of 1000 records from the Medical Information Mart for Intensive Care (MIMIC) Chest X-ray Database, this investigation employed Claude.ai to extract initial radiological report keywords. ChatGPT then generated radiology reports using a consistent 3-step prompt template outline. Various lexical and sentence similarity techniques were employed to evaluate the correspondence between the AI assistant-generated reports and reference reports authored by medical professionals. Results: Results showed varying performance among NLP models, with Bart (Bidirectional and Auto-Regressive Transformers) and XLM (Cross-lingual Language Model) displaying high proficiency (mean similarity scores up to 99.3%), closely mirroring physician reports. Conversely, DeBERTa (Decoding-enhanced BERT with disentangled attention) and sequence-matching models scored lower, indicating less alignment with medical language. In the Impression section, the Word-Embedding model excelled with a mean similarity of 84.4%, while others like the Jaccard index showed lower performance. Conclusion: Overall, the study highlights significant variations across NLP models in their ability to generate radiology reports consistent with medical professionals' language. Pairwise comparisons and Kruskal–Wallis tests confirmed these differences, emphasizing the need for careful selection and evaluation of NLP models in radiology report generation. This research underscores the potential of ChatGPT to streamline and improve the radiology reporting process, with implications for enhancing efficiency and accuracy in clinical practice. © 2024 The Association of University Radiologists
Related Docs
1. Artificial Intelligence in Science and Society: The Vision of Usern, IEEE Access (2025)
Experts (# of related papers)
Nima Rezaei (1)