Performance of Ai Chatbots on Controversial Topics in Oral Medicine, Pathology, and Radiology

Performance of Ai Chatbots on Controversial Topics in Oral Medicine, Pathology, and Radiology Publisher Pubmed

Mohammadrahimi H^{1, 2} ; Khoury ZH³ ; Alamdari MI⁴ ; Rokhshad R² ; Motie P⁵ ; Parsa A⁶ ; Tavares T⁷ ; Sciubba JJ⁸ ; Price JB^{1, 6} ; Sultan AS^{1, 6, 9}

Show Affiliations

1. Division of Artificial Intelligence Research, University of Maryland School of Dentistry, Baltimore, MD, United States
2. Topic Group Dental Diagnostics and Digital Dentistry, ITU/WHO Focus Group AI on Health, Berlin, Germany
3. Department of Oral Diagnostic Sciences and Research, Meharry Medical College School of Dentistry, Nashville, TN, United States
4. Department of Oral and Maxillofacial Radiology, School of Dentistry, Shahid Beheshti University of Medical Sciences, Tehran, Iran
5. Medical Image and Signal Processing Research Center, Medical University of Isfahan, Isfahan, Iran
6. Department of Oncology and Diagnostic Sciences, University of Maryland School of Dentistry, Baltimore, MD, United States
7. Department of Comprehensive Dentistry, UT Health San Antonio School of Dentistry, San Antonio, TX, United States
8. Department of Otolaryngology, Head & Neck Surgery, The Johns Hopkins University, Baltimore, MD, United States
9. University of Maryland Marlene and Stewart Greenebaum Comprehensive Cancer Center, Baltimore, MD, United States

Source: Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology Published:2024

Abstract

Objectives: In this study, we assessed 6 different artificial intelligence (AI) chatbots (Bing, GPT-3.5, GPT-4, Google Bard, Claude, Sage) responses to controversial and difficult questions in oral pathology, oral medicine, and oral radiology. Study Design: The chatbots’ answers were evaluated by board-certified specialists using a modified version of the global quality score on a 5-point Likert scale. The quality and validity of chatbot citations were evaluated. Results: Claude had the highest mean score of 4.341 ± 0.582 for oral pathology and medicine. Bing had the lowest scores of 3.447 ± 0.566. In oral radiology, GPT-4 had the highest mean score of 3.621 ± 1.009 and Bing the lowest score of 2.379 ± 0.978. GPT-4 achieved the highest mean score of 4.066 ± 0.825 for performance across all disciplines. 82 out of 349 (23.50%) of generated citations from chatbots were fake. Conclusions: The most superior chatbot in providing high-quality information for controversial topics in various dental disciplines was GPT-4. Although the majority of chatbots performed well, it is suggested that developers of AI medical chatbots incorporate scientific citation authenticators to validate the outputted citations given the relatively high number of fabricated citations. © 2024 Elsevier Inc.

Related Docs

1. Efficacy and Empathy of Ai Chatbots in Answering Frequently Asked Questions on Oral Oncology, Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology (2025)

2. Artificial Intelligence for Classification and Detection of Oral Mucosa Lesions on Photographs: A Systematic Review and Meta-Analysis, Clinical Oral Investigations (2024)

3. Artificial Intelligence-Powered Cognitive Behavioral Therapy Chatbots, a Systematic Review, Iranian Journal of Psychiatry (2025)

Experts (# of related papers)

Parisa Soltani (1)

Style	Citing Format
MLA	Mohammadrahimi H, et al.. "Performance of Ai Chatbots on Controversial Topics in Oral Medicine, Pathology, and Radiology." Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology, vol. 137, no. 5, 2024, pp. 508-514.
APA	Mohammadrahimi H, Khoury ZH, Alamdari MI, Rokhshad R, Motie P, Parsa A, Tavares T, Sciubba JJ, Price JB, Sultan AS (2024). Performance of Ai Chatbots on Controversial Topics in Oral Medicine, Pathology, and Radiology. Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology, 137(5), 508-514.
Chicago	Mohammadrahimi H, Khoury ZH, Alamdari MI, Rokhshad R, Motie P, Parsa A, Tavares T, Sciubba JJ, Price JB, Sultan AS. "Performance of Ai Chatbots on Controversial Topics in Oral Medicine, Pathology, and Radiology." Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology 137, no. 5 (2024): 508-514.
Harvard	Mohammadrahimi H et al. (2024) 'Performance of Ai Chatbots on Controversial Topics in Oral Medicine, Pathology, and Radiology', Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology, 137(5), pp. 508-514.
Vancouver	Mohammadrahimi H, Khoury ZH, Alamdari MI, Rokhshad R, Motie P, Parsa A, et al.. Performance of Ai Chatbots on Controversial Topics in Oral Medicine, Pathology, and Radiology. Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology. 2024;137(5):508-514.
BibTex	@article{ author = {Mohammadrahimi H and Khoury ZH and Alamdari MI and Rokhshad R and Motie P and Parsa A and Tavares T and Sciubba JJ and Price JB and Sultan AS}, title = {Performance of Ai Chatbots on Controversial Topics in Oral Medicine, Pathology, and Radiology}, journal = {Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology}, volume = {137}, number = {5}, pages = {508-514}, year = {2024} }
RIS	TY - JOUR AU - Mohammadrahimi H AU - Khoury ZH AU - Alamdari MI AU - Rokhshad R AU - Motie P AU - Parsa A AU - Tavares T AU - Sciubba JJ AU - Price JB AU - Sultan AS TI - Performance of Ai Chatbots on Controversial Topics in Oral Medicine, Pathology, and Radiology JO - Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology VL - 137 IS - 5 SP - 508 EP - 514 PY - 2024 ER -

Science Communicator Platform

Authors

Authors Affiliations

Abstract