Efficacy and Empathy of Ai Chatbots in Answering Frequently Asked Questions on Oral Oncology

Efficacy and Empathy of Ai Chatbots in Answering Frequently Asked Questions on Oral Oncology Publisher

Rokhshad R¹ ; Khoury ZH² ; Mohammadrahimi H³ ; Motie P⁴ ; Price JB⁵ ; Tavares T⁶ ; Jessri M^{7, 8} ; Bavarian R^{9, 10} ; Sciubba JJ¹¹ ; Sultan AS^{5, 12}

Show Affiliations

1. Department of Pediatric Dentistry, Loma Linda School of Dentistry, CA, United States
2. Department of Oral Diagnostic Sciences & Research, School of Dentistry, Meharry Medical College, TN, United States
3. Department of Dentistry and Oral Health, Aarhus University, Aarhus, Denmark
4. Medical Image and Signal Processing Research Center, Isfahan University of Medical Sciences, Isfahan, Iran
5. Division of Artificial Intelligence Research, Department of Oncology and Diagnostic Sciences, University of Maryland School of Dentistry, Baltimore, MD, United States
6. Department of Comprehensive Dentistry, UT Health San Antonio, School of Dentistry, San Antonio, TX, United States
7. Oral Medicine and Pathology Department, School of Dentistry, University of Queensland, Herston, QLD, Australia
8. Oral Medicine Department, MetroNorth Hospital and Health Services, Queensland Health, QLD, Australia
9. Department of Oral and Maxillofacial Surgery, Massachusetts General Hospital, Boston, MA, United States
10. Department of Oral and Maxillofacial Surgery, Harvard School of Dental Medicine, Boston, MA, United States
11. Department of Otolaryngology, Head & Neck Surgery, The Johns Hopkins University, Baltimore, MD, United States
12. University of Maryland Marlene and Stewart Greenebaum Comprehensive Cancer Center, Baltimore, MD, United States

Source: Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology Published:2025

Abstract

Objectives: Artificial intelligence chatbots have demonstrated feasibility and efficacy in improving health outcomes. In this study, responses from 5 different publicly available AI chatbots—Bing, GPT-3.5, GPT-4, Google Bard, and Claude—to frequently asked questions related to oral cancer were evaluated. Study design: Relevant patient-related frequently asked questions about oral cancer were obtained from two main sources: public health websites and social media platforms. From these sources, 20 oral cancer-related questions were selected. Four board-certified specialists in oral medicine/oral and maxillofacial pathology assessed the answers using modified version of the global quality score on a 5-point Likert scale. Additionally, readability was measured using the Flesch–Kincaid Grade Level and Flesch Reading Ease scores. Responses were also assessed for empathy using a validated 5-point scale. Results: Specialists ranked GPT-4 with highest total score of 17.3 ± 1.5, while Bing received the lowest at 14.9 ± 2.2. Bard had the highest Flesch Reading Ease score of 62 ± 7; and ChatGPT-3.5 and Claude received the lowest scores (more challenging readability). GPT-4 and Bard emerged as the most superior chatbots in terms of empathy and accurate citations on patient-related frequently asked questions pertaining to oral cancer. GPT-4 had highest overall quality, whereas Bing showed the lowest level of quality, empathy, and accuracy for citations. Conclusion: GPT-4 demonstrated the highest quality responses to frequently asked questions pertaining to oral cancer. Although impressive in their ability to guide patients on common oral cancer topics, most chatbots did not perform well when assessed for empathy or citation accuracy. © 2025

Related Docs

1. Performance of Ai Chatbots on Controversial Topics in Oral Medicine, Pathology, and Radiology, Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology (2024)

2. Artificial Intelligence-Powered Cognitive Behavioral Therapy Chatbots, a Systematic Review, Iranian Journal of Psychiatry (2025)

3. Assessment of the Quality of Life of the Patients With Treated Oral Cancer in Iran, Oral and Maxillofacial Surgery (2017)

4. Whether Enough Attention Is Being Paid to the Ethical Concerns Regarding the Use of Artificial Intelligence in Dentistry?, Dental Hypotheses (2023)

5. The Application of Artificial Intelligence in the Field of Mental Health: A Systematic Review, BMC Psychiatry (2025)

6. Investigating the Usefulness of Artificial Intelligence-Driven Robots in Developing Empathy for English for Medical Purposes Communication: The Role-Play of Asian and African Students, Computers in Human Behavior (2025)

Experts (# of related papers)

View all Related Experts

Bahareh Tahani (1)

Sayed Mohammad Razavi (1)

Style	Citing Format
MLA	Rokhshad R, et al.. "Efficacy and Empathy of Ai Chatbots in Answering Frequently Asked Questions on Oral Oncology." Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology, vol. , no. , 2025, pp. -.
APA	Rokhshad R, Khoury ZH, Mohammadrahimi H, Motie P, Price JB, Tavares T, Jessri M, Bavarian R, Sciubba JJ, Sultan AS (2025). Efficacy and Empathy of Ai Chatbots in Answering Frequently Asked Questions on Oral Oncology. Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology, (), -.
Chicago	Rokhshad R, Khoury ZH, Mohammadrahimi H, Motie P, Price JB, Tavares T, Jessri M, Bavarian R, Sciubba JJ, Sultan AS. "Efficacy and Empathy of Ai Chatbots in Answering Frequently Asked Questions on Oral Oncology." Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology , no. (2025): -.
Harvard	Rokhshad R et al. (2025) 'Efficacy and Empathy of Ai Chatbots in Answering Frequently Asked Questions on Oral Oncology', Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology, (), pp. -.
Vancouver	Rokhshad R, Khoury ZH, Mohammadrahimi H, Motie P, Price JB, Tavares T, et al.. Efficacy and Empathy of Ai Chatbots in Answering Frequently Asked Questions on Oral Oncology. Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology. 2025;():-.
BibTex	@article{ author = {Rokhshad R and Khoury ZH and Mohammadrahimi H and Motie P and Price JB and Tavares T and Jessri M and Bavarian R and Sciubba JJ and Sultan AS}, title = {Efficacy and Empathy of Ai Chatbots in Answering Frequently Asked Questions on Oral Oncology}, journal = {Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology}, volume = {}, number = {}, pages = {-}, year = {2025} }
RIS	TY - JOUR AU - Rokhshad R AU - Khoury ZH AU - Mohammadrahimi H AU - Motie P AU - Price JB AU - Tavares T AU - Jessri M AU - Bavarian R AU - Sciubba JJ AU - Sultan AS TI - Efficacy and Empathy of Ai Chatbots in Answering Frequently Asked Questions on Oral Oncology JO - Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology VL - IS - SP - EP - PY - 2025 ER -

Science Communicator Platform

Authors

Authors Affiliations

Abstract