Reliability of the Quadas-2 Checklist in Ai-Based Medical Imaging: A Comparison of Human Reviewers and Chatgpt-4O Publisher

Summary: Can AI imaging studies be rated consistently? An assessment found inconsistent QUADAS-2 bias judgments, suggesting a need for AI-specific tools and validation. #MedicalAI #ResearchQuality

Dashti M ; Khosraviani F ; Ghadimi N ; Kalantari S ; Amirzade Iranaq M H ; Hemmati G ; Arzani S ; Pirayesh Z ; Motie P
Source: Imaging Science in Dentistry Published:2026

Abstract

Purpose: Artificial intelligence is reshaping medical imaging, yet reliable bias assessment in this context remains insufficiently studied. It remains unclear whether the widely used QUADAS-2 checklist provides reproducible evaluations in AI-based imaging studies. Therefore, we compared human and ChatGPT-4o assessments to examine the reproducibility of QUADAS-2 in this setting. Materials and Methods: Twenty PubMed-indexed AI-based dental imaging studies were analyzed by five human reviewers and five independent ChatGPT-4o runs using identical categorical outputs. Reliability within and between groups was quantified using Fleiss' kappa. Results: Human agreement ranged from -0.07 to 0.22, with only the reference standard domain reaching fair reliability (κ=0.22, P<0.001). ChatGPT-4o produced kappa values from -0.17 to 0.56, with the highest consistency observed in the index test domain (κ=0.56, P<0.001). Between-group kappa values remained near zero across domains, except in items uniformly classified as low risk. These patterns suggest that QUADAS-2 may yield inconsistent judgments across both groups, despite evaluation of 20 studies and repeated AI classifications. Conclusion: These findings suggest that QUADAS-2 may have limited reproducibility when applied to AI-based imaging research under the present study conditions, although the observed inconsistency may also reflect features of the study design and assessment setting. These findings support consideration of AI-specific tools such as QUADAS-AI for future bias assessment in AI-driven medical imaging. Copyright © 2026 by Korean Academy of Oral and Maxillofacial Radiology This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.