Performance of Large Language Models Conducting Systematic Review Tasks in Prosthodontics Publisher Pubmed



Rokhshad R ; Motie P ; Shirani M ; Sameie A ; Revillaleon M
Source: Journal of Prosthetic Dentistry Published:2026

Abstract

Statement of problem: Systematic reviews (SRs) are time-consuming and resource-intensive processes. Whether large language models (LLMs) can improve the process is unclear. Purpose: The purpose of this study was to evaluate the accuracy and reliability of 4 LLMs (GPT-4, Gemini, Claude, and Elicit) in performing SR tasks (full-text screening, data extraction, and risk of bias assessment) at 3 different sequential periods of time (0, 15, and 30 days). Material and methods: A comprehensive systematic search was conducted across 5 databases in December 2024, with 59 articles evaluated for screening (2 used for pilot) and 31 for data extraction (2 used for pilot). A 3-pronged prompting strategy was used, including persona-based initialization, few-shot learning, and structured population, intervention, control, outcome (PICO) criteria. Performance was assessed through 3 repeated evaluations at 2-week intervals by measuring accuracy and reliability using standard metrics (accuracy, precision, F1-score, sensitivity, and specificity) against expert assessments, data extraction quality on a 0 to 5 scale, and risk of bias agreement via the Cohen kappa, with statistical analysis using Kruskal-Wallis and Dunn post-hoc tests (α=.05). Results: In full-text screening, Claude achieved the highest sensitivity at 97%, while Claude and Elicit both showed strong overall performance with 86% accuracy and 87% F1-scores. All models maintained sensitivity above 90%. For data extraction, GPT-4 consistently performed best with median scores of 5.0, while Claude and Gemini showed similar capabilities. Significant differences only appeared in labeling and modeling tasks during Week 1 (P=.04). Risk of bias assessment agreement with experts varied from 55% to 90% across different criteria. Conclusions: LLMs show potential for SR efficiency (especially for data extraction) but require human oversight because of variable performance across models and tasks. Copyright © 2026 Published by Elsevier Inc.