Exploring the Methodological Quality and Risk of Bias in 200 Systematic Reviews: A Comparative Study of Robis and Amstar-2 Tools

Exploring the Methodological Quality and Risk of Bias in 200 Systematic Reviews: A Comparative Study of Robis and Amstar-2 Tools Publisher

Lunny C ; Jain N ; Nazari T ; Kosaner Kliess M ; Santos L ; Goodman I ; Osman AAM ; Berrone S ; Dadam MN ; Brenna CTA ; Hussein H ; Dahdal G ; Cespedes DA ; Ferri N Show All Authors

Lunny C
Jain N
Nazari T
Kosaner Kliess M
Santos L
Goodman I
Osman AAM
Berrone S
Dadam MN
Brenna CTA
Hussein H
Dahdal G
Cespedes DA
Ferri N
Kanji S
Chi Y
Pieper D
Shea B
Parker A
Neupane D
Khan PA
Rangira D
Kolaski K
Ridley B
Berour A
Sun K
Hamidi Rad RH
Ouyang Z
Reid EK
Perezneri I
Barakat SO
Bargeri S
Gianola S
Castellini G
Whitelaw S
Stevens A
Kolekar SB
Wong K
Major P
Bagheri E
Tricco AC

Source: Research Synthesis Methods Published:2025

Abstract

AMSTAR-2 (A Measurement Tool to Assess Systematic Reviews, version 2) and ROBIS are tools used to assess the methodological quality and the risk of bias in a systematic review (SR). We applied AMSTAR-2 and ROBIS to a sample of 200 published SRs. We investigated the overlap in their methodological constructs, responses by item, and overall, percentage agreement, direction of effect, and timing of assessments. AMSTAR-2 contains 16 items and ROBIS 24 items. Three items in AMSTAR-2 and nine in ROBIS did not overlap in construct. Of the 200 SRs, 73% were low or critically low quality using AMSTAR-2, and 81% had a high risk of bias using ROBIS. The median time to complete AMSTAR-2 and ROBIS was 51 and 64 minutes, respectively. When assessment times were calibrated to the number of items in each tool, each item took an average of 3.2 minutes per item for AMSTAR-2 compared to 2.7 minutes for ROBIS. Nine percent of SRs had opposing ratings (i.e., AMSTAR-2 was high quality while ROBIS was high risk). In both tools, three-quarters of items showed more than 70% agreement between raters after extensive training and piloting. AMSTAR-2 and ROBIS provide complementary rather than interchangeable assessments of systematic reviews. AMSTAR-2 may be preferable when efficiency is prioritized and methodological rigour is the focus, whereas ROBIS offers a deeper examination of potential biases and external validity. Given the widespread reliance on systematic reviews for policy and practice, selecting the appropriate appraisal tool remains crucial. Future research should explore strategies to integrate the strengths of both instruments while minimizing the burden on assessors. © 2025 Elsevier B.V., All rights reserved.

Related Docs

View other Related Docs

1. Association of Sugar Sweetened Beverages Consumption With Non-Alcoholic Fatty Liver Disease: A Systematic Review and Meta-Analysis, European Journal of Nutrition (2019)

2. Methodology of Systematic Review in the Health System, Payesh (2024)

3. Effects of Berberine and Barberry on Selected Inflammatory Biomarkers in Adults: A Systematic Review and Dose–Response Meta-Analysis of Randomized Clinical Trials, Phytotherapy Research (2023)

Experts (# of related papers)

View all Related Experts

Shirin Irani (1)

Elham Ebrahimi (1)

Style	Citing Format
MLA	Lunny C, et al.. "Exploring the Methodological Quality and Risk of Bias in 200 Systematic Reviews: A Comparative Study of Robis and Amstar-2 Tools." Research Synthesis Methods, vol. , no. , 2025, pp. -.
APA	Lunny C, Jain N, Nazari T, Kosaner Kliess M, Santos L, Goodman I, Osman AAM, Berrone S, Dadam MN, Brenna CTA, Hussein H, Dahdal G, Cespedes DA, Ferri N, Kanji S, Chi Y, Pieper D, Shea B, Parker A, ... Tricco AC (2025). Exploring the Methodological Quality and Risk of Bias in 200 Systematic Reviews: A Comparative Study of Robis and Amstar-2 Tools. Research Synthesis Methods, (), -.
Chicago	Lunny C, Jain N, Nazari T, Kosaner Kliess M, Santos L, Goodman I, Osman AAM, et al.. "Exploring the Methodological Quality and Risk of Bias in 200 Systematic Reviews: A Comparative Study of Robis and Amstar-2 Tools." Research Synthesis Methods , no. (2025): -.
Harvard	Lunny C et al. (2025) 'Exploring the Methodological Quality and Risk of Bias in 200 Systematic Reviews: A Comparative Study of Robis and Amstar-2 Tools', Research Synthesis Methods, (), pp. -.
Vancouver	Lunny C, Jain N, Nazari T, Kosaner Kliess M, Santos L, Goodman I, et al.. Exploring the Methodological Quality and Risk of Bias in 200 Systematic Reviews: A Comparative Study of Robis and Amstar-2 Tools. Research Synthesis Methods. 2025;():-.
BibTex	@article{ author = {Lunny C and Jain N and Nazari T and Kosaner Kliess M and Santos L and Goodman I and Osman AAM and Berrone S and Dadam MN and Brenna CTA and Hussein H and Dahdal G and Cespedes DA and Ferri N and Kanji S and Chi Y and Pieper D and Shea B and Parker A and Neupane D and Khan PA and Rangira D and Kolaski K and Ridley B and Berour A and Sun K and Hamidi Rad RH and Ouyang Z and Reid EK and Perezneri I and Barakat SO and Bargeri S and Gianola S and Castellini G and Whitelaw S and Stevens A and Kolekar SB and Wong K and Major P and Bagheri E and Tricco AC}, title = {Exploring the Methodological Quality and Risk of Bias in 200 Systematic Reviews: A Comparative Study of Robis and Amstar-2 Tools}, journal = {Research Synthesis Methods}, volume = {}, number = {}, pages = {-}, year = {2025} }
RIS	TY - JOUR AU - Lunny C AU - Jain N AU - Nazari T AU - Kosaner Kliess M AU - Santos L AU - Goodman I AU - Osman AAM AU - Berrone S AU - Dadam MN AU - Brenna CTA AU - Hussein H AU - Dahdal G AU - Cespedes DA AU - Ferri N AU - Kanji S AU - Chi Y AU - Pieper D AU - Shea B AU - Parker A AU - Neupane D AU - Khan PA AU - Rangira D AU - Kolaski K AU - Ridley B AU - Berour A AU - Sun K AU - Hamidi Rad RH AU - Ouyang Z AU - Reid EK AU - Perezneri I AU - Barakat SO AU - Bargeri S AU - Gianola S AU - Castellini G AU - Whitelaw S AU - Stevens A AU - Kolekar SB AU - Wong K AU - Major P AU - Bagheri E AU - Tricco AC TI - Exploring the Methodological Quality and Risk of Bias in 200 Systematic Reviews: A Comparative Study of Robis and Amstar-2 Tools JO - Research Synthesis Methods VL - IS - SP - EP - PY - 2025 ER -

Style

Citing Format

MLA

Lunny C, et al.. "Exploring the Methodological Quality and Risk of Bias in 200 Systematic Reviews: A Comparative Study of Robis and Amstar-2 Tools." Research Synthesis Methods, vol. , no. , 2025, pp. -.

APA

Lunny C, Jain N, Nazari T, Kosaner Kliess M, Santos L, Goodman I, Osman AAM, Berrone S, Dadam MN, Brenna CTA, Hussein H, Dahdal G, Cespedes DA, Ferri N, Kanji S, Chi Y, Pieper D, Shea B, Parker A, ... Tricco AC (2025). Exploring the Methodological Quality and Risk of Bias in 200 Systematic Reviews: A Comparative Study of Robis and Amstar-2 Tools. Research Synthesis Methods, (), -.

Chicago

Lunny C, Jain N, Nazari T, Kosaner Kliess M, Santos L, Goodman I, Osman AAM, et al.. "Exploring the Methodological Quality and Risk of Bias in 200 Systematic Reviews: A Comparative Study of Robis and Amstar-2 Tools." Research Synthesis Methods , no. (2025): -.

Harvard

Lunny C et al. (2025) 'Exploring the Methodological Quality and Risk of Bias in 200 Systematic Reviews: A Comparative Study of Robis and Amstar-2 Tools', Research Synthesis Methods, (), pp. -.

Vancouver

Lunny C, Jain N, Nazari T, Kosaner Kliess M, Santos L, Goodman I, et al.. Exploring the Methodological Quality and Risk of Bias in 200 Systematic Reviews: A Comparative Study of Robis and Amstar-2 Tools. Research Synthesis Methods. 2025;():-.

BibTex

@article{ author = {Lunny C and Jain N and Nazari T and Kosaner Kliess M and Santos L and Goodman I and Osman AAM and Berrone S and Dadam MN and Brenna CTA and Hussein H and Dahdal G and Cespedes DA and Ferri N and Kanji S and Chi Y and Pieper D and Shea B and Parker A and Neupane D and Khan PA and Rangira D and Kolaski K and Ridley B and Berour A and Sun K and Hamidi Rad RH and Ouyang Z and Reid EK and Perezneri I and Barakat SO and Bargeri S and Gianola S and Castellini G and Whitelaw S and Stevens A and Kolekar SB and Wong K and Major P and Bagheri E and Tricco AC}, title = {Exploring the Methodological Quality and Risk of Bias in 200 Systematic Reviews: A Comparative Study of Robis and Amstar-2 Tools}, journal = {Research Synthesis Methods}, volume = {}, number = {}, pages = {-}, year = {2025} }

RIS

TY - JOUR
AU - Lunny C
AU - Jain N
AU - Nazari T
AU - Kosaner Kliess M
AU - Santos L
AU - Goodman I
AU - Osman AAM
AU - Berrone S
AU - Dadam MN
AU - Brenna CTA
AU - Hussein H
AU - Dahdal G
AU - Cespedes DA
AU - Ferri N
AU - Kanji S
AU - Chi Y
AU - Pieper D
AU - Shea B
AU - Parker A
AU - Neupane D
AU - Khan PA
AU - Rangira D
AU - Kolaski K
AU - Ridley B
AU - Berour A
AU - Sun K
AU - Hamidi Rad RH
AU - Ouyang Z
AU - Reid EK
AU - Perezneri I
AU - Barakat SO
AU - Bargeri S
AU - Gianola S
AU - Castellini G
AU - Whitelaw S
AU - Stevens A
AU - Kolekar SB
AU - Wong K
AU - Major P
AU - Bagheri E
AU - Tricco AC
TI - Exploring the Methodological Quality and Risk of Bias in 200 Systematic Reviews: A Comparative Study of Robis and Amstar-2 Tools
JO - Research Synthesis Methods
VL -
IS -
SP -
EP -
PY - 2025
ER -

Science Communicator Platform

Authors

Abstract