Systematic Review skill for Codex

A systematic review is defined by its method, not its length: the question and methods are fixed in a protocol before results are seen, searches are reproducible, screening and extraction are done by two people independently, every study’s risk of bias is assessed, and the report accounts for every record from search to inclusion.

Health and policy reviews inform real decisions. Codex can draft protocols, search strings, forms, code and text, but a second human reviewer is required for screening and extraction, and an information specialist and a statistician should review the search and analysis. Say so in the plan.

Workflow

- [ ] 1. Frame the question (PICO or variant) and check for existing reviews
- [ ] 2. Write and register the protocol before screening
- [ ] 3. Build and peer-review the search; run it in 2+ databases plus grey literature
- [ ] 4. De-duplicate; screen titles/abstracts, then full texts, in duplicate
- [ ] 5. Extract data in duplicate with a piloted form
- [ ] 6. Assess risk of bias per study (per outcome where the tool requires)
- [ ] 7. Synthesise: meta-analysis if appropriate, else structured synthesis (SWiM)
- [ ] 8. Rate certainty with GRADE; write to PRISMA 2020

1. Question

Framework	Use for	Elements
PICO	Interventions	Population, Intervention, Comparator, Outcome(s) (+ study design, time)
PECO	Exposures	Population, Exposure, Comparator, Outcome
PIRD / PIT	Diagnostic accuracy	Population, Index test, Reference standard, Diagnosis
SPIDER	Qualitative evidence	Sample, Phenomenon of Interest, Design, Evaluation, Research type

Search PROSPERO, the Cochrane Library and recent literature for an existing or ongoing review first. If one exists and is recent, an update or a different question is better than a duplicate.

2. Protocol

Follow PRISMA-P. Fix: eligibility criteria, primary and secondary outcomes (with time points), databases, screening process, extraction items, risk-of-bias tool, effect measures, synthesis and heterogeneity plan, subgroup and sensitivity analyses, and certainty assessment. Register on PROSPERO (health outcomes) or OSF Registries (other fields) before screening starts. Any later change is reported as a deviation with reason.

3. Search

Build one concept block per PICO element (usually P and I only; outcomes are often poorly indexed), combining controlled vocabulary and free text:

#1 "Diabetes Mellitus, Type 2"[Mesh] OR "type 2 diabet*"[tiab] OR T2DM[tiab]
#2 "Telemedicine"[Mesh] OR telehealth[tiab] OR telemonitor*[tiab] OR "remote monitor*"[tiab] OR mHealth[tiab]
#3 randomized controlled trial[pt] OR randomi*[tiab] OR placebo[tiab] OR trial[tiab]
#4 #1 AND #2 AND #3

• Translate the strategy for each database (MEDLINE/PubMed, Embase, CENTRAL, plus discipline databases such as CINAHL, PsycINFO, Scopus, Web of Science, IEEE Xplore). Syntax, field tags and thesauri differ; never paste one string into all.
• Grey literature and registries: ClinicalTrials.gov, WHO ICTRP, preprint servers, theses, regulator documents. Plus citation chasing of included studies.
• Record for each source: interface, date run, full strategy, limits, hits. Report the full strategies in a supplement (PRISMA-S).
• Ask for a peer review of the search by an information specialist (PRESS checklist).
• Re-run searches shortly before final analysis if months have passed.

Database access usually needs institutional login and network; the user runs searches that require credentials, and exports results (RIS, CSV) into the project.

4. Screening

• De-duplicate across databases (reference manager or deduplication tool), keeping a count of removed duplicates.
• Pilot the criteria on 50-100 records with both reviewers; refine wording, not scope.
• Two reviewers screen independently (Rayyan, Covidence, or ASReview for active-learning prioritisation). Report Cohen’s kappa or percent agreement; resolve conflicts by discussion or a third reviewer.
• Full-text exclusions need a single primary reason each; those reasons appear in the flow diagram.

5. Extraction

Build the form in a spreadsheet or the screening tool; pilot on 2-3 studies. Typical fields: study ID, design, setting, country, funding, participant characteristics, arms and n, intervention details, comparator, outcomes with definitions and time points, effect data (means, SDs, events, n), risk-of-bias judgements, notes. Extract in duplicate; contact authors for missing data and record the attempts.

6. Risk of bias

Design	Tool
Randomised trials	Cochrane RoB 2 (per outcome, five domains)
Non-randomised studies of interventions	ROBINS-I (or its updated version)
Diagnostic accuracy	QUADAS-2
Observational exposure studies	ROBINS-E or a validated scale agreed in the protocol
Systematic reviews (overviews)	AMSTAR 2, ROBIS

Present a traffic-light summary per study and domain; robvis in R draws these.

7. Synthesis

Meta-analyse only when studies are similar enough in population, intervention, comparator and outcome. Default model: random effects with REML for tau-squared and Hartung-Knapp-Sidik-Jonkman confidence intervals, in R metafor or meta:

library(metafor)
dat <- escalc(measure = "SMD", m1i = m1, sd1i = sd1, n1i = n1,
              m2i = m2, sd2i = sd2, n2i = n2, data = studies)
res <- rma(yi, vi, data = dat, method = "REML", test = "knha")
summary(res); predict(res)          # includes the prediction interval
forest(res); funnel(res)

• Report heterogeneity (tau-squared, I-squared) and the prediction interval, not only the pooled estimate.
• Funnel plots and Egger’s test only with roughly 10 or more studies.
• Run pre-specified sensitivity analyses (excluding high risk-of-bias studies) and label any unplanned analysis as post hoc.
• When meta-analysis is not appropriate, follow SWiM guidance: group studies, use a consistent effect direction metric, and tabulate; never vote-count significance.

8. Certainty and reporting

Rate each key outcome with GRADE (high, moderate, low, very low), considering risk of bias, inconsistency, indirectness, imprecision and publication bias. Present a Summary of Findings table. Write the report against the PRISMA 2020 checklist and include the flow diagram. For the item-by-item reporting checklist and the flow-diagram template, read references/prisma-reporting.md when drafting the manuscript or checking it before submission.

Gotchas

• Changing outcomes or criteria after seeing results without reporting it is outcome switching; the protocol exists to prevent this.
• Search filters for study design save time but cut sensitivity; use validated filters.
• Multiple reports of one study (conference abstract, journal paper, follow-up) count as one study; link them before analysis.
• Mixing change scores and final values, or different scales, needs care (SMD, or converting); check units of every extracted number.
• Single-reviewer screening by an AI or a person alone does not meet the method; AI can prioritise or act as a check, documented in methods.
• Narrative reviews dressed as systematic ones get rejected; if the protocol cannot be followed, call it a scoping or narrative review ($literature-review).

Verification

• Flow diagram numbers add up at every stage.
• Every included study has extraction and risk-of-bias records from two reviewers.
• Search strategies in the supplement reproduce the reported hit counts.
• All PRISMA 2020 items are addressed or marked not applicable with reason.
• Analysis code reruns from the extraction file to the same estimates.

Output (project structure)

review/
  protocol.md            # PRISMA-P, registration ID, deviations log
  search/                # one file per database: strategy, date, hits, exports (RIS)
  screening/             # decisions with reasons, kappa calculation
  extraction/form.xlsx   # piloted form; dual extraction + reconciled sheet
  rob/                   # risk-of-bias judgements with support quotes
  analysis/              # R scripts, outputs, forest and funnel plots
  manuscript/            # report, PRISMA checklist, flow diagram, SoF tables