Literature Review skill for Codex

A good review is organised by ideas, not by paper. Each section makes a claim about the field, supports it with several works, says where they disagree and why, and ends with what remains unknown. A reader should be able to rerun your search and get the same set.

Choose the review type first

Need	Type	Rigor
Related-work section for a paper or design doc	Narrative / focused	Documented search, 15-50 works
Map what exists on a broad topic, find gaps	Scoping review	Protocol-light, explicit inclusion criteria, charting table
Answer an effect-size question with formal appraisal	Systematic review	Full protocol and PRISMA: use $systematic-review

If the user needs claims like “intervention X improves Y”, that is a systematic review; say so rather than producing a narrative review that looks authoritative.

Workflow

- [ ] 1. Write the review question and inclusion criteria
- [ ] 2. Build the search strings and log every search
- [ ] 3. Collect candidates, de-duplicate by DOI / arXiv ID
- [ ] 4. Screen titles and abstracts against criteria
- [ ] 5. Chain citations from the core set (backward and forward)
- [ ] 6. Read included works and fill the concept matrix
- [ ] 7. Write the synthesis by theme, then the gaps
- [ ] 8. Verify every citation resolves and says what you claim

1. Question and criteria

Write these down before searching; they decide every later inclusion.

Question: How do retrieval-augmented LLM systems evaluate faithfulness to sources?
Include: peer-reviewed or arXiv 2020-2026; English; proposes or compares a faithfulness metric
Exclude: pure summarisation without retrieval; position papers with no evaluation
Core concepts: retrieval augmentation | faithfulness / attribution / groundedness | evaluation

2. Search strings

Combine synonyms with OR inside a concept and AND across concepts. Search at least two databases because coverage differs.

("retrieval-augmented" OR "retrieval augmented" OR RAG) AND
(faithfulness OR attribution OR groundedness OR hallucination) AND
(evaluation OR benchmark OR metric)

Programmatic search needs network (ask for approval if the Codex sandbox blocks it):

# OpenAlex: broad, open metadata; add mailto for the polite pool
curl -s "https://api.openalex.org/works?search=retrieval%20augmented%20faithfulness&filter=publication_year:2020-2026&per-page=50&[email protected]" \
  | jq -r '.results[] | [.publication_year, .cited_by_count, .doi, .title] | @tsv'

# Semantic Scholar: good for CS, has citation graph endpoints
curl -s "https://api.semanticscholar.org/graph/v1/paper/search?query=retrieval+augmented+faithfulness&year=2020-2026&fields=title,year,citationCount,externalIds&limit=50"

Log each search: database, exact string, filters, date run, hit count. That log goes in the methods paragraph.

Source	Strength	Watch for
OpenAlex	Open, all disciplines, filters, citation counts	Abstracts sometimes missing
Semantic Scholar	CS/biomed, citation and reference endpoints	Rate limits without an API key
Google Scholar	Widest recall, grey literature	No API; results not reproducible; do not scrape
arXiv	Preprints, newest work	Not peer-reviewed; check for later published version
PubMed	Biomedicine, MeSH terms	Use MeSH plus free text
ACL Anthology, DBLP	NLP and CS venues	Metadata only

3-4. De-duplicate and screen

• Normalise on DOI, then arXiv ID, then lowercased title plus year. Preprint and journal versions of one work count once; cite the published version, note the preprint.
• Screen in two passes: title/abstract, then full text. Record the exclusion reason per paper (wrong population, no evaluation, out of date range). Keep the counts.

5. Citation chaining

From the 5-10 most central papers:

• Backward: read their reference lists for older foundational work.
• Forward: who cites them? (/graph/v1/paper/{id}/citations in Semantic Scholar, filter=cites:W123 in OpenAlex.)
• Stop when a round of chaining yields no new includable work.

6. Concept matrix

One row per included work, one column per theme or variable you compare. This is where the synthesis comes from.

| Work | Year | Method | Data | Metric(s) | Main finding | Limits |
|---|---|---|---|---|---|---|
| Author et al. | 2023 | NLI-based attribution | 3 QA sets | AutoAIS | correlates 0.6 with humans | English only |

For appraising individual papers in depth, apply $paper-reading.

7. Synthesis

Structure by theme, with each paragraph shaped as claim, evidence, tension, implication:

**Model-based faithfulness metrics agree with humans only moderately.** Three studies
report correlations between 0.5 and 0.7 on QA tasks [A; B; C], but [D] finds near-zero
correlation on long-form answers, likely because its annotators judged partial support
as unsupported. Agreement for long outputs remains untested at scale.

Rules:

• Every paragraph cites more than one work, or explicitly says the claim rests on one.
• Compare like with like: note when studies used different definitions or datasets.
• Separate what the papers show from what the authors speculate.
• Gaps section: questions nobody has studied, conflicting results, populations or settings missing, methods no one has tried.

Gotchas

• Citation count reflects age and field size, not quality. A 2025 paper with 12 citations may matter more than a 2019 paper with 900.
• Preprints get revised; check the latest arXiv version (v3) and whether results changed.
• Check for retractions before relying on a paper (Crossref metadata includes Retraction Watch data; the publisher page shows notices).
• Do not invent references. Every entry must come from a search result you opened. If metadata is incomplete, fetch it by DOI instead of filling gaps from memory.
• Summarising each paper in turn (“A did X. B did Y.”) is an annotated bibliography, not a review. Rewrite into themes.
• Manage the reference list with $citation-management (BibTeX, CSL styles).

Verification

• Every citation key resolves to a real DOI or URL and the cited sentence is supported.
• The search log reproduces: rerun one query and confirm the hit count is close.
• Each theme cites at least two works or is flagged as single-source.
• Counts add up: identified, duplicates removed, screened, excluded (by reason), included.

Output template

# <Topic>: literature review
## Scope and method
Question, inclusion/exclusion criteria, databases, search strings, dates run,
counts (identified -> deduplicated -> screened -> included), chaining rounds.

## Themes
### <Theme 1 as a claim>
### <Theme 2 as a claim>

## Disagreements and why

## Gaps and future work

## Limitations of this review
(language, databases not searched, single screener, date cut-off)

## References