Paper Reading skill for Codex

A useful paper summary separates what the authors claim, what their evidence actually supports, and what a practitioner should do with it. Restating the abstract in different words is not a reading.

Get the right version

• arXiv: check the version history (/abs/2401.01234v3) and whether a peer-reviewed version exists (venue, journal DOI). Results sometimes change between versions.
• Look for the official code and data links, an OpenReview page with reviews (ICLR, NeurIPS and others publish them), errata and retraction notices.
• If the user supplies a PDF, extract text with pdftotext -layout paper.pdf - or $pdf-processing; check figures and tables visually, since extraction mangles them.

Fetching papers and code needs network; in an offline sandbox, work from the provided file and list what you could not check.

Three passes

Pass 1: shape (10 minutes)

Read title, abstract, introduction’s contribution list, section headings, figures and tables, conclusion. Answer:

1. What problem, for whom, and why existing approaches fall short (according to authors)?
2. What are the claimed contributions? List them, numbered, in the authors’ terms.
3. What kind of paper: new method, empirical study, benchmark/dataset, theory, system, position, survey?
4. Is it worth a deeper read for the user’s purpose? If not, stop and say why.

Pass 2: evidence

For each claimed contribution, find the experiment or proof that supports it and judge it.

Check	Questions
Baselines	Are they strong, recent and tuned with similar effort? Missing the obvious simple baseline?
Datasets	Standard and appropriate? Same splits as prior work? Any overlap between train and test?
Contamination	For LLM work: could the benchmark be in the pretraining data? Was it checked?
Metrics	Do they measure the claim? Is the improvement within noise?
Statistics	Multiple seeds/runs? Variance, confidence intervals or significance tests reported?
Ablations	Does removing each component show it matters?
Compute	Is the gain explained by more parameters, data or compute rather than the idea?
Generality	Tested on more than one dataset, domain, scale or model family?
Cherry-picking	Qualitative examples selected how? Are failure cases shown?
Theory	Are assumptions realistic? Do proofs cover the setting used in experiments?

Write down the single most important table or figure and what it actually shows.

Pass 3: reproducibility and limits

• Code, weights and data released? Licence? Does the repo match the paper’s version?
• Are hyperparameters, prompts, hardware and training time given?
• What do the authors list as limitations, and what did they leave out?
• Has anyone reproduced or refuted it? Search for the title plus “reproduction”, “replication”, or check citing papers in Semantic Scholar.
• Conflicts of interest: does the authors’ organisation sell the method or product?

Trust rating

Rating	Meaning
Strong	Claims match evidence; solid baselines; multiple runs; code available; ideally reproduced
Promising	Good idea, evidence suggestive but limited (one dataset, single seed, no code)
Weak	Gains within noise, weak baselines, or key confounder unaddressed
Unsupported	Main claim not backed by the experiments shown

State the rating with the one or two reasons that decided it.

Explaining to the user

Match depth to the request:

• “Explain this paper”: plain-language problem, idea, result, and one analogy. Define each term on first use. Include a small worked example or pseudo-code of the core method.
• “Should we use this”: add applicability to their system (data size, latency, licence, engineering cost) and a minimal experiment to test it on their data.
• “Review this paper” (as a reviewer): structure as summary, strengths, weaknesses, questions for authors, minor issues, and an overall assessment. Be specific and fair; critique the work, never the authors.

Gotchas

• “State of the art” usually means on specific benchmarks at a specific date. Name them.
• Relative improvements hide scale: “30% error reduction” from 1.0% to 0.7% may not matter.
• Single-seed deep learning results often vary by more than the reported gain.
• Benchmarks saturate and leak; recent LLM papers on old benchmarks need contamination checks.
• Figures with truncated or log axes exaggerate differences. Read the numbers.
• Do not reproduce long passages, figures or tables verbatim from copyrighted papers. Summarise and cite with page or section numbers.
• Never invent numbers, author names, or venues. If the PDF is unreadable at a point, say so.
• For a set of papers on one topic, use $literature-review instead.

Verification

• Every number quoted in the summary appears in the paper (cite table or section).
• The contribution list matches the paper’s own; disagreements with it are labelled as yours.
• Weaknesses are concrete (“Table 3 uses one seed; the 0.4 gain is within the 0.6 std in Table 7”), not generic (“more experiments needed”).

Output template

# <Paper title> (<venue/arXiv id, version>, <year>)
**One line:** <what they did and found>
**Type:** method | empirical | benchmark | theory | system | survey
**Trust:** Strong | Promising | Weak | Unsupported - <reason>

## Problem and idea
## Claimed contributions
1. ...
## What the evidence shows
- Claim 1 -> <table/figure>: <supported / partly / not>, because ...
## Weaknesses and open questions
## Reproducibility
Code: <link/none> | Data: <...> | Reproduced by others: <yes/no/unknown>
## So what for us
<applicability, cost, the smallest experiment that would test it on our data>