RAG Chunking skill for Codex

Done well means: chunks follow the document’s own structure, each one makes sense on its own (with a contextual header), sizes are measured in tokens, tables and code blocks are never cut in half, metadata points back to the exact source location, and the chosen settings beat alternatives on a retrieval eval — not on intuition.

Defaults to start from

Document type	Split on	Target size	Overlap	Extra
Markdown / docs sites	Headings (H1–H3), then paragraphs	300–600 tokens	0–10%	Heading path as header
HTML pages	Semantic tags (h*, section, article) after boilerplate removal	300–600	0–10%	Drop nav, footer, cookie banners
PDFs (reports, manuals)	Detected headings; fall back to page + paragraph	400–800	10–15%	Page number metadata; strip running headers/footers
Contracts / policies	Clause numbering (1.2, 1.2.3)	One clause, merge tiny ones	0	Clause id as citation anchor
FAQs / tickets	One Q&A or one ticket per chunk	as-is up to ~800	0	Question text in header
Spreadsheets / CSV	Row groups with header row repeated	20–50 rows	0	Or convert rows to sentences; or query with SQL instead
Code	Function/class (AST or tree-sitter)	one symbol, cap ~800	0	File path + signature in header
Chat / transcripts	Speaker turns grouped by topic or time window	300–600	1–2 turns	Speaker and timestamp metadata

These are starting points. The right size depends on query type: short factual lookups favor smaller chunks; “explain how X works” favors larger ones or parent-child.

Workflow

• 1. Sample 20 documents per type and look at the extracted text (not the original)
• 2. Pick a strategy per type from the table
• 3. Implement token-aware splitting with structure boundaries
• 4. Add contextual headers and metadata
• 5. Inspect chunk statistics and 20 random chunks
• 6. Compare 2–3 configurations on the retrieval eval

3. Structure-aware, token-sized splitting

import re

def count_tokens(text: str) -> int:
    try:
        import tiktoken                       # optional dependency
        return len(tiktoken.get_encoding("cl100k_base").encode(text))
    except ImportError:
        return max(1, len(text) // 4)         # rough fallback

HEADING = re.compile(r"^(#{1,3})\s+(.*)$", re.M)

def split_markdown(doc_id: str, text: str, max_tokens=500, min_tokens=80):
    sections, path = [], []
    parts = HEADING.split(text)               # [pre, hashes, title, body, hashes, title, body...]
    if parts[0].strip():
        sections.append(([], parts[0]))
    for i in range(1, len(parts), 3):
        level, title, body = len(parts[i]), parts[i + 1].strip(), parts[i + 2]
        path = path[: level - 1] + [title]
        sections.append((list(path), body))

    chunks = []
    for heading_path, body in sections:
        buf = ""
        for block in split_blocks(body):      # paragraphs, but fenced code and tables whole
            if buf and count_tokens(buf + block) > max_tokens:
                chunks.append((heading_path, buf.strip()))
                buf = ""
            buf += block + "\n\n"
        if buf.strip():
            chunks.append((heading_path, buf.strip()))
    return merge_small(chunks, min_tokens)    # fold tiny chunks into their neighbor

def split_blocks(body: str):
    blocks, cur, in_fence = [], [], False
    for line in body.splitlines():
        if line.strip().startswith("```"):
            in_fence = not in_fence
        if not line.strip() and not in_fence and cur:
            blocks.append("\n".join(cur)); cur = []
        else:
            cur.append(line)
    if cur:
        blocks.append("\n".join(cur))
    return blocks

A single block larger than max_tokens (huge table, long code) gets its own chunk or is split by rows/lines with the table header or function signature repeated — never mid-row or mid-statement.

Libraries that already do this (LangChain MarkdownHeaderTextSplitter / RecursiveCharacterTextSplitter.from_tiktoken_encoder, LlamaIndex node parsers, unstructured partitioning) are fine if the repo already depends on them; installing new ones needs network and possibly approval.

4. Contextual headers and metadata

Prepend a short header to the text that gets embedded, so a chunk saying “this applies only to annual plans” carries what “this” is:

Document: Refund policy (v12) > Eligibility > Annual plans
<chunk text>

Optionally generate a one-sentence LLM summary of the chunk’s role in the document and prepend it (“contextual retrieval”). It improves recall on ambiguous chunks but costs one model call per chunk at ingest — measure before adopting, and run it through Batch.

Metadata per chunk: doc_id, chunk_index, heading_path, page (PDF), char_start/char_end or anchor for citation, doc_type, lang, updated_at, ACL. Store the raw text separately from the embedded text so citations show clean content.

Parent-child (small-to-big)

Embed small child chunks (100–200 tokens) for precise matching; at answer time return their parent section (800–1,500 tokens) for context. Deduplicate when several children share a parent. Use when users ask detailed questions whose answers need surrounding context.

5. Inspect

import statistics
sizes = [count_tokens(c.text) for c in chunks]
print(len(sizes), statistics.median(sizes), max(sizes), sum(s < 50 for s in sizes))

Red flags: many chunks under 50 tokens (headers alone, nav fragments), max far above the target (an unsplit table), identical chunks across documents (boilerplate), chunks starting mid-sentence.

6. Compare on retrieval eval

Run the same question set (see rag-evaluation) for 2–3 configurations, e.g. 300/0, 500/50 and parent-child. Choose by recall@k at the k you actually prompt with, then by index size and cost. Re-chunking requires re-embedding — budget it.

Gotchas

• Splitting by characters, not tokens: sizes vary 2–3× across languages and code.
• Fixed-size windows that cut tables, lists and code blocks in half.
• Overlap as a fix for bad boundaries: it inflates the index and duplicates hits; fix boundaries instead.
• PDF running headers/footers (“Confidential — Page 7”) repeated in every chunk dominate embeddings; strip them.
• Losing the heading path: chunk text alone often omits the subject.
• Changing chunking without bumping a prep_version leaves a mixed index.

Verification

• Chunk stats within target ranges; zero chunks over the embedding model’s input limit.
• 20 random chunks read by eye: each understandable alone, no broken tables or code.
• Retrieval eval of the chosen config ≥ the alternatives tested.

Output contract

| Doc type | Strategy | Size / overlap | Chunks | Median tokens |
|---|---|---|---|---|
| Docs site | headings → paragraphs, header path prefix | 500 / 0 | 18,240 | 312 |
| PDF manuals | headings, page metadata, footers stripped | 700 / 80 | 9,880 | 540 |
| Pricing CSV | 25-row groups, header repeated | n/a | 140 | 410 |

**Eval:** recall@8 300/0 = 0.79, 500/0 = 0.84, parent-child = 0.86 → chose parent-child
**prep_version:** 3 (re-embed scheduled via Batch)