LlamaIndex RAG skill for Codex

Done well means: ingestion is an incremental pipeline (unchanged documents are skipped), vectors live in a real vector store rather than the default in-memory one, every answer exposes its source_nodes, retrieval parameters were chosen from an eval set, and tests run without live model calls.

Orient first

pip show llama-index-core 2>/dev/null | grep Version
pip list 2>/dev/null | grep -i llama-index
grep -rn "Settings\.\|VectorStoreIndex\|IngestionPipeline\|StorageContext" --include=*.py . | head -30

• LlamaIndex is split into llama-index-core plus integration packages (llama-index-llms-openai, llama-index-embeddings-openai, llama-index-vector-stores-qdrant, llama-index-vector-stores-postgres, …). Imports for core live under llama_index.core.
• Read AGENTS.md for the package manager; adding integration packages needs network.
• Find where Settings (global LLM, embed model, chunking defaults) is configured. Hidden global defaults are the most common source of “it used the wrong model”.

Configure explicitly

import os
from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

Settings.llm = OpenAI(model=os.environ.get("CHAT_MODEL", "gpt-5-mini"), temperature=0, timeout=60)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

Set both in one module imported at startup. If you skip Settings.embed_model, LlamaIndex falls back to its default provider and can silently call an API you did not plan for. Pass llm= and embed_model= explicitly to components in libraries that must not depend on globals.

Ingestion pipeline

from llama_index.core import SimpleDirectoryReader
from llama_index.core.ingestion import IngestionPipeline, DocstoreStrategy
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.vector_stores.qdrant import QdrantVectorStore
import qdrant_client

client = qdrant_client.QdrantClient(url=os.environ["QDRANT_URL"])
vector_store = QdrantVectorStore(client=client, collection_name="kb")

pipeline = IngestionPipeline(
    transformations=[
        SentenceSplitter(chunk_size=800, chunk_overlap=100),
        Settings.embed_model,
    ],
    vector_store=vector_store,
    docstore=SimpleDocumentStore(),               # use a Redis/Postgres-backed docstore in production
    docstore_strategy=DocstoreStrategy.UPSERTS,
)

docs = SimpleDirectoryReader("data/", recursive=True, filename_as_id=True).load_data()
for d in docs:
    d.metadata["tenant_id"] = tenant_id
nodes = pipeline.run(documents=docs, show_progress=True)
pipeline.persist("./pipeline_storage")          # docstore + transformation cache

• filename_as_id=True (or setting doc.id_) gives stable document IDs; without them the docstore cannot detect changes and every run re-embeds everything.
• DocstoreStrategy.UPSERTS re-processes changed documents and replaces their nodes; UPSERTS_AND_DELETE also removes nodes of documents missing from the input. Use the delete strategy only when the input is the complete corpus, and confirm with the user first.
• Metadata set on documents propagates to nodes. Keep it small: by default metadata is also embedded and sent to the LLM. Hide noisy keys with doc.excluded_embed_metadata_keys and doc.excluded_llm_metadata_keys.

Chunking choices by document type (headings, tables, code) are covered in $rag-chunking. For PDFs with tables or scans, parse to markdown first ($document-ai-ocr), then ingest.

Build the index from the vector store

from llama_index.core import VectorStoreIndex

index = VectorStoreIndex.from_vector_store(vector_store)

Do not call VectorStoreIndex.from_documents in request handlers: it re-chunks and re-embeds. index.storage_context.persist() and load_index_from_storage are for the default local stores; with an external vector store, reconnect via from_vector_store instead.

Query engine with filters, cutoff and rerank

from llama_index.core.vector_stores import MetadataFilters, MetadataFilter, FilterOperator
from llama_index.core.postprocessor import SimilarityPostprocessor

filters = MetadataFilters(filters=[
    MetadataFilter(key="tenant_id", value=tenant_id, operator=FilterOperator.EQ),
])

query_engine = index.as_query_engine(
    similarity_top_k=20,
    filters=filters,
    node_postprocessors=[
        reranker,                                      # e.g. a cross-encoder or Cohere rerank postprocessor, top_n=5
        SimilarityPostprocessor(similarity_cutoff=0.2),
    ],
    response_mode="compact",
)

response = query_engine.query("How do I rotate API keys?")
print(response.response)
for sn in response.source_nodes:
    print(round(sn.score or 0, 3), sn.node.metadata.get("file_name"), sn.node.node_id)

Knob	Start	Move it when
similarity_top_k	10-20 with a reranker, 4-6 without	Context recall low (raise) or answers diluted (lower)
Reranker top_n	4-6	Answers miss facts spread over several chunks
response_mode	compact	tree_summarize for “summarize all of X” questions
similarity_cutoff	Off until measured	Scores vary per embedding model; validate before enabling

Rerankers and hybrid retrieval details: $hybrid-search-reranking. Reranker postprocessors come from integration packages (for example llama-index-postprocessor-cohere-rerank) or core SentenceTransformerRerank; check the installed package for the class name.

Retriever-only use

When another framework or your own code generates the answer, use just the retriever:

retriever = index.as_retriever(similarity_top_k=8, filters=filters)
nodes = retriever.retrieve(question)
context = "\n\n".join(f"[{i}] {n.node.get_content()}" for i, n in enumerate(nodes, 1))

Chat with history

index.as_chat_engine(chat_mode="condense_plus_context") rewrites follow-up questions using history, then retrieves. Store chat history per user/session server-side; never trust a client-sent history for tenant or permission data.

Agents and workflows

For multi-step or tool-using flows, LlamaIndex Workflows (event-driven steps) and its agent classes can wrap query engines as tools. Keep the query engine the unit of retrieval and test it on its own; the agent layer should only decide when to call it. Check the installed version’s docs before writing workflow code, since these APIs have changed across releases.

Testing

from llama_index.core import Document, Settings, VectorStoreIndex
from llama_index.core.llms import MockLLM
from llama_index.core.embeddings import MockEmbedding

def test_query_engine_returns_sources():
    Settings.llm = MockLLM(max_tokens=32)
    Settings.embed_model = MockEmbedding(embed_dim=8)
    index = VectorStoreIndex.from_documents([Document(text="Rotate keys in Settings > API.", id_="kb-1")])
    resp = index.as_query_engine(similarity_top_k=1).query("rotate keys")
    assert resp.source_nodes[0].node.ref_doc_id == "kb-1"

Mocks prove wiring, not quality. Measure retrieval and answer quality with an eval set ($rag-evaluation).

Gotchas

• Changing chunk_size or the embedding model without clearing the pipeline cache and vector store mixes old and new nodes. Re-ingest into a new collection and swap.
• Metadata filters are pushed to the vector store; operator support differs by store. Check that the filter actually narrows results (log result tenant IDs in a test).
• SimpleDirectoryReader parses PDFs page-by-page with a basic parser; tables come out as jumbled text. Use a layout-aware parser for table-heavy documents.
• Async: use aquery/arun in async servers; nest_asyncio hacks belong in notebooks only.
• Large ingestion runs: set num_workers on pipeline.run and batch embedding requests; respect provider rate limits.

Verification

• Second ingestion run over unchanged data processes 0 changed documents (check the returned node count or logs).
• A query for a known fact returns the expected ref_doc_id in source_nodes.
• Tenant filter test: query as tenant A never returns tenant B nodes.
• Eval metrics recorded before and after any change to chunking, top-k or reranking.

Output summary template

**Ingestion:** IngestionPipeline (SentenceSplitter 800/100 + text-embedding-3-small) -> Qdrant `kb`
**Incremental:** Postgres docstore, UPSERTS; rerun over unchanged corpus processed 0 docs
**Query engine:** top_k=20 -> rerank top_n=5, tenant filter, compact response mode
**Eval (90 questions):** context recall 0.81 -> 0.89, faithfulness 0.90 -> 0.92