Hybrid Search and Reranking skill for Codex

Done well means: lexical and dense retrievers each return a deep candidate list, fusion is rank-based unless score weights were tuned on labelled queries, a reranker reorders only the top candidates within a latency budget, and every change is justified by Recall@k and nDCG@k on a judged query set rather than by eyeballing three searches.

Why hybrid, and where each half fails

Query shape	BM25 / full-text	Dense vectors
Exact identifiers (ERR_4012, SKU-88-B, function names)	Strong	Weak: tokens get smeared
Paraphrase, synonyms, “how do I…”	Weak	Strong
Rare proper nouns, new product names	Strong	Weak if the embedding model never saw them
Multilingual, query and doc in different languages	Fails	Works with a multilingual model
Very short queries (1-2 words)	Strong	Noisy

Hybrid wins when your query log mixes these shapes, which is almost always true for product, support and code search. If the log is pure natural-language questions over prose, measure before adding the lexical half; it can add noise.

Workflow

- [ ] 1. Collect 50-200 real queries with judged relevant docs (qrels); include ID-style queries
- [ ] 2. Baseline: run lexical and dense separately, save TREC runs, score them
- [ ] 3. Fuse with RRF (k=60, depth 50-100 per side); score the fused run
- [ ] 4. Add a reranker over the fused top 50-100; score again and record p95 latency
- [ ] 5. Tune one knob at a time (depth, k, weights, rerank top_n) against the same qrels
- [ ] 6. Ship behind a flag, log both rankings, compare click/answer metrics online

Read AGENTS.md first for the search stack, test command and any existing eval harness. If the repo already has a RAG eval set, reuse its queries: see $rag-evaluation.

Measure with the bundled script

Export each retriever’s results as TREC run lines (qid Q0 doc_id rank score tag) and the judgments as qrels (qid 0 doc_id relevance), then:

python3 scripts/rrf_eval.py --qrels qrels.txt bm25.run dense.run --fuse --k 60 --depth 100 \
  --cutoff 10 --write-fused fused.run

It prints Recall@10, MRR@10 and nDCG@10 for each run and the RRF fusion, and writes the fused run so you can feed it to a reranker and score that too. Standard library only.

Fusion: pick RRF first

Reciprocal Rank Fusion scores each document as sum(1 / (k + rank_i)) over the retrievers that returned it. It ignores raw scores, so BM25’s unbounded scores and cosine similarities never need calibrating.

Situation	Fusion
No labelled data, or score scales differ	RRF, k = 60
Labelled queries and one side clearly stronger	Weighted RRF (multiply each term by a weight)
Need a tunable dial for product owners	Convex combination a * norm(dense) + (1-a) * norm(bm25) with min-max normalisation per query, a tuned on qrels
One retriever returns nothing for most queries	Do not fuse blindly; route by query type instead

Retrieval depth matters more than k: fusing two top-10 lists cannot recover a document ranked 30th by both. Pull 50-100 per side, fuse, then cut.

Postgres: full-text + pgvector in one query

WITH dense AS (
  SELECT id, row_number() OVER (ORDER BY embedding <=> $1) AS r
  FROM chunks
  ORDER BY embedding <=> $1
  LIMIT 100
), lexical AS (
  SELECT c.id, row_number() OVER (ORDER BY ts_rank_cd(c.tsv, q) DESC) AS r
  FROM chunks c, websearch_to_tsquery('english', $2) AS q
  WHERE c.tsv @@ q
  ORDER BY ts_rank_cd(c.tsv, q) DESC
  LIMIT 100
)
SELECT id,
       COALESCE(1.0 / (60 + dense.r), 0) + COALESCE(1.0 / (60 + lexical.r), 0) AS rrf
FROM dense FULL OUTER JOIN lexical USING (id)
ORDER BY rrf DESC
LIMIT 20;

ts_rank_cd is not BM25 (no IDF saturation). If lexical quality matters, use a BM25 extension such as ParadeDB pg_search, or an external engine. Index tuning for the dense half is covered in $pgvector-search.

Qdrant: dense + sparse with server-side fusion

from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://localhost:6333")
hits = client.query_points(
    collection_name="docs",
    prefetch=[
        models.Prefetch(query=models.SparseVector(indices=s_idx, values=s_val), using="sparse", limit=100),
        models.Prefetch(query=dense_vec, using="dense", limit=100),
    ],
    query=models.FusionQuery(fusion=models.Fusion.RRF),
    limit=20,
    with_payload=True,
).points

The sparse vectors come from BM25-style or SPLADE encoders computed at ingest; Qdrant can apply IDF on sparse vectors when the sparse config sets the IDF modifier.

Elasticsearch / OpenSearch

• Elasticsearch: an rrf retriever wrapping a standard (match query) retriever and a knn retriever; set rank_window_size to your fusion depth. Check that your license tier includes RRF before designing around it.
• OpenSearch: a hybrid query plus a search pipeline with a normalization-processor (min-max + arithmetic mean weights) or the RRF processor in recent versions.
• Weaviate: collection.query.hybrid(query=..., alpha=0.5); alpha=1 is pure vector, alpha=0 pure BM25. Tune alpha on qrels, do not leave it at the default by habit.

Reranking

A reranker scores (query, passage) pairs jointly, so it catches relevance the bi-encoder missed. It is the single biggest precision gain in most RAG stacks, and the most expensive step.

Option	When	Notes
Hosted API (Cohere Rerank, Voyage, Jina)	Fastest to ship, multilingual	Network call per query; send only passage text you may share
Open cross-encoder (BAAI/bge-reranker-v2-m3, cross-encoder/ms-marco-MiniLM-L-6-v2)	Data must stay local, steady traffic	GPU for > ~20 QPS at 100 candidates
LLM-as-reranker (listwise prompt)	Small candidate sets, complex relevance rules	Slow and costly; cache aggressively
No reranker	Latency budget < ~50 ms or fused nDCG already near ceiling	Measure first

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)

def rerank(query: str, candidates: list[dict], top_n: int = 8) -> list[dict]:
    pairs = [(query, c["text"]) for c in candidates]
    scores = reranker.predict(pairs, batch_size=32)
    ranked = sorted(zip(candidates, scores), key=lambda t: float(t[1]), reverse=True)
    return [dict(c, rerank_score=float(s)) for c, s in ranked[:top_n]]

import cohere

co = cohere.ClientV2()  # reads CO_API_KEY; needs network
res = co.rerank(model="rerank-v3.5", query=query, documents=[c["text"] for c in cands], top_n=8)
top = [cands[r.index] | {"rerank_score": r.relevance_score} for r in res.results]

Downloading model weights or calling a hosted API needs network; in a Codex sandbox with network off, ask for approval or use weights already cached in the project.

Rerank rules:

• Rerank 50-100 candidates, return 5-10. Reranking 1,000 burns latency for little gain.
• Truncate passages to the reranker’s max length deliberately (title + first N tokens), not by silent tokenizer truncation in the middle of a table.
• Rerank scores are not calibrated probabilities across queries. A fixed cutoff like score > 0.3 needs validation on your qrels before it gates what the LLM sees.
• Keep the fused rank as a tie-breaker and log both orders for debugging.

Gotchas

• Tokenisation mismatch: BM25 with a stemming English analyzer mangles code identifiers. Index a second field with a keyword or whitespace analyzer for IDs and paths.
• Filters must apply to both retrievers. Filtering only the dense side lets forbidden or out-of-tenant documents in through the lexical side. Treat this as a security bug.
• Chunk-level vs document-level: dedupe by parent document after fusion if the UI shows documents, otherwise one long doc floods the top 10.
• Score-based fusion without per-query normalisation is wrong: BM25 scores vary by query length.
• Empty lexical results (stop-word-only queries) are normal; RRF handles them, weighted sums with fixed normalisation often do not.
• Rerankers trained on web search (MS MARCO) can underperform on legal or code text. Compare two rerankers on your qrels before committing.

Verification

• scripts/rrf_eval.py shows fused Recall@k at or above the better single retriever. If fusion is worse, check depth, ID mismatches between runs, and filter parity.
• Reranked nDCG@10 beats fused nDCG@10 on the same qrels; if not, the reranker or its truncation is wrong for the domain.
• p95 end-to-end retrieval latency is inside the budget (state it, e.g. 300 ms) under the expected concurrency.
• Spot-check the 10 queries where the new pipeline lost to the baseline; write down why.

Report template

**Query set:** 140 judged queries (38 identifier-style), qrels v3
| Pipeline | Recall@10 | nDCG@10 | p95 ms |
|---|---|---|---|
| BM25 (english + keyword field) | 0.61 | 0.48 | 18 |
| Dense (text-embedding-3-small, HNSW) | 0.66 | 0.51 | 22 |
| RRF k=60, depth 100 | 0.78 | 0.58 | 41 |
| RRF + bge-reranker-v2-m3 top 60 -> 8 | 0.78 | 0.69 | 165 |
**Decision:** ship RRF + rerank behind `search.hybrid`; tenant filter applied to both sides
**Regressions:** 6 queries, all long natural-language questions where BM25 added noise