Why plain vector search isn't enough

Vector search matches on meaning, which is powerful — and exactly why it sometimes fails:

• It misses exact terms. A query for the error code "ERR_2043" or a product name "Xeon-9" may not land near the right chunk in embedding space, because those tokens carry little semantic signal. Keyword search nails them.
• The right chunk is often present but not #1. First-pass retrieval is a cheap filter; it reliably gets the answer into the top-20, but the generator only reads the top-4. Ordering matters.
• Top-k is often redundant. The four closest chunks can be four near-copies of the same passage, wasting context and starving the model of complementary evidence.

Four techniques address these: hybrid search, reranking, query rewriting, and MMR.

1. Hybrid search (keyword + vector)

Run both a keyword search (BM25 — the classic term-frequency ranking behind Postgres full-text, Elasticsearch, Azure AI Search) and a vector search, then fuse their results. The two are complementary: keyword catches exact tokens, vector catches paraphrases.

The clean way to combine them is Reciprocal Rank Fusion (RRF) — it uses only each result's rank position, not its raw score, so you never have to reconcile BM25 scores (unbounded) with cosine scores (0–1). An item's fused score is the sum of 1 / (K + rank) across the lists it appears in:

def reciprocal_rank_fusion(ranked_lists: list[list[str]], K: int = 60) -> list[str]:
    # ranked_lists: several result lists (e.g. [keyword_ids, vector_ids]), each already ordered best-first.
    scores: dict[str, float] = {}                     # doc_id -> fused score (a dict used as an accumulator)
    for results in ranked_lists:                       # for each ranked list…
        for rank, doc_id in enumerate(results, start=1):   # rank = 1, 2, 3, …  (1-based position)
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (K + rank)  # add this list's contribution
    # Sort ids by fused score, highest first; `key` picks the score, reverse=True = descending.
    return sorted(scores, key=lambda doc_id: scores[doc_id], reverse=True)

# A doc ranked #1 by keyword AND #2 by vector accumulates two contributions → it rises to the top.
fused = reciprocal_rank_fusion([keyword_ids, vector_ids])   # e.g. ["c7", "c3", "c9", …]

K (commonly 60) dampens the difference between top ranks so no single list dominates. RRF is the default fusion in many hybrid engines because it's parameter-light and robust.

2. Reranking with a cross-encoder

First-pass retrieval (vector or hybrid) is a bi-encoder: query and chunks are embedded separately, so it's fast (embed once, reuse) but coarse — it never looks at the query and chunk together. A cross-encoder does: it feeds (query, chunk) as one input to a model that outputs a single relevance score. Far more accurate, far too slow to run over the whole corpus — so you use it as a second stage:

retrieve a cheap top-N (e.g. 20–50) → rerank with a cross-encoder → keep the top-k (e.g. 4).

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")   # a small, fast cross-encoder

def rerank(query: str, candidates: list[str], k: int = 4) -> list[str]:
    pairs = [(query, chunk) for chunk in candidates]       # build (query, chunk) pairs — the cross-encoder input
    scores = reranker.predict(pairs)                        # one relevance score per pair (query+chunk seen together)
    # Pair each candidate with its score, sort by score desc, take the top k:
    ranked = sorted(zip(candidates, scores), key=lambda cs: cs[1], reverse=True)
    return [chunk for chunk, _score in ranked[:k]]          # return just the top-k chunk texts

# Two-stage retrieval: cheap wide net, then precise reorder.
shortlist = search(question, k=25)      # first pass: fast bi-encoder / hybrid, get 25 candidates
top = rerank(question, shortlist, k=4)  # second pass: cross-encoder picks the best 4

Managed rerank APIs (Cohere Rerank, Azure AI Search semantic ranker, Voyage) do the same thing behind an endpoint. Reranking is often the single highest-ROI upgrade to a RAG pipeline.

3. Query rewriting & multi-query

Users write short, vague, or oddly-phrased questions ("it broke again"). You can improve retrieval by rewriting the query before searching — or generating several variations and unioning their results:

def multi_query(question: str, n: int = 3) -> list[str]:
    # Ask an LLM to paraphrase the question n different ways to widen retrieval coverage.
    prompt = (f"Rewrite the question below in {n} different ways that a search system might match. "
              f"Return one per line, no numbering.\n\nQuestion: {question}")
    text = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
    ).choices[0].message.content
    return [line.strip() for line in text.splitlines() if line.strip()]   # split lines, drop blanks (like filter+trim)

def retrieve_multi(question: str, k: int = 4) -> list[str]:
    variants = [question] + multi_query(question)     # the original plus its paraphrases
    lists = [search(v, k=10) for v in variants]        # retrieve for each variant → several ranked lists
    return reciprocal_rank_fusion(lists)[:k]           # fuse them all with RRF, keep top-k

Related patterns you'll meet: HyDE (generate a hypothetical answer, embed that instead of the question — often closer to the target chunks in embedding space) and query decomposition (split a multi-part question into sub-questions retrieved independently).

4. MMR — diversity in the top-k

Your top-4 by similarity can be four near-duplicates. Maximal Marginal Relevance (MMR) selects chunks that are relevant to the query and different from what you've already picked, controlled by a knob λ (relevance vs. diversity):

import numpy as np

def mmr(query_vec, cand_vecs: list[np.ndarray], lambda_: float = 0.7, k: int = 4) -> list[int]:
    selected: list[int] = []                                  # indices we've chosen so far
    remaining = list(range(len(cand_vecs)))                   # candidate indices still available
    while remaining and len(selected) < k:
        best_i, best_score = None, -1e9
        for i in remaining:                                   # score every remaining candidate…
            relevance = cos(query_vec, cand_vecs[i])          # how well it matches the query
            # redundancy = the MAX similarity to anything already selected (0 if nothing selected yet):
            redundancy = max((cos(cand_vecs[i], cand_vecs[j]) for j in selected), default=0.0)
            score = lambda_ * relevance - (1 - lambda_) * redundancy   # reward relevance, penalize overlap
            if score > best_score:                            # keep the best candidate this round
                best_i, best_score = i, score
        selected.append(best_i)                               # commit the winner…
        remaining.remove(best_i)                              # …and remove it from the pool
    return selected                                           # k indices: relevant AND mutually diverse
# λ = 1.0 → pure relevance (ignore diversity);  λ = 0.0 → pure diversity.  ~0.7 is a common start.

MMR is a light, LLM-free reranking pass. It shines when your corpus has lots of near-duplicate passages (FAQs, versioned docs) that would otherwise crowd out complementary evidence.

Putting it together

A strong production retrieval pipeline usually stacks these:

query → [rewrite / multi-query] → hybrid search (BM25 + vector) → RRF fuse
      → cross-encoder rerank → MMR (optional) → top-k → generate

You don't need all of it on day one. The usual highest-ROI order to add them: hybrid search, then reranking, then query rewriting/MMR where the failure analysis says you need them — each validated against the golden set from RAG Fundamentals.

Pitfalls

• Fusing raw scores. BM25 and cosine live on different scales; fuse by rank (RRF), not raw score.
• Reranking too few candidates. If the first pass didn't retrieve the answer at all, no reranker can reorder it in — widen the shortlist (N) before blaming the reranker.
• Rewriting everything. Query expansion on already-clear queries adds cost/latency and can drift intent.
• Chasing diversity blindly. Low MMR λ can push genuinely-relevant chunks out; keep λ relevance-weighted (~0.7).
• No measurement. Every one of these is a hypothesis — validate each against the golden set, don't stack them on faith.

Recap

• Pure vector search misses exact terms, buries the answer below #1, and returns redundant chunks — the four techniques here fix each.
• Hybrid search runs keyword + vector and fuses by rank with RRF (no score-scale tuning).
• Cross-encoder reranking re-scores (query, chunk) together as a precise second stage over a cheap shortlist — usually the biggest single quality win.
• Query rewriting / multi-query (and HyDE, decomposition) rescue vague queries; use them selectively.
• MMR trades a little relevance for diversity to drop near-duplicates from the top-k.
• Stack them incrementally, validating each against the golden set.
• Worked examples map to the exercises: reciprocal_rank_fusion (Ex 1), rerank (Ex 2), multi_query (Ex 3), mmr (Ex 4).

keyword_ids = ["c1", "c8", "c3", "c5"]   # BM25 order
vector_ids  = ["c8", "c2", "c3", "c9"]   # cosine order

def rrf(ranked_lists, K=60):
    scores = {}
    for results in ranked_lists:
        for rank, doc_id in enumerate(results, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (K + rank)
    return sorted(scores, key=lambda d: scores[d], reverse=True)

print(rrf([keyword_ids, vector_ids]))   # -> ['c8', 'c3', ...]

from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def rerank(query, candidates, k=4):
    scores = reranker.predict([(query, c) for c in candidates])
    ranked = sorted(zip(candidates, scores), key=lambda cs: cs[1], reverse=True)
    return [c for c, _ in ranked[:k]]

shortlist = search(question, k=25)          # first pass buries the best chunk mid-list
print("first-pass top4:", shortlist[:4])    # answer chunk likely absent
print("reranked top4:", rerank(question, shortlist))   # answer chunk now present

def multi_query(q, n=3):
    prompt = (f"Rewrite the question in {n} different ways a search system might match. "
              f"One per line, no numbering.\n\nQuestion: {q}")
    text = client.chat.completions.create(model="gpt-4o-mini",
        messages=[{"role":"user","content":prompt}]).choices[0].message.content
    return [l.strip() for l in text.splitlines() if l.strip()]

variants = [q] + multi_query(q)
fused = rrf([search(v, k=10) for v in variants])[:4]

import numpy as np
def mmr(qv, cvs, lambda_=0.7, k=4):
    selected, remaining = [], list(range(len(cvs)))
    while remaining and len(selected) < k:
        best_i, best = None, -1e9
        for i in remaining:
            rel = cos(qv, cvs[i])
            red = max((cos(cvs[i], cvs[j]) for j in selected), default=0.0)
            s = lambda_ * rel - (1 - lambda_) * red
            if s > best: best_i, best = i, s
        selected.append(best_i); remaining.remove(best_i)
    return selected

cand = search(q, k=8)
cvs = [np.array(embed(c)) for c in cand]
qv = np.array(embed(q))
print("λ=1.0:", [cand[i][:40] for i in mmr(qv, cvs, 1.0)])   # may repeat near-duplicates
print("λ=0.5:", [cand[i][:40] for i in mmr(qv, cvs, 0.5)])   # surfaces a distinct second angle