Hybrid Search Made RAG Worse? Find Where the Right Chunk Disappeared

If adding lexical search or a reranker makes your RAG answers worse, first locate the earliest stage that loses useful evidence. Check whether the right chunk was absent from the candidate lists, pushed below a fusion cutoff, demoted by the reranker, or removed when the prompt was assembled. Changing the embedding model before separating those cases can fix the wrong problem.

This guide is for developers with a working retrieval-augmented generation pipeline and at least a small set of queries with known supporting passages. The diagnostic below is our proposed workflow. The numerical example is deliberately fictional and locally calculated, not a benchmark or a claim that hybrid search generally underperforms vector search.

Start with one failed query and a useful chunk ID

Choose a query whose answer is actually supported by the indexed corpus. Mark every acceptable supporting chunk, not just the first one you happened to find. A different passage can be equally valid; conversely, a passage mentioning the same product may not answer the question.

Freeze the query, corpus revision, access context, retrieval settings and model versions for the comparison. Then collect a small ledger:

Stage	Record	Question to answer
Lexical and vector retrieval	Ordered chunk IDs separately, candidate counts, query/filter versions	Did either arm find acceptable evidence?
Fusion	Input lists, fused ranks, retained IDs, cutoff	Did useful evidence survive the merge?
Reranking	Exact input and output IDs, model/version, cutoff	Was useful evidence available but demoted?
Context assembly	Selected IDs, source revisions, truncation decisions	Did the model receive the supporting text?
Answer	Supported claims and citations against the frozen evidence	Did generation use the available evidence correctly?

Keep this trace access-controlled and short-lived. IDs, queries and derived scores can themselves reveal sensitive information. Prefer synthetic or approved evaluation queries; do not turn debugging into a new store of customer prompts or full documents.

The figure’s order is the diagnostic: candidates → fusion → reranking → context. At each boundary, compare the acceptable chunk IDs before and after the operation. If your system has additional retrieval branches or retries, trace each branch explicitly rather than assuming this simple pipeline describes it.

The right chunk is missing before fusion. What should change?

Inspect the actual indexed text and query representation. An error code in the user question cannot produce an exact-token match if that code is absent from the indexed field. Check ingestion freshness, field selection, language processing, chunk boundaries and the vector query before changing ranking weights.

Also distinguish the requested candidate count from the number actually returned. In pgvector, approximate-index filtering happens after the index scan, so selective filters can leave fewer results. Iterative scans, available from 0.8.0, can search farther subject to scan limits. That is a product-specific diagnostic, not a rule for every vector database. Measure recall and latency under your real filter selectivity. pgvector filtering and iterative scans

Keep authorization filters intact. A relevant chunk the user cannot access is not a valid retrieval target. Use authorized evaluation labels and investigate retrieval inside that boundary. Our RAG permission-revocation guide covers the separate problem of keeping cached and retrieved evidence consistent with permissions.

A reranker that only reorders its input cannot recover an absent chunk. If your “reranker” also issues new searches, treat those searches as another retrieval stage and record them separately.

The chunk was retrieved. Why did reciprocal rank fusion bury it?

Reciprocal rank fusion, or RRF, combines positions in ranked lists. It does not simply add a BM25 score to a cosine score. In the one-based, equal-weight convention used for this example, a document receives 1 / (rank_constant + rank) from each list containing it, and those contributions are summed. Elastic documents this convention and a default rank_constant of 60. Elastic RRF

Consider three fictional chunks. Only A supports the exact answer; B and C are related but insufficient. The lexical list is [A, B, C], while the vector list is [B, C]. With a rank constant of 60:

Chunk	Lexical rank	Vector rank	Fused score, rounded
A, useful evidence	1	absent	0.016393
B, related only	2	1	0.032522
C, related only	3	2	0.032002

The fused order is B, C, A. A top-two cutoff removes A, even though one retriever placed it first. This calculation demonstrates a possible failure mechanism, not its frequency. Agreement between retrieval methods is a ranking signal, not proof that a chunk answers the question.

We ran the following self-contained calculation locally using Python’s standard library. It uses no embeddings, database or LLM:

from fractions import Fraction

lists = [["A", "B", "C"], ["B", "C"]]
scores = {}
for ranked in lists:
    assert len(ranked) == len(set(ranked))
    for rank, chunk in enumerate(ranked, start=1):
        scores[chunk] = scores.get(chunk, Fraction(0)) + Fraction(1, 60 + rank)
order = sorted(scores, key=lambda chunk: (-scores[chunk], chunk))
assert order == ["B", "C", "A"]
assert "A" not in order[:2]
assert "A" in order[:3]
print([(chunk, round(float(scores[chunk]), 6)) for chunk in order])

The output is [('B', 0.032522), ('C', 0.032002), ('A', 0.016393)]. The alphabetical tie-break is only for deterministic behavior in this example; no tie occurs here. Production systems may use different tie handling, weighting and rank conventions.

Do not copy the constant blindly across products. Qdrant’s current documentation uses zero-based positions and a default constant of 2 for its documented RRF formula. Its Query API executes prefetches first, then applies the main query to those results. Confirm the semantics of your deployed version before comparing scores or porting a configuration. Qdrant hybrid queries

Which limit should I increase?

Give each limit a separate name in the trace:

• Per-arm candidate budget: how many results each retriever contributes.
• Fusion window: how far down each list the fusion operation considers.
• Reranker input budget: how many fused candidates reach the reranker.
• Final context budget: how many passages, and how much text, reach generation.

Elastic’s rank_window_size limits individual input lists, while the request’s size limits the final ranked set. These are separate from rank_constant, which changes rank contributions rather than fetching missing candidates. Elastic RRF parameters

In the fictional example, retaining three fused candidates keeps A available for a later stage. It does not guarantee that a real reranker will promote A or that adding more context will improve an answer. Increase the budget at the demonstrated loss boundary, then measure the resulting quality, latency and variable inference cost. Do not widen every stage at once.

Did a score threshold silently remove the answer?

Record the score’s producer and meaning beside any threshold. Azure AI Search distinguishes BM25 scores, vector-search scores, RRF scores and semantic reranker scores. Its semantic reranking occurs after RRF and has a separate score. A threshold calibrated for one stage is therefore not automatically meaningful at another. Azure scoring documentation

For a safe offline test, compare the candidate IDs immediately before and after your cutoff on authorized examples. Evaluate both answerable and unanswerable questions: relaxing a cutoff may recover useful passages while also admitting more irrelevant ones. Tune abstention against the cost of unsupported answers, rather than treating a larger returned set as success.

The chunk reaches the reranker but the answer is still wrong

Separate two checks. First, compare the reranker’s input and output on the same candidate set. If useful passages move below the output cutoff, inspect the exact text it received, including title/body fields, length limits and query formatting. Test a change against a held-out query set, not just the failure that motivated it.

Second, inspect the assembled context. A selected chunk ID is not evidence that its supporting sentence survived deduplication, truncation or prompt packing. If the supporting text is present and the answer still contradicts it, investigate generation and citation behavior separately. Retrieval recall alone cannot certify the final answer.

Use stable labels and separate measurements: acceptable-evidence recall at each budget, answer support, abstention errors, latency and cost. For recall, define the denominator as the labeled acceptable evidence available to that authorized query; record incomplete labels. For questions with no valid corpus answer, evaluate false acceptance separately instead of dividing by zero. If you use an LLM judge, test its reliability too; swapped-order checks help expose one source of evaluation instability.

The decision rule is simple: change the earliest demonstrated failure, retain a simpler baseline, and keep the change only if the measured trade-off helps your intended readers or users. Hybrid retrieval and reranking are tools to evaluate, not automatic upgrades.