Reranking for search and RAG: cross-encoders, ColBERT and LLM rerankers

Key points

• Two-stage retrieval splits the work: a cheap first stage finds candidates, an expensive reranker orders them.
• A bi-encoder embeds query and document separately; a cross-encoder reads them together, which is more accurate and cannot be precomputed.
• ColBERT keeps one vector per token and scores with MaxSim, a middle ground between the two.
• LLM rerankers can rank listwise with a sliding window; they are strong and the slowest option.
• A reranker cannot recover a passage the first stage missed, so measure first-stage recall at the candidate depth you rerank.
• Reranker scores are not probabilities; set any cutoff on labeled examples from your own data.

Two-stage retrieval: why rerank at all

Search systems face a speed and quality trade-off. The models that judge relevance best are too slow to run over a whole collection, and the methods fast enough to search millions of documents, BM25 and vector search over an ANN index, judge relevance more coarsely. Two-stage retrieval uses both: the first stage returns a candidate list, say the top 100, and a reranker scores only those candidates and reorders them.

Nogueira and Cho describe exactly this pipeline in their monoBERT paper: a standard method such as BM25 retrieves a large number of candidates, a thousand in their example, a more expensive model reranks them, and only the top ten or fifty go on to the next step. In a RAG system the next step is the prompt. A better ordering means the passage that answers the question is among the few the model reads, which helps against hallucinated answers and keeps the prompt short.

The first stage is often hybrid search with BM25 and vectors, merged with reciprocal rank fusion. Fusion combines ranks; a reranker reads the text. They do different jobs and are often used together.

Bi-encoders and cross-encoders

A bi-encoder encodes the query and each document separately into one vector each, and relevance is the similarity of the two vectors. Document vectors can be computed once and indexed, which is what makes vector search fast. The cost is that the query and the document never see each other inside the model: everything the document says is squeezed into one vector before anyone knows the question. See text embeddings explained for how these models are trained.

A cross-encoder takes the query and one document as a single input and outputs a relevance score. Every layer of the transformer can attend across both texts, so it notices that a passage answers the question rather than just sharing its topic. In monoBERT, the query is fed as BERT's sentence A and the passage as sentence B, the query is truncated to 64 tokens and the pair to 512, and the vector of the special CLS token goes through a single-layer network that outputs the probability that the passage is relevant. The model is fine-tuned with cross-entropy loss on relevant and non-relevant passages from the BM25 top 1,000. The authors report it reached the top of the MS MARCO passage ranking leaderboard, 27% better in MRR@10, relative, than the previous state of the art.

The price is that nothing can be precomputed: every query and document pair is a full forward pass. The Sentence Transformers documentation puts it concretely: clustering 10,000 sentences with a cross-encoder means scoring about 50 million pairs, roughly 65 hours, while a bi-encoder embeds them in about 5 seconds. Its recommended pattern is the two-stage one: retrieve the top 100 with a bi-encoder, then rerank those 100 hits with a cross-encoder.

Late interaction: ColBERT and ColBERTv2

ColBERT (Khattab and Zaharia, SIGIR 2020) sits between the two. Like a bi-encoder it encodes query and document separately, so document representations can be computed offline. Unlike a bi-encoder it keeps one small vector per token instead of pooling them into one. The score is a sum of maximum similarities, which the paper calls MaxSim: for each query token vector, find its highest cosine similarity with any document token vector, then add those maxima up across the query tokens.

This keeps much of the fine-grained matching of a cross-encoder at a fraction of the compute. The authors report ColBERT runs two orders of magnitude faster and needs four orders of magnitude fewer FLOPs per query than comparable BERT-based rankers, with competitive effectiveness, and that it can rerank a candidate list or retrieve end-to-end from a vector index. Queries are padded with BERT's mask tokens to a fixed length, which the paper calls query augmentation and finds essential to its effectiveness.

The catch is storage: one vector per token instead of one per document multiplies the index size. ColBERTv2 (Santhanam and colleagues, NAACL 2022) addresses this with residual compression and improves quality with denoised supervision; the authors report a 6 to 10 times smaller space footprint than earlier late interaction models.

LLM rerankers: pointwise, pairwise and listwise

A general-purpose LLM can rerank too, by prompting rather than by fine-tuning a ranking model. There are three shapes.

• Pointwise: ask about one passage at a time, for example whether it answers the query, and use the probability of "Yes" as the score. One call per candidate, easy to run in parallel.
• Pairwise: ask which of two passages is more relevant and aggregate the comparisons. Many calls, so it suits short lists.
• Listwise: show the model several numbered passages and ask for the ranked order. The model compares passages against each other directly.

RankGPT (Sun and colleagues, EMNLP 2023) is the well-known listwise method. Each passage gets an identifier, and the model outputs a permutation, an ordering such as passage 2, then 3, then 1, instead of scores. Because a prompt holds only so many passages, it uses a sliding window: rank the last w passages, slide the window toward the top by s, rank again, and repeat, so strong passages bubble up from the bottom. In the paper's main experiments it reranks the BM25 top 100 with a window of 20 and a step of 10. The authors report that properly instructed LLMs deliver competitive, even superior results to supervised methods on popular benchmarks, and that a 440M model distilled from ChatGPT's rankings outperformed a 3B supervised model on BEIR.

LLM reranking costs the most per query, adds the latency of several sequential calls, and needs the output parsed and checked: a permutation can skip or repeat an identifier. Treat it like any other model output, with the validation described in LLM structured outputs. The distillation result points to a common pattern: use a large model to label rankings offline and train a small reranker to serve them.

Comparing reranker types

Hosted rerank APIs, Cohere Rerank for example, wrap a cross-encoder-style model behind one call that takes a query and a list of documents and returns each document's index with a relevance score. Open models, such as the MS MARCO cross-encoders loaded through Sentence Transformers, run on your own hardware.

Candidate depth, top-n and the latency budget

Two numbers shape a reranking step: how many candidates go in, the depth, and how many come out, top-n. Depth sets the ceiling. If the right passage is at position 150 and you rerank 100, the reranker never sees it. Measure the first stage's recall at several depths on labeled queries, and choose the depth where recall levels off. Top-n is what the next step can use: the number of passages that fit the prompt, or the results a page shows.

Reranker latency grows with depth times passage length. A cross-encoder scores pairs in batches on a GPU, so batch all candidates into as few forward passes as memory allows, truncate passages to the model's maximum input length, and cap depth by your latency budget rather than by habit. Chunk sizes matter here too: shorter chunks rerank faster, which is one more reason to read chunking strategies for RAG.

def rerank(query, candidates, score, top_n=5, min_score=None):
    # candidates: (doc_id, text) pairs from the first stage, best first.
    # score(query, text) -> float stands in for the reranking model;
    # a real cross-encoder scores the whole list in batched forward passes.
    scored = [(doc_id, score(query, text)) for doc_id, text in candidates]
    scored.sort(key=lambda r: r[1], reverse=True)
    if min_score is not None:
        scored = [r for r in scored if r[1] >= min_score]
    return scored[:top_n]

def overlap_score(query, text):
    # Toy scorer for the demo: share of query words found in the text.
    q = set(query.lower().split())
    return round(len(q & set(text.lower().split())) / len(q), 2)

candidates = [
    ("d2", "how to reset a password"),
    ("d3", "request a refund within the policy window"),
    ("d1", "refund policy for annual plans"),
]
print(rerank("annual refund policy", candidates, overlap_score, top_n=2))
# [('d1', 1.0), ('d3', 0.67)]

• Keep the first-stage rank as a tie-breaker, and log both ranks so you can see what the reranker changed.
• Have a fallback: if the reranker times out, return the first-stage order rather than an error.
• Rerank after deduplication and access-control filtering, so the budget is not spent on passages the user cannot see.
• Long documents may be truncated or split by the reranker. Cohere's documentation, for example, says its v5.0 model truncates documents to max_tokens_per_doc, 4,096 by default, and recommends chunking documents yourself for more control.

Score calibration and thresholds

Reranker scores are good at ordering candidates for one query and poor as absolute measures. Some cross-encoders output raw logits, others pass them through a sigmoid into 0 to 1; either way, a score is not the probability that a passage is relevant, and the spread of scores shifts from query to query. Cohere's documentation says its relevance scores are normalized to the range 0 to 1, that a score is query dependent, that a document scoring 0.9 is not twice as relevant as one scoring 0.45, and that the most important output is the rank.

A cutoff is still useful, for example to tell a RAG system to say it does not know when nothing is relevant. Set it from data. Cohere's documentation describes a method: pick 30 to 50 representative queries, pair each with a document you judge borderline relevant, score the pairs, and use the average as a reference threshold. More generally, label query and passage pairs from your own traffic and choose the cutoff from the precision and recall you need on both sides of it. If you need scores that behave like probabilities, fit a calibration curve, such as Platt scaling or isotonic regression, on those labeled pairs.

Recheck the threshold whenever the reranker model or version changes. A new model can shift every score while leaving the ordering much the same, which silently changes what a fixed cutoff lets through.

Evaluating rerankers: nDCG@10 and BEIR

Rerankers are usually compared with nDCG@10, which rewards putting relevant documents high in the top ten and handles graded relevance, and MRR, the mean of one over the rank of the first relevant result. BEIR (Thakur and colleagues, NeurIPS 2021 Datasets and Benchmarks) is the standard zero-shot test: 18 public datasets across different retrieval tasks and domains, scored on models that were not trained on them. Its findings are a useful map of the field: BM25 is a robust baseline, and reranking and late-interaction models achieved the best average zero-shot performance, at a high computational cost.

Public scores tell you which models to try. Your own queries tell you which to ship. Run the first stage, then the reranker, on a labeled set and compare the metrics before and after, query by query, not just on average: a reranker that helps most queries can still hurt one kind badly. How to evaluate RAG covers building the labeled set.

def reciprocal_rank(ranked_ids, relevant):
    for rank, doc_id in enumerate(ranked_ids, start=1):
        if doc_id in relevant:
            return 1 / rank
    return 0.0

def mrr(runs, qrels):
    # runs: query -> ranked doc ids; qrels: query -> set of relevant ids
    return sum(reciprocal_rank(runs[q], qrels[q]) for q in qrels) / len(qrels)

qrels = {"q1": {"d3"}, "q2": {"d7"}, "q3": {"d9"}}
first = {"q1": ["d1", "d2", "d3"], "q2": ["d7", "d5", "d6"], "q3": ["d4", "d8", "d2"]}
reranked = {"q1": ["d3", "d1", "d2"], "q2": ["d5", "d7", "d6"], "q3": ["d8", "d4", "d2"]}

print(round(mrr(first, qrels), 3), round(mrr(reranked, qrels), 3))  # 0.444 0.5
for q in qrels:  # per query: q1 improves, q2 gets worse, q3 was never retrieved
    before = reciprocal_rank(first[q], qrels[q])
    after = reciprocal_rank(reranked[q], qrels[q])
    print(q, round(before, 2), "->", round(after, 2))

The example shows the three outcomes to look for. The average rose, one query improved, one got worse, and one could not improve at all because its relevant document was never in the candidate list. That last kind is a first-stage problem, and no reranker fixes it.

When reranking does not help

• The first stage misses the answer. A reranker only reorders what it is given; fix retrieval recall first.
• The next step reads every candidate anyway. If all 20 candidates go into the prompt, reordering them matters much less than which 20 they are.
• The queries are exact lookups: an order number, an error code, a product SKU. BM25 or a database query already puts the match first.
• The domain is far from the reranker's training data. Many open rerankers are trained on MS MARCO web questions; on code, legal text or another language, test before trusting them, and consider fine-tuning on your own labeled pairs.
• The relevant part of a long document sits past the reranker's input limit, so it is truncated before the model sees it.
• The latency or cost budget leaves no room. A smaller cross-encoder, a shallower depth, or reranking only when the first-stage scores are close can fit where a large model does not; see how to reduce LLM costs.

Treat the reranker as a component with its own evaluation, depth, budget and fallback, and keep it only if your labeled queries show it earns its place.

If you want search or a RAG feature built into your product, see our AI agent development service: the code lives in your GitHub from day one, and AI-assisted work is reviewed by an engineer.