Decision Models as Rerankers
with Doug Turnbull & Trey Grainger
I gave this as a guest session in Doug Turnbull and Trey Grainger’s AI-Powered Search course on November 18th, 2026, in the week on modern RAG and automated ranking. The course is private, so there’s no public recording. This is the written edition: what I said, the slides and notebooks, and the questions the room asked.
The short version. A decision model asked “does this document answer the query?” reranks about as well as the purpose-built rerankers, for a fraction of the price. The bigger change is that you write the question, so the reranker can finally be tuned to your domain and to what each query means.
Why rerankers exist
Most search pipelines I see now have two tiers. The first is algorithmic: a BM25 query and a dense vector query, merged with reciprocal rank fusion. That step takes milliseconds or less, and adding more queries or a deeper top K is a cheap way to get a longer list.
The second tier is a model that reads the query and each document together and puts the list in order. That’s where latency jumps an order of magnitude, to hundreds of milliseconds, and where you start paying per document. So you don’t hand it everything. In the animation above, only the top 8 go to the reranker. Document 5 is relevant and sits below the cut, so nothing downstream can bring it back. The reranker owns the order; the first stage and the cut own what’s in the list.
Reranking became essential with RAG. The first RAG systems people built in 2023 were often semantic search only, with no reranker, on models that hallucinated easily. A vector index has no notion of “no match”: ask for K and you get the K nearest, however far away they are. Hand a shaky model a plausible-looking miss and it goes off the rails. Rerankers fixed that by throwing out results that were similar but not relevant, and false positives are also why almost every vector database now ships keyword search and a hybrid of the two.
That gap at the top of the list became a business. Cohere, Voyage, Jina and Mixedbread all sell a hosted reranker, mostly closed weights, priced by the token. Both Voyage (MongoDB) and Jina (Elastic) have been acquired. None of them let you tune what “relevant” means.
How a decision model fits ranking
A decision model answers a typed question with a probability. Jev calls its true-or-false question a Noul. Ask “does this document answer the query?” and you get back a probability that the answer is true. The rerank is just a sort on those probabilities.
In that example the randomised trial, the best evidence in the list, comes fifth from BM25 because it says “D3” and “hip” instead of repeating the query’s words. Jev gives it 0.96 and it moves to second. The passages at the bottom get under 0.1, which is the other thing probabilities buy you: a fixed threshold drops the off-topic results without anyone picking a top K.
There are three ways to ask:
- Pair. One call per document, run in parallel. It has the best tail latency per call, but a query waits for the slowest of 30 calls, and that’s 30 requests against your rate limit for every search. (Learning-to-rank people would call this pointwise.)
- Batch. All the documents in one call, one yes-or-no question each. Same quality as pair for Jev (0.501 against 0.502), and it’s what you want when you’re returning a ranked list.
- Choice. All the documents in one call and a single question: which one is best? The probabilities share one unit across the list, so it ranks but can’t prune. Use it when you only need one answer, as in RAG.
Which shape works best depends on the model. Clef-flash scores 0.498 one document at a time and 0.283 given the whole list, which is worse than the BM25 order it was meant to improve. OpenAI does best as a chooser. Benchmark the shape, not just the model.
What the benchmark says
Every system reranks the same BM25 top 30 on three BEIR corpora (SciFact, NFCorpus and FiQA) with a generic prompt, scored on nDCG@10. The full results are on the reranker comparison (free account).
- Perplexity tops it.
pplx-deciderv1.3, asked to choose, scores 0.511 for $0.20 per 1,000 queries, level with Mixedbread v3.1 listwise, the best purpose-built reranker in the test. The 4B flash model, the one you can run in the free Colab notebook, scores 0.502 for $0.10. - Jev is close behind, at 0.502 pair and 0.501 batch, beside Voyage (0.504) and above Cohere (0.486).
- The models from the bigger labs don’t do as well here, and cost more. That doesn’t make them bad decision models. I just wouldn’t pick them for reranking.
These gaps are small, a hundredth of nDCG or less, and BEIR is a well-worn dataset that models can overfit to. Swapping rerankers in and out is cheap, so evaluate on your own data.
Latency is a few hundred milliseconds at the median for every decision model. The tail is where they differ: Jev batch had a p95 of about 1.4 seconds on 30 documents. I ran these over my home network on developer tiers, some of them in a model’s busy first days, so do your own latency tests before you trust mine.
Customize it
Everything above used one generic prompt, to keep the comparison fair. That’s also the least interesting way to use a decision model. A hosted reranker has one fixed idea of relevant. A decision model reads your instructions on every call, so you can tell it what relevant means for your application: “pick the one document that answers this support question”, or “find the result most likely to convert”. You can tune the prompt and A/B test it like anything else, and different parts of your application can rerank with different instructions.
Be honest about where that pays. On BEIR, a corpus-tailored prompt did no better than the generic one, because BEIR’s labels are topical relevance and the generic prompt already asks about that. Customization pays when your relevance isn’t topical: freshness, authority, audience, version.
Where I think this goes for search teams is classifying the query’s intent first and routing it to a reranking rubric written for that intent. A support question gets “find the knowledge-base article that answers it”; a product search gets a commercial rubric. I haven’t built this yet, and the notebooks don’t do it. It’s a good place to experiment. Doug’s bag of decisions goes further, asking an LLM to write a set of yes-or-no criteria for each query and scoring every document against all of them.
Are rerankers dead?
Not yet. You have to ask “is it dead?” about every search technology before it goes general-purpose.
The purpose-built rerankers still win on tail latency and on running in your own environment, and I don’t see anyone stopping tuning open-weight rerankers. What’s already happened is price pressure. A month before this talk, the best reranking in my test cost $1.42 per 1,000 queries. Then a general-purpose decision model matched it for $0.20, and Mixedbread cut its own price to an estimated $0.27. If you were planning to launch a reranker startup, that window was two or three years ago.
If you’re running a search system that works, don’t rip it out. Build the new pipeline beside it and prove it’s better first. But if I were setting one up today, I’d pick one decision model family and use it everywhere: reranking, classifying documents, query intent. One key, one eval harness, one kind of number. I already use Jev as a document classifier for a client, where it replaced LLM classification for orders of magnitude less money.
Questions from the room
The questions are paraphrased.
Were these benchmarks run with a basic prompt? Yes, a short, generic “does this document answer the query” prompt, on purpose. It’s the fair way to compare models, and it doesn’t use the thing that makes decision models interesting. I’d expect a very long prompt to add some latency, but not much.
Do decision models replace classic learning to rank? The answer in the room, which I agree with: they replace the off-the-shelf rerankers, not LTR. A cross-encoder only ever sees the query text and the document text; you can’t tell it your domain is K–12 education. A decision model takes a rubric on every call, which is like tuning it per call. But LTR models learn from signals a model reading text can’t see: popularity, recency, location, clicks. For news, jobs and e-commerce, where behaviour data is rich, a well-trained LTR model is hard to beat. They work together, too: a decision model can generate features for an LTR model.
Is the relevance signal underneath any different from a cross-encoder? Probably not much. Both are transformers reading the query and document together, and I’d guess they start from similar weights. The difference is the interface. A decision model is confined to a set of answers, which keeps it fast and makes it a structured-data tool. If you need something generative, like deciding which tool to call next in an agentic search loop, you still want a regular LLM.
Is the rubric the same for every query? In my benchmark, yes: one generic instruction for every query. But you could replace that instruction with an LLM’s output for each query, which is the idea behind per-query rubrics.
Is the score a probability, and should you threshold it? The providers call it a probability. I always end up needing a threshold. Small or sparse datasets are where it bites: right now I’m working with construction data, where a brand-new project might have three invoices, and something always comes back for a query even when nothing is relevant. I tune the threshold per application. The deck uses 0.1 to drop off-topic passages; a much higher bar makes sense when you only want results you’re confident answer the query. Someone in the room made a good follow-on point: treat the scores as relative to each other, because the same documents in a different order can come back with slightly different probabilities.
Did you fine-tune, or run an optimization loop on the prompt? No. These are straight reranking runs scored on BEIR. I think a loop that tunes the prompt (a GEPA-style optimizer, say) would go a long way, especially for a domain prompt. BEIR is fairly saturated, so it’ll pay more on your own data.
Does listwise reranking suffer from position bias? I’m sure it’s there: models handed a list tend to favour what’s already near the top. I haven’t tested for it. Pairwise sidesteps it, because each call sees one document. It’s an architecture choice, and a reason to design your benchmark to shuffle the list.
Slides and notebooks
- The slides
- Notebooks: a hosted walkthrough that takes an API key for Jev, Perplexity, Clef or OpenAI, and a free Colab notebook that runs Perplexity’s open 4B decision model on a T4 with no keys (open in Colab)
hev-rerank, the Python wrapper:rerank()with pair, batch or choice, across models- The posts behind the numbers: Hev meets Jev, Jev has company, OpenAI makes four and Reranking just got cheaper