Evaluating Hybrid Search In and Out of Domain with Vespa

A measured look at when vector search wins, when it fails, and why hybrid search stays reliable across both

I built a faceted product search engine on Vespa, then measured six ranking methods against human ratings of which results are relevant. In-domain, the embedding methods win and plain hybrid looks unnecessary. Then I changed the data, and the result flipped completely. This article walks through the build, the measurement, and what it all means for choosing a ranker. The full runnable notebook is linked at the end.

Before we build anything, here is the whole finding in a single chart:

Figure 1. Six methods on two datasets: vector sits near the top in-domain but drops to last out-of-domain.

Both panels use the same methods, the same engine, and the same metric. The only thing I changed between them was the data, and that alone was enough to send vector search from near the top all the way down to last. The rest of the article is how I got there.

The one-engine idea

The reason to do this on Vespa is that keyword search, vector search, structured filtering, and grouping all run in one query over one set of documents. The usual alternative is to run several separate systems (a vector database for embeddings, a keyword engine for text, a metadata store for the facet counts) and write glue code to merge their results, which is also where they drift out of sync.

Figure 2. Several systems glued together and kept in sync, versus Vespa answering everything from one query.

A product is a single document holding three kinds of data: BM25-indexed text, structured attributes like price and category, and an embedding tensor. The rest of this section builds that, step by step. First lets start with our imports:

Step 1: define the product schema

One document holds the three data shapes. Text fields get a BM25 index. The structured fields become attributes, which is what makes them filterable and groupable for facets. The embedding is produced inside Vespa from the title and description; the "passage: " prefix matches how e5 was trained, and embed e5 names the embedder component defined in Step 2. There is also a doc_tokens field holding the document's token ids, which the cross-encoder in Step 3 needs.

Step 2: add the in-engine models

Two components run inside Vespa. The e5 embedder turns text into a 384-dimension vector at feed time and at query time, so there is no external embedding service to call. The tokenizer turns text into token ids for the cross-encoder. The cross-encoder itself is a 23 MB quantized ONNX model that ships inside the application package, so it is downloaded locally once.

Step 3: attach the six rank profiles

Ranking is configured on the schema, over the same fields. I compare six of them here. bm25 and semantic are the two pure methods. hybrid multiplies the scores (the naive approach). hybrid_rrf fuses by rank position. hybrid_norm normalizes both scores to a common range before adding. hybrid_xe reranks the top candidates with the cross-encoder, reading query and document tokens together.

Step 4: deploy

Bundle the schema and components into a package and push it to a Vespa Cloud trial. The first deploy is the slow one, since Vespa downloads the e5 model and the package carries the cross-encoder. The endpoint is live in a couple of minutes.

Step 5: load and prepare the catalog

Take the first 10,000 US products from the Amazon ESCI parquet. Why 10,000 and not 1,000? Because the first 1,000 rows hold only 6 phone cases and no screen protectors at all, while the first 10,000 hold 38 and 12, so the demo queries actually have something to find. ESCI has no price or category field, so I make them up from the product id. The numbers are fake, but the faceting works the same whether the data is real or not.

Step 6: feed

Send plain documents. Vespa embeds and tokenizes each one as it arrives, so there is nothing to precompute. The feed is bound by model inference on the content node, about 18 minutes for 10,000 documents.

With the catalog fed, the rest of the work is querying and measuring.

What a vector database can't return

A pure vector database answers one question, "what are the nearest k documents," and returns a ranked list. It has no operator for "how many matches per category" or "what is the price distribution." Those are the sidebar of every e-commerce search page, and on a vector store you build them with a second query against a separate metadata system, then hope the two stay in sync.

Step 7: query for hits and facets together. On Vespa, one request retrieves, ranks, counts by category, and buckets by price, all over the same match set. A small helper pulls the grouping block out of the response, which arrives alongside the hits:

For "phone case" over the 10,000-product catalog, that one request returns ranked hits plus the category counts and a price histogram:

Figure 3. Price histogram for "phone case", computed in the same request as the ranked hits.

The category facets add up to 821 documents and the price buckets add up to the same 821, because both count the same matched set. Filters work the same way: when a shopper clicks "under $100," the price limit applies to both the keyword and the vector search, and the vector search only looks at products under $100 rather than fetching fifty neighbors and throwing most of them away. None of this needs a second database, and the counts can never disagree with the results.

Why hybrid exists: two ways search breaks

Now for the ranking side of things. Search "smartphone protector" with keywords alone and you get a telescope and three copies of a USB charger, because the real screen protectors share no words with the query. Search the same thing with vectors and you get screen protectors immediately, no shared words required. That is one way search breaks, when the words the shopper types do not match the words in the product.

Vectors have the opposite weakness. Search "12v 500ma", the rating of a small power adapter. Keyword search is decisive: it scores the exact 12V 500mA adapters far above everything else, because the spec matches the text directly. Vector search finds the same adapters, but ranks them only a hair above an unrelated 500W power bank, because it has no firm notion of "exact" and reads "power", "500", and "mA" as roughly similar. So keyword locks onto a precise spec while vector blurs it. The two methods break in opposite directions, keyword search when the words differ, vector search when the query is an exact spec or code. Real traffic has both kinds, so you want both kinds of search. The rest of the article checks whether the data backs that up.

In-domain: the cross-encoder wins, naive fusion is a trap

Step 8: score the methods with NDCG. ESCI comes with human ratings for each result (Exact, Substitute, Complement, Irrelevant), which I turn into graded scores and measure with NDCG@10 over each method's top-10. Every method is the same query with a different ranking profile, so switching between them is a one-line change:

Running that over 500 queries gives:

Figure 4. NDCG@10 over 500 ESCI queries (in-domain). The cross-encoder leads, naive fusion barely beats keyword search.

Three things stand out. The first is that multiplying the two scores barely beats keyword search (+0.017). The reason is a scale mismatch: vector closeness sits around 0.6 while BM25 ranges past 18, so the keyword score dominates the product and drags in noise. The hybrid_norm profile, which normalizes both signals before adding them, recovers about +0.06. So the way you combine the two signals turns out to matter more than whether you combine them at all.

The second is that the cross-encoder is the strongest method at 0.649. It reads the query and document together through one transformer instead of comparing two independent vectors, and it wins decisively on exact-term queries. It also costs about 1.4 seconds per query, which is why it only reranks the top two dozen results.

And the third is that in-domain, every method that uses the embedding beats plain keyword search. On product data, which is close to what a general embedding model trains on, the dense signal is just good. If I stopped here, the honest conclusion would be "use vectors or a reranker and skip the lexical arm." So I changed the data.

Are these gaps real?

Step 9: test the gaps for significance. Before reading too much into small differences, I ran a paired bootstrap over the 500 per-query scores: resample the queries 10,000 times, recompute each mean difference, and report a 95% confidence interval and p-value.

Every one of these gaps is significant. The cross-encoder genuinely tops the table, ahead of even the dense model (p=0.038). The dense model comes a clear second, significantly above the best fusion (p=0.015). And proper fusion beats both naive fusion and keyword search. So the in-domain order is real and not noise: the cross-encoder, then vector, then proper hybrid, with naive fusion barely above keyword search. The one thing the numbers do not support is "vectors win" on their own, since the cross-encoder stays reliably above them. (An earlier run on 100 queries put vector and the cross-encoder in a tie; going to 500 queries broke the tie toward the cross-encoder. Both the scale and the query mix move the numbers, which is why I ran the bootstrap.) And that is the in-domain result. Out of domain, it changes completely.

Out-of-domain: the result flips

Step 10: repeat out of domain. I deployed a second app with the same shape of schema and ran the identical experiment on NFCorpus, a standard medical-retrieval benchmark of 3,633 abstracts and 323 queries, full of clinical vocabulary the embedding model never trained on:

Here is what came back:

Figure 5. NDCG@10 over 323 NFCorpus queries (out-of-domain). Vector is last; hybrid ties the cross-encoder.

Vector dropped from near the top to dead last. BM25 now beats it, because exact term matching still works on medical terms regardless of training. Every method that keeps a lexical signal stays ahead of pure vector, and the two at the top are normalized hybrid and the cross-encoder, tied at 0.346. That tie is what surprised me: in domain the cross-encoder led the field by 0.022, and out of domain that lead is gone, with a cheap hybrid matching a reranker that costs far more to run. This lines up with Vespa's own published BEIR work, where dense models underperform BM25 out of domain and fusion closes the gap. I reached a similar conclusion independently, on a different dataset.

What this actually tells you

The in-domain table says use vectors. The out-of-domain table says use BM25. Both are traps, because you rarely control how far your real traffic drifts from your model's training data. A new product line, a more technical vertical, or a term the model never saw all push you further out of domain, and vector quality drops with every step.

Two methods stay strong in both columns: the cross-encoder and normalized hybrid. The difference between them is cost. In domain the cross-encoder is worth the cost, leading the field in return for a transformer pass on every result it reranks. Out of domain that lead disappears and normalized hybrid matches it for a fraction of the compute. So the method I would reach for first is the cheap one that never collapses: normalized hybrid sits near the top in both domains, while pure vector falls from second place to last. The same lesson shows up query by query: no single method wins every type, with vectors handling the worded-differently queries and the cross-encoder handling the exact-term ones. So you do not pick one method forever; you run the one that fits the query. Because every method here is just a rank profile in the same app, switching is a config change, not a rebuild.

These numbers may look close, but NDCG differences usually are. On benchmarks like BEIR, the gap between a decent system and a top one is often just a few hundredths. The in-domain bootstrap showed which of those small gaps are real. Out of domain the swing is bigger and you do not need a test to see it: the best hybrid scores about 15% higher than vector, and vector drops from near the top to last.

A few honest limits. Hybrid does not always win; in-domain it does not, and the numbers say so. None of the rank profiles are tuned, they are standard forms, so this is not a result I fit the weights to get. And two datasets show a trend, not the whole picture; a wider set of benchmarks would tell you how the curve really bends between them.

Where I would take this next

The cross-encoder here is a general model trained on web pages, not on product data, so it still makes domain mistakes. The natural next step is to train a ranker on ESCI's own human ratings rather than rely on off-the-shelf models, which is what Vespa's learning-to-rank series does on this dataset (and a useful reality check, since that work found a trained model only narrowly improves on a strong dense baseline). A third dataset further out of domain would fill in the curve between the two points I measured here.

The lesson I take from this is narrow but useful. Run the same methods on familiar and unfamiliar data, and the ranking you should trust changes. Doing that comparison was easy only because all six methods are rank profiles over the same documents and the facets come from the same query, so it was a handful of config changes rather than a new system. The runnable notebook, with every number and chart above reproduced from the code, is here: companion notebook. It runs end to end on a Vespa Cloud free trial.

Resources

Improving Zero-Shot Ranking with Vespa Hybrid Search

Hybrid search tutorial

Improving Product Search with Learning to Rank