Measure the Retriever Before You Add It
SPLADE + dense fusion is the retrieval upgrade everyone recommends this year, and the case for it is real: learned-sparse recovers the exact identifiers dense embeddings quietly drop. Before adding a transformer dependency to our retrieval stack, we measured it against the hybrid we already run. On the cases it should win, SPLADE⊕dense beat our hybrid by +0.005 (inside the noise) and as a graph-completion signal it tied plain tf-idf at AP 0.01. So we didn't ship it; we kept it as a benchmark arm for the corpus where it genuinely wins. This post is about measuring the upgrade, on your own corpus, before you take on the dependency.
The upgrade worth checking
Every few weeks a retrieval technique gets a wave of "you should be using this." The current one is well earned: teams running RAG over technical documentation find that dense embeddings quietly drop exact identifiers (a part number like A-1187-CX, a model code, an acronym that appears verbatim in the query) and that fusing a SPLADE learned-sparse retriever with the dense one recovers them. That is a real, repeatedly-reported result, and it came up in an r/RAG thread pointed at our own retrieval work.
The reasonable-sounding next step is to add it. The disciplined step is to measure what it buys on your data against what you already run, because "better in general" and "better for you" are different claims, and the second one costs a dependency.
It helps to place SPLADE in the terms we already use. Our retrieval stack has three content arms:
- B1: BM25, classic lexical / sparse.
- B2: dense, a bge embedding.
- B3: hybrid, B1 and B2 fused with reciprocal-rank fusion.
SPLADE is a learned-sparse retriever: a transformer predicts a weighted bag of vocabulary terms per document, expanding a query term toward its relatives while staying a sparse, invertible index. In our terms it is a stronger B1, and "SPLADE + dense" is a better B3. So the experiment is clean: add a SPLADE arm and a SPLADE⊕dense fusion arm to the existing ladder, freeze everything else, and measure.
Does it beat the hybrid we already have?
Our test set is real definition-change dependents from the Federal Register, scored as recall at a matched materialization budget:
| arm | recall |
|---|---|
| B1 · BM25 | 0.763 |
| B2 · dense | 0.747 |
| B3 · hybrid (RRF) | 0.761 |
| SPLADE | 0.752 |
| SPLADE ⊕ dense | 0.766 |
SPLADE⊕dense lands at 0.766 against the hybrid's 0.761 (+0.005, inside the noise) and plain SPLADE actually came in below BM25. Look closer and it was slightly worse than the hybrid on the high-overlap dependents it should have found most easily.
None of that is a knock on SPLADE. It is a fact about the corpus. Learned-sparse expansion earns its keep when the query carries exact identifiers that dense embeddings smear: part numbers, SKUs, error codes. Regulatory prose has almost none of those; it is sentences. There are no A-1187-CX tokens here for learned-sparse weighting to rescue, so the extra machinery has nothing to grip, and BM25+dense RRF already captures what lexical signal there is. The upgrade is real; the conditions that make it pay off are simply not present in this data.
Does a stronger embedder help anywhere else in the pipeline?
There is a subtler place a better embedder could have helped. We compare text in retrieval and when we reconstruct dependency graphs from messy documents, and those graphs always have missing edges. We showed previously that recovering a missing edge from graph topology fails. Could a stronger similarity signal recover it instead? We dropped 30% of the real edges in one corpus and asked each predictor to bring them back:
| predictor | edge AP |
|---|---|
| tf-idf | 0.014 |
| dense (bge) | 0.011 |
| SPLADE | 0.010 |
All three are near-random, and SPLADE (the strongest of them) does not beat plain tf-idf; none moves downstream recall off the observed floor. A dependency edge exists for a reason that has nothing to do with the two endpoints reading alike, so no amount of embedding quality invents it. The lever that closes that gap is disciplined extraction (reading the stated reference out of the text); a better similarity score does not close it. A stronger content model was the wrong tool in both places we tried it.
Where SPLADE actually wins, and the decision
Say it plainly, because the point is fairness: the r/RAG result is real, and it's the right tool for its corpus. On identifier-dense technical documentation, exact-token recall is exactly SPLADE's strength and dense-only pipelines genuinely lose part numbers and codes. Our tests don't exercise that strength (legal and regulatory prose), so they neither confirm nor refute it. They only show it is the wrong lever for this data.
So the decision wrote itself, and we wrote it down: do not add SPLADE + dense on this evidence. Shipping it would mean carrying a transformer / SPLADE-model dependency to pay for a benefit our corpora don't show. What we did keep is the measuring stick: SPLADE stays a ready, self-contained arm in our benchmark harness, so the day an identifier-dense corpus is in scope, the experiment is one command away and the adoption call is made on numbers.
The lesson
The instinct when a technique is trending is to adopt it; the discipline is to A/B it against what you already run, on your own corpus, and only then decide. A retriever being genuinely better in general does not make it better for your data, and every dependency you add is paid for whether or not it earns its keep. Two numbers settled this one (a +0.005 that was noise, and an AP that tied tf-idf) and both took an afternoon to get.
Measure the upgrade before you add it. And when a genuinely good technique doesn't move your numbers, keep the arm, so you'll know the moment your data changes and it does.
The retrieval primitives, source-available
Our BM25 / dense / hybrid arms and the citation-graph closure module ship in the source-available runtime: compose them under a token budget, and benchmark your own arms against them. ReDevOps RAG on GitHub →
#RAG #SPLADE #HybridSearch #InformationRetrieval #DenseRetrieval #LearnedSparse #Embeddings #BM25 #Benchmarks #Evaluation #MLEngineering #AIEngineering #ContextEngineering #RetrievalAugmentedGeneration #OpenSource