The Same Text, Sliced Every Which Way
RAG systems make a choice they'd rather not: how big should a chunk be? Small chunks match precisely but lose context; big chunks keep context but bury the answer. Here's a structural trick that lets a system refuse to choose — slice at several sizes at once, and let the same content reappear in a way you can detect exactly and for free.
Every retrieval-augmented generation system makes a decision it would rather not make: how big should a chunk of text be?
Cut the documents into small chunks and the system matches a question precisely, but it hands the language model a sentence with no surrounding context. Cut them into big chunks and the context comes back, but the one relevant sentence is now buried in paragraphs of unrelated text. Every RAG system in production today has picked a number and lived with the trade-off. I’ve been working on a way to refuse the choice.
A quick reminder of what RAG does
If you’ve read about Rizzo, the assistant on this site, the mechanics will be familiar. A RAG system splits your content into chunks, turns each chunk into a vector — a list of numbers that captures its meaning — and stores those vectors. When you ask a question, your question becomes a vector too, the system finds the chunks whose vectors sit closest to it, and it hands those chunks to a language model as context. The model answers from what it was given rather than from memory.
The chunk size is set when the index is built, long before anyone asks a question. That’s the decision I want to defer.
Slice it every which way
Here’s the trick. Instead of committing to one chunk size, slice the same document at several fixed sizes at once — and start every cut from the very beginning of the document, so all the sizes share a common grid.
Do that, and something tidy happens: the same content keeps showing up at different sizes in the search results. A relevant passage gets retrieved as a small chunk and as the medium chunk that surrounds it and as the large chunk that surrounds that. You didn’t retrieve one view of the answer. You retrieved several, nested inside each other.
The containment forest
Because every size starts from the same origin, this nesting is exact, not approximate. A large chunk contains the smaller chunks that fall within its span, byte for byte. The large contains the medium; the medium contains the small. Stack those relationships up across a whole set of retrieved chunks and you get a structure worth naming — a containment forest.
The best part is that you can recover the whole forest for free. Take a short hash fingerprint of each chunk — an ordinary SHA-256 — and matching fingerprints tell you exactly which chunk sits inside which. No model, no embedding comparison, no judgment call. Just string matching that either lines up or doesn’t.
Upward substitution — a free safety guarantee
That exactness buys a genuinely useful operation. Suppose a small chunk comes back as highly relevant to a question, but it’s tight — it nails the match and loses the context around it. Because you know, exactly, which larger chunk contains it, you can swap the small chunk for its parent and be certain you haven’t dropped a single relevant byte. The parent still contains the small chunk’s text, verbatim, plus the surrounding context.
Call it trading up. It gives a RAG system a concrete lever it didn’t have before: retrieve small for precision, then, when there’s room in the token budget, trade up to the roomier parent for context — with a guarantee, not a hope, that nothing relevant was lost in the swap.
But doesn’t cosine similarity already do this?
It’s a fair question. Retrieval systems already compute cosine similarity between vectors, and cosine similarity does carry a signal about which chunks contain which. But it can’t do the job cleanly, for two reasons.
First, it’s a noisy proxy. To use it as a yes-or-no test for containment you’d need a threshold, and there’s no threshold you can know in advance that gets it right.
Second, and more decisively, cosine similarity is symmetric: it gives the same score for a pair of chunks in either direction. That means it fundamentally cannot tell you which chunk contains which — and direction is the whole point, because trading up only works if you know which one is the parent.
The hash markers have neither problem. They’re exact, and they point in a direction. The honest framing is this: cosine is a noisy signal, and the markers are ground truth.
Why this matters
The value here is that the chunk-size decision stops being a guess made at indexing time and becomes an informed choice made at question time. A system can retrieve at several sizes, look at how much room it actually has in its token budget for a given question, and decide then and there whether to keep the tight match or trade up to its parent. It’s the same content either way — the forest just gives you a safe, exact way to pick your resolution.
It’s open source
The proof-of-concept is public:
github.com/mrissover/slicing-proof — Java 21, Spring Boot, MIT-licensed. It depends on two open-source services, Ollama and Qdrant, that both run locally, and it pins its dependencies and its random seed so the results reproduce cleanly. The whole pipeline runs on a laptop.
Where this is going
This is early, active research — a result I’m still building on, not a finished story. If the idea is interesting to you, the code is the best place to poke at it, and I’ll keep a running account of the work on the site’s Research page as it develops.