Searching by meaning: vector databases and retrieval · 08 / 08

From retrieval to RAG

We can retrieve the right passages, by meaning, by the letter, by their fusion. But retrieving is not answering. How do we wire our engine to a language model so it composes a grounded answer, without making things up?

In the previous chapter, we married the two worlds of search: meaning, captured by vectors, and the letter, pinned down by BM25, fused by rank to stay robust. So we can now retrieve the right passages from a store, whatever the shape of the query. That is an achievement. But it is also a frustration that has been building for eight chapters: when you ask your store a real question, you do not want a pile of documents to read yourself. You want an answer, written, and grounded on those documents. Retrieving is not answering. This final chapter takes that step.

The brilliant candidate who is not allowed to open their notes

Picture an exceptional candidate at a major oral exam. They have vast general knowledge, they speak with ease, they string arguments together. But they sit the exam empty-handed: not a single note allowed. Ask them a general-knowledge question and they shine. Ask them the exact amount of your invoice number 8831, and they are lost: that information was never part of what they learned. A language model on its own is that candidate.

It suffers from three very specific gaps:

Its knowledge is frozen. It was trained on data cut off at a certain date. Everything after that escapes it. It does not know what it could not have read.
It ignores your private documents. Your personal store, your invoices, your notes, your contracts: none of that was in its training. Your particular is invisible to it.
It fills the gaps. This is the most dangerous one. Rather than admit ignorance, a model spontaneously produces a plausible answer, stated with the same confidence as a correct one. This is hallucination : a structural flaw of training, which pushes the model to always say something, even when it should say it does not know.

A foundation model is therefore wonderful on the general and blind on your particular. The problem is not that it is dim, it is brilliant. The problem is that its notes have been confiscated. The whole idea of this chapter is the opposite gesture: give the notes back, at the right moment.

The RAG idea: slip it the right notes before it answers

What if, just before letting the model answer, we went and fetched from your store the passages that speak to the question, and laid them under its eyes? It would no longer answer from memory, but by reading. This is exactly the architecture called retrieval-augmented generation , or RAG. It breaks into three stages, and the first is already perfectly familiar to you.

\underbrace{\mathcal{C} = \text{top-}k\big(\text{hybrid}(q)\big)}_{\text{1. retrieve}} \qquad \underbrace{p = \text{prompt}(q, \mathcal{C})}_{\text{2. assemble}} \qquad \underbrace{r = \text{LLM}(p)}_{\text{3. generate}}

Here is how the chain reads. First, for a question $q$ , we build the context $\mathcal{C}$ by keeping the $k$ best passages returned by the previous chapter’s hybrid search. Then we assemble a prompt $p$ that contains the question and those passages. Finally, we hand that prompt to the model, which produces the answer $r$ . The remarkable point is the first term: the retrieval in RAG is nothing other than the search we built throughout the course. RAG does not replace what we learned, it crowns it. Embeddings, nearest neighbors, HNSW, durable index, hybrid fusion: all of it was the foundation that makes this final stage possible.

Ground the answer, or stay silent

Handing the passages to the model is not enough: we must also require that it use them, and only them. This is the notion of grounding : the answer must rest on the provided context, not on the model’s internal memory, and it must cite its sources so they can be checked. A reader must be able to trace every claim back to the passage that supports it.

Then comes the question that makes all the difference: what happens when the context does not contain the answer? The model’s temptation, as we saw, is to fill the void. The right behavior is the opposite: say so. Answering “I cannot find this information in your documents” is a good answer, not a failure. It is the direct remedy to hallucination: we no longer ask the model to know, we ask it to read, to cite, and to admit when the source is missing. The honest refusal is a feature, not a defect.

This shift has a heavy consequence, which we must face squarely: the quality of the answer is now capped by the quality of the retrieval. If the right passage is not retrieved, or if it does not make it into the context, the most grounded model in the world can only refuse. That is exactly what the component below puts into your hands.

Wire the pipeline yourself

The component below runs the full pipeline on a small personal store. Pick a query, then switch between the two worlds: the model alone, which answers from memory, and the model plus RAG, which first reads the retrieved passages. In RAG mode, a lever sets the context budget $k$ , that is the number of passages we allow into the model’s view. Watch what happens when you squeeze it.

Query

Model

Context budget k

Retrieved passages

#1in context
Invoice FR-2024-8831, consulting services, amount 4,200 euros.
#2over budget
Strategy board meeting of March 12, minutes.
#3over budget
Invoice FR-2024-2207, annual hosting.
#4over budget
Invoice FR-2023-1180, software license.
#5over budget
Expense report, trip to Lyon.
#6over budget
Staff internal regulations.

Answer

grounded

Invoice 8831 corresponds to consulting services for an amount of 4,200 euros. [1]

Cited sources: d1

What you observe

The right passage has entered the context. The model grounds its answer on it and cites its source. The answer is correct because retrieval did its job.

Three questions to ask yourself while playing:

On the query “amount of the consulting invoice”, stay in LLM alone mode. What does the model answer, and does that figure come from your store? Then switch to LLM plus RAG with a budget $k = 1$ : what happens, and why is the right passage not there yet?
Still on this query, raise $k$ to $2$ . The answer changes status. Which passage just entered the context, at what rank was it, and why can the model finally answer?
Switch to the query “due date of invoice 2207”. Compare the model alone and RAG. Which one invents, which one refuses? In what way is the refusal here the best possible answer?

Three horizons to go further

The pipeline we just wired is RAG in its simplest form. Three refinements separate it from real systems. We formalize them here, because these are the ones you will meet as soon as you leave the toy corpus.

Split before you index: chunking

So far, one passage equaled one document. But a real document runs forty pages. Indexing it as a single block would be absurd: its answer would drown the useful paragraph in an ocean of text, and it would never fit in the model’s context. So we cut it into short pieces before indexing. This is chunking .

The size remains to be chosen. Too large, and each passage mixes several ideas and wastes the budget; too small, and we cut an idea in two, so each piece becomes incomprehensible on its own. So we make the passages overlap: each piece repeats the end of the previous one, so that a sentence cut by a boundary is found whole in at least one passage. If a document has length $L$ (in tokens), and we choose passages of size $s$ with overlap $o$ , the number of passages produced is about:

n_{\text{passages}} \approx \left\lceil \frac{L - s}{s - o} \right\rceil + 1

This formula reads: each new passage advances by $s - o$ tokens (the size, minus what we repeat from the previous one), the step of a sliding window. We count how many steps it takes to cover the whole length, plus the initial passage. The larger the overlap $o$ , the smaller the step, so the more passages we produce: the safety against cuts is paid in index volume.

The context window: a budget you cannot exceed

We handled the lever $k$ without really naming it. What bounds it is a hard constraint of the model: its context window , the maximum amount of text, measured in tokens, that it can read at once. We therefore cannot pour the whole store into the prompt, only the $k$ best passages. It is precisely this ceiling that makes the retrieval ranking decisive: if the right passage comes out at rank $12$ and the budget only holds $8$ , it is lost, exactly as in the component at $k = 1$ .

From this comes a very useful refinement in practice: reranking . We first let the fast retrieval (BM25, vectors) return a large batch of candidates, say fifty. Then a more expensive but finer model, often a cross-encoder that reads the query and each passage together, reorders them so the very best take the top spots, the ones that will fit in the window. Two stages: a first, fast one that rakes wide, a second, precise one that sorts tight.

The model that searches on its own: agentic retrieval

In everything above, the pipeline is a one-way trip: one query, one search, one answer. But some questions cannot be resolved in a single search. “Compare the amount of my consulting invoice with my annual training budget” requires fetching two distinct things, then confronting them. A one-way trip fails.

The idea that lifts this limit is to give the model search not as a fixed step, but as a tool it wields itself, as many times as it deems useful. This is the ReAct loop (for Reasoning and Acting, Yao et al. 2022), which alternates three gestures until the model judges it can conclude:

\text{Reason} \;\rightarrow\; \text{Act (search)} \;\rightarrow\; \text{Observe} \;\rightarrow\; \text{Reason} \;\rightarrow\; \cdots

The model reasons about what it is missing, decides what to search for, observes the passages returned, then starts again based on what it just learned. Retrieval becomes dynamic, driven by the model itself. This is the bridge to what we now call agents: a model that no longer merely answers, but goes and fetches, in a loop, what it needs in order to answer.

Exercises

Solution to exercise 1: counting the passages of a chunking

A document has $L = 1{,}000$ tokens. We split it into passages of size $s = 200$ tokens, with an overlap $o = 50$ tokens. How many passages do we get?

Step 1. The step of the window. Each new passage advances by the size minus the overlap.

s - o = 200 - 50 = 150

Step 2. The number of steps after the first passage. We apply the formula, starting with the fraction.

\frac{L - s}{s - o} = \frac{1{,}000 - 200}{150} = \frac{800}{150} \approx 5.33

Step 3. We round up. A partial window still counts, to cover the end of the document.

\lceil 5.33 \rceil = 6

Step 4. We add the initial passage.

n_{\text{passages}} = 6 + 1 = 7

Result. The document produces $7$ passages. Note that with zero overlap ( $o = 0$ ), the step would be $200$ and we would drop to $5$ passages: the overlap, which guards against cuts, mechanically increases the index volume.

Solution to exercise 2: why k=1 refuses and k=2 answers

On the component’s query “amount of the consulting invoice”, the model refuses at $k = 1$ but answers at $k = 2$ . We explain this switch through the ranks.

Step 1. The hybrid ranking. Retrieval returns, in order, passage d0 (“strategy board meeting”) at rank $1$ , then passage d1 (the consulting invoice, which carries the amount) at rank $2$ . The distractor d0 wins rank $1$ because it is the closest by meaning to the word “consulting”, while d1, the actual invoice, comes just behind.

Step 2. The budget k=1. We keep only one passage in the context: d0. But d0 is about a meeting, not an amount. No passage in the context carries the answer.

\text{context} = \{\, d0 \,\} \quad\Rightarrow\quad \text{no source} \quad\Rightarrow\quad \text{honest refusal}

Step 3. The budget k=2. We keep the first two passages: d0 and d1. This time, d1, the invoice, enters the context.

\text{context} = \{\, d0,\ d1 \,\} \quad\Rightarrow\quad d1 \text{ carries the amount} \quad\Rightarrow\quad \text{grounded answer, source } [2]

Result. The right passage was there all along, correctly retrieved at rank $2$ . Only the context budget decided whether it entered the model’s view. That is the whole lesson of the chapter: a correct retrieval is not enough, the rank of the right passage must also fit in the window.

In one sentence

RAG turns a search into an answer: it retrieves the best passages (via the course’s hybrid search), slips them into the model’s bounded context, and asks it to compose an answer grounded on them while citing its sources, or to refuse honestly when the source is missing, so that the quality of the answer stays capped by that of the retrieval.

Quiz

1. Why does a language model alone hallucinate a private invoice amount?
2. In the RAG pipeline, what does the context given to the model refer to?
3. What does the fact that RAG refuses at k=1 but answers at k=2 on the same query reveal?

The closing word

Here we are at the end of the thread. Think back over the path: we started from an almost philosophical idea, meaning as a position in space, and built, step after step, everything needed to make it a real engine. Represent a text as a vector. Measure proximity, and run into the curse of dimensionality. Bypass exhaustive comparison with HNSW, then place that choice within the landscape of approximate indexes. Give ourselves a differential oracle to verify without fooling ourselves. Make the index durable, able to survive a restart. Marry it with the lexical side so as to miss neither a synonym nor an exact number. And, today, wire all of it to a language model to move from the retrieved document to the written answer.

Each chapter answered a limit left by the previous one, and this last one closes the loop: RAG only exists because everything else exists first. That is the beauty of this chain, and it is what I wanted to share by writing it. The sovereign engine I am building will keep evolving, it will become the memory of a personal assistant that reads my own documents to answer me. But the essential is now in your hands: you know what hides behind these buzzwords, not the marketing version, the version that compiles. Thank you for walking this path.

Sources

Lewis, P. et al. (2020). “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” NeurIPS 2020. arXiv:2005.11401
Karpukhin, V. et al. (2020). “Dense Passage Retrieval for Open-Domain Question Answering.” EMNLP 2020. arXiv:2004.04906
Yao, S. et al. (2022). “ReAct: Synergizing Reasoning and Acting in Language Models.” ICLR 2023. arXiv:2210.03629

Going further

Ji, Z. et al. (2023). “Survey of Hallucination in Natural Language Generation.” ACM Computing Surveys 55(12), 1-38. DOI 10.1145/3571730
Gao, Y. et al. (2023). “Retrieval-Augmented Generation for Large Language Models: A Survey.” arXiv:2312.10997