RAG

What RAG means

RAG stands for retrieval augmented generation, meaning retrieval and augmentation before the text is produced. The process has three steps. First the user's question is converted into a sequence of numbers, a so called embedding. Then the system looks for the passages in your own sources that sit closest to that question. Those passages are finally handed to the language model along with the question, and the model phrases the answer from them.

The decisive point is that the model learns nothing. It receives the relevant passages for that single run, much as an examiner hands a candidate a page of text before asking a question. That is why a detail can be changed by changing the document rather than by a new training run.

Why that suits business knowledge better

A training changes the model permanently and is needed again with every correction. It needs many examples, causes cost and is hard to follow. RAG separates knowledge and language: the knowledge sits in the source, the language comes from the model. For prices, processes, policies and product data that is the right split, because details change and their origin should stay traceable.

A second advantage is verifiability. Because the system knows which passages it used, it can hand them over as well. An answer with a source can be checked, an answer without one cannot.

How a knowledge base is maintained

• Split it up: long documents are divided into passages because a passage is the unit of retrieval. Passages that are too large water down the search, too small ones lose the context.
• Enrich them: headings, product groups and validity dates belong with the passage, otherwise the search finds text without context.
• Update them: outdated passages have to disappear. A knowledge base holding two truths returns both, and therefore contradictions.
• Respect rights: personnel files and confidential contracts do not belong in the same base as public product data.

Where it fails

Most disappointments originate in retrieval, not in the model. If the knowledge base does not contain the matching passage, the answer can only be wrong. If too much is retrieved, the model loses the thread and mixes things up. And if the search works on matching words rather than meaning, the system misses the passage that says the same thing in other words. A test with real questions from daily work shows these weaknesses faster than any technical check.

Frequent questions

• Do I need a vector database? For small volumes a file with entries is enough. From a few thousand passages a suitable database becomes worthwhile.
• Does RAG understand my internal terminology? Only if it appears in the sources or is accounted for in the search.
• Is RAG alone enough? No, it needs a maintained source. Answer quality follows the quality of the knowledge base.