Vector database

What a vector database does

A vector database stores text passages as sequences of numbers, so called vectors, and finds the passages most similar in content for a query. It therefore searches by meaning rather than by words. That is the difference from classical full text search: a query about an invoice also finds a passage talking about a payment record, because both sit in the same corner of meaning.

In a RAG setup it is the retrieval part. The language model phrases the answer, the database supplies the content. If the search fails or returns the wrong thing, the answer can only be wrong.

How searching by meaning works

The process has three steps. On ingest, every passage is converted into a vector, an embedding. The vectors are stored together with a reference to the original passage. On a query, the question is also converted into a vector, and the database looks for the vectors closest to it. Closeness here means similarity of meaning.

Quality therefore depends on two places: the quality of the embedding model and the splitting of the content. A good model with badly split passages returns bad hits.

When a database is worth it

• From a few thousand passages: below that a file with entries searched in memory is enough. The difference shows only with volume.
• With constant change: when content is added continuously, a solution that can swap individual entries is needed.
• With several users: as soon as several queries run at once, a service of its own makes sense.
• With access rights: when users may only reach their own content, filters at document level are needed.

Which mistakes lower the hit rate

The first mistake is the wrong passage size. Passages that are too large contain much that is irrelevant, which dilutes the vector. Ones that are too small lose context, so a sentence is found without its reference. A middle size with an overlap between neighbouring passages works best.

The second mistake is missing metadata. Without details on product group, language and validity, the search finds text without context or returns outdated passages. The third mistake is missing maintenance. A vector database holding two contradictory passages returns both and therefore an answer that contradicts itself.

The fourth mistake is searching by words when numbers are asked for. A vector understands closeness in meaning but not the difference between two similar model numbers. For exact details an additional search for character strings is needed, combined with the result of the vector search.

Frequent questions

• Do I need a separate database? Not to start with. A file with entries and a calculation in memory is enough.
• What does ingesting cost? Every passage has to be converted into a vector once, and that is a call to the embedding service.
• Can one database serve several languages? Yes, when the embedding model works multilingually.