RAG in Practice: What It Actually Is, and When You Don't Need It

Midnight, the front desk, a guest walks in with a foreign passport. The receptionist - third shift of their life - opens the internal assistant and asks: "how do I register a foreign guest." The assistant answers in detail, confidently, and incorrectly - it found a fragment of the regulations about Russian citizens in the knowledge base and neatly paraphrased it.

This isn't a made-up scene. I debugged exactly this failure in an assistant for hotel automation: it answers staff questions from a base of internal instructions and regulations. Under the hood it uses RAG - the setup people often describe as the thing that "gets rid of hallucinations." It does not get rid of them. It relocates them: out of the model's head and into your search.

By the end of this article you'll understand what RAG is, what parts it's assembled from, and - more importantly - how to tell a task where you genuinely need it from a task where it will only give you three new ways to get things wrong.

RAG in plain terms: search + generation

A language model knows what was in its training data. Your internal regulations were not in there. Ask it about your check-in procedure and it will either honestly say it doesn't know, or invent a plausible answer. The second happens more often.

There are essentially two options. The first is to fine-tune the model on your documents: expensive, slow, and every time a regulation changes you have to repeat the whole procedure. The second is to leave the model alone entirely and, before asking it a question, find a suitable piece of text in your own documents and put it directly into the prompt: "here's an excerpt from the instructions, answer strictly based on it." That is RAG (retrieval-augmented generation - "generation grounded in what was found").

So RAG is two separate stages, and it's useful to keep them apart in your head from the very beginning:

• Retrieval. Given the user's question, pull several of the most relevant text fragments out of the base.
• Generation. Hand the model the question along with those fragments and ask it to formulate an answer.

From this follows the key insight for a beginner, the one that saves weeks: the model in a RAG system doesn't "know" anything about your knowledge base. It reads only what was placed into the prompt right now. If retrieval brought back garbage, the model will confidently and elegantly paraphrase garbage. So when the assistant answers incorrectly, the first question is never "which model are we on," it's "what did retrieval find."

flowchart LR
  Q["Вопрос"] --> S["Поиск по базе"]
  S --> CH["Найденные чанки"]
  CH --> PR["Промпт: вопрос + чанки"]
  PR --> M["Модель"]
  M --> A["Ответ"]
  KB[("База знаний")] --- S
  KB -.->|"модель ее<br/>не видит"| M
  S -.->|"нашел мусор"| A2["Модель уверенно<br/>перескажет мусор"]

What it's actually made of

Now the details - using that same hotel knowledge base as the example.

Chunks. You can't search over whole documents: a twenty-page instruction won't fit into a prompt, and most of it isn't relevant to the question anyway. So texts get cut into fragments - these are called chunks. Our base has about fifteen hundred of them. A chunk is both the unit of search and the unit the model will ultimately read.

Embeddings. To search by meaning rather than by letters, each chunk is turned into a set of numbers - a vector. A separate vectorizer model does this; ours is GigaEmbeddings. The point of the conversion is that texts similar in content get similar sets of numbers, even if they're written in different words. The user's question is run through the same model, and you look for the chunks whose numbers came out closest.

Vector database. The place where these vectors live and where you search them quickly. In our case, ChromaDB. Essentially it's a specialized store that can answer the request "give me the k vectors closest to this one."

Note an important detail that beginners almost always conflate: vectorization and generation are different models and different stages. GigaEmbeddings turns texts into numbers at the retrieval stage. GigaChat writes the answer at the generation stage. They are not interchangeable, and they can be swapped independently: changing the generator model won't improve retrieval, and changing the vectorizer requires recomputing the entire base from scratch.

Similarity threshold. Vector search always returns a result - even if there's nothing on the topic in the base, you'll get back the "least dissimilar" thing. So you need a threshold: how close does something have to be before a fragment counts as found at all. In our setup, fragments scoring below 0.11 are discarded entirely, and only those above 0.3 reach the model. These numbers were tuned empirically, for a specific base and a specific vectorizer. Don't carry them over to your own system: yours will be different.

Hybrid search. Initially retrieval was purely semantic - vectors only. It missed on specialized terminology: a narrow term from an industry regulation gets smeared out in the vector and lost among texts that are similar in general meaning. We had to add plain keyword matching and merge the results. It sounds boring and old-fashioned, but that's exactly what fixes an entire class of errors.

The first honest problem: similar texts aren't the same as correct ones

Vector search answers the question "what here is similar?" But the user needs an answer to "what here is correct?" These are not the same thing, and all the practical work on RAG lives in that gap. Two failures from real operation.

Failure one: zero fragments for an entire section. On the topic of registering foreign guests, retrieval returned nothing at all. Not "poor results" - zero: a document-domain filter was cutting off the needed fragments before they were ever considered. The assistant didn't stay silent, though - it answered in generalities, and from the outside it looked like "the model is a bit weak." The fix wasn't the model, it was a fallback: if nothing is found on a topic, a backup search kicks in without the restricting filter. And separately - a regular check that no section of the base returns emptiness.

Failure two: a chunk without context lies. All fragments in one section were being treated as relating to foreigners, even though some of them were about Russian citizens. The reason is simple: during chunking, a fragment loses what was obvious from the heading and the neighboring paragraph. Inside the piece of text itself there may not be a single word about which category of guests it applies to. Retrieval honestly finds something similar, the model honestly paraphrases what was found, and the answer is wrong. We had to tag fragments with categories during chunking and add an explanatory preamble to them - a short text that gives the chunk back its lost context.

These two stories add up to the conclusion I consider the main practical takeaway: answer quality more often comes down not to the model, but to how the documents are chunked and labeled. The temptation to solve the problem by swapping models is enormous - it's one line in a config. But the fix almost always lies at the data preparation stage.

flowchart TB
  T["Задача"] --> C1{"Ответ содержится<br/>в связном тексте?"}
  C1 -->|"нет: цифры, таблицы"| N1["Запрос к данным,<br/>а не RAG"]
  C1 -->|"да"| C2{"Текста много?"}
  C2 -->|"нет"| N2["Положить текст<br/>прямо в промпт"]
  C2 -->|"да"| C3{"Неизвестно заранее,<br/>где лежит нужное?"}
  C3 -->|"нет"| N3["Обычный поиск<br/>по разделу"]
  C3 -->|"да"| YES["Это случай для RAG"]

When you don't need RAG

Now for the part all of this was leading up to. RAG is a tool for one specific situation: the answer is contained in connected prose, there's a lot of that prose, and you don't know in advance where exactly the needed part sits. If even one of those conditions doesn't hold, you're building an elaborate construction where something simple would have done.

• Tables. This is the most common and most expensive mistake. When the input is a table rather than text, RAG works badly. Questions about tables are "how many," "who has more," "show me everything for this period." Such questions are solved with filters and groupings - small operations over data, not search by meaning. Vector search will find you "similar rows" - but similar rows don't add up to a correct total.
• Exact reference data. If an entity has a code, a SKU, or an identifier, look it up directly. An exact match is faster, more reliable, and doesn't need thresholds tuned.
• Short regulations. If the entire set of rules is a few pages, you can put all of it into the model's prompt. No vector database, no chunking, no thresholds. Fewer parts means fewer places to break.
• One document per question. When the user already knows which instruction holds the answer, you're solving a navigation problem, not a search problem.

The rule I've arrived at: it's worth setting up RAG once you've already tried to get by without it and hit a wall. Not the other way around.