Understanding RAG, when AI needs your knowledge
What RAG really is, in plain words. When you need it, when not. The shortest RAG explainer on the web.
The problem
You ask ChatGPT a question whose answer is in your own handbook. ChatGPT doesn't know it, because it was trained on the world's knowledge, not your handbook. Now you have two options: paste the information again on every question, or build a solution that does it automatically. The second solution is called RAG. Retrieval-Augmented Generation.
What the words mean
"Retrieval" is just a fancy word for "searching". "Augmented" means "enriched". "Generation" means "producing the answer".
So: when you ask a question, the system first searches your own material, packs the matching text snippets automatically into the prompt, and the model answers your question with that extra information. You don't see any of that. You ask normally, the answer comes back fitting, just with your knowledge inside.
When do you really need RAG
You need RAG when three things are true:
- You have your own material that wasn't in the model's training (handbooks, customer protocols, internal processes, product specs, medical patient files, legal cases)
- The material is too big to copy into the prompt every time (more than a few pages)
- You want the answer to refer precisely to that material and not be hallucinated
If one of those isn't true, you don't need RAG. If your material fits into a chat context, just paste it in. If you don't have private data and standard info is enough, ChatGPT alone is fine.
How a RAG system is built
The skeleton always has the same four parts:
- Your documents sitting somewhere. PDFs, Word files, databases, doesn't matter.
- A splitter that cuts those documents into smaller pieces, called chunks. Usually a few paragraphs per chunk.
- A vector database that stores those chunks so they can be searched semantically (not "which chunks contain word X" but "which chunks mean roughly the same as my question").
- An application that takes your question, searches the vector DB, pulls the most relevant chunks, packs them with the question into the prompt, and produces the answer.
That's RAG. Full architecture in four sentences.
Why semantic search instead of keyword search
Because keyword search fails when your user phrases things differently from your document. If the handbook says "we use magic-link authentication" and the user asks "how do I log in without a password", a keyword search finds nothing because no word matches. A semantic search understands both texts mean the same thing and finds the right spot.
The magic happens via so-called embeddings. Long lists of numbers that represent a text block and that can be compared mathematically. If two texts mean similar things, their embeddings are mathematically close. That's all.
You don't have to implement that yourself. There are ready-made tools that handle the embedding plumbing for you. You feed in text, get vectors out, store them, search later.
Typical pitfalls
Wrong chunk size: Too small and the chunk has no context. Too big and too much irrelevance comes along. Sweet spot is usually 300 to 800 words per chunk.
Missing metadata: If you don't store which document a chunk came from, you can't cite sources. Always store at least document title + page + date.
No update strategy: What happens when a document changes? If you don't re-chunk and re-index, your RAG gives stale answers. An update mechanism is mandatory.
Too many irrelevant hits: If your vector DB returns thirty chunks for every question and most are unrelated, the model gets confused. Usually five to ten hits are enough. Quality over quantity.
RAG vs. long context
Newer models have context windows of one million tokens. Meaning you could theoretically paste a whole book instead of building RAG. When is that worth it?
For one-off use: paste the book and ask. Done.
For repeated use across many users and many questions: still RAG. Tokens cost money. With 1000 queries a day, each pasting the whole book, that gets very expensive. RAG only pulls the relevant snippets and saves the rest.
What RAG is NOT
RAG doesn't replace good training. If your model isn't medically trained, no amount of good RAG documents will produce sensible medical advice. RAG gives the model facts. It doesn't change what the model can do.
RAG also doesn't replace database queries. If your question is "how many customers did we win this week", you need SQL, not RAG. If your question is "what's in our welcome email", RAG fits.
A worked example, how a law firm uses RAG
Law firm has 2,000 PDF court rulings in the archive. Every lawyer regularly looks things up for their cases. The PDFs don't all fit into Claude's context, and manually searching the intranet takes long.
Solution: the firm runs all PDFs through a RAG system, once. Afterwards a lawyer asks in a Claude UI "was there a ruling on home-office work expenses before 2024?" and gets three relevant rulings back with citation + reference. Time saved: 30 minutes per question. Error rate drops because Claude can cite concrete rulings instead of inventing them.
That's RAG in practice.
What you can do now
If you want to build RAG yourself, there are ready-made toolkits: LlamaIndex, LangChain, Haystack for Python. Vercel AI SDK for JavaScript. All with tutorials for first steps.
If you want to use RAG without building it: ChatGPT has "Custom GPTs" with file upload, Claude has "Projects" with its own knowledge area, both do this automatically for small document sets. That's RAG for non-developers, packaged into the normal chat UI.
Where it goes from here
In Level 4 we show how you can move not only documents but also memory and knowledge graphs portably across multiple AI tools. That's a step beyond RAG: not just "search documents" but "structured knowledge that's stored and linked".
And in Level 6 you build the backend for it yourself, as your own MCP server with vector-DB integration. Then you don't just know what RAG is, you have your own RAG service that fits your project.