What is RAG? Source-grounded AI explained
RAG — Retrieval-Augmented Generation — means the AI looks something up before it answers. Instead of recalling from training, the system searches your documents, puts the most relevant passages in front of the model, and asks it to answer from those, with citations back to the source. It fixes the training cutoff, the "it's never seen my files" problem, and a large share of confident invention. It does not fix all of it.
The problem, stated properly
A language model is, in effect, a very good compression of a lot of text it read once. That gives it fluency and breadth and three specific weaknesses: it stops knowing things after its training cutoff, it has never seen anything private, and when it doesn't know something it produces a plausible answer anyway, because producing plausible text is the whole job.
You can't retrain a model every time you write a document. So instead you change where the answer comes from.
The four steps
- Index. Your documents are split into passages — a few paragraphs each — and every passage is converted into a numeric representation that captures its meaning. This happens once, up front.
- Retrieve. Your question is converted the same way, and the system finds the passages closest to it in meaning. Not keyword matching: "what did we agree about late delivery" can find a paragraph that says "shipment delays" and never uses your words.
- Augment. Those passages are placed into the model's context along with your question and an instruction to answer from the supplied material.
- Generate and cite. The model writes the answer, and because the system knows which passages it was given, it can point each claim back at its source.
That's it. The clever part is step 2; the trustworthy part is step 4.
Why "just paste it into the chat" isn't the same thing
Attaching a document to a conversation puts the whole thing in the model's context for that conversation. That works fine for one paper. It stops working when there are ninety of them, when you'll ask questions next month too, or when the relevant paragraph is on page 340 and the surrounding 339 pages are noise competing for the model's attention.
RAG indexes once and retrieves per question. You pay the setup cost a single time and then ask across the whole collection indefinitely. We wrote about the trade-off in more depth in GPT-5 vs Gemini — large-context models have narrowed the gap for single documents, but not for collections you return to.
Where RAG still fails
Worth being precise, because "grounded" gets used as if it meant "correct":
- Retrieval misses. If the right passage isn't retrieved, the model answers from what it was given — confidently and wrongly. Questions that need synthesis across many scattered passages ("what changed between the 2023 and 2026 policies?") are the classic hard case.
- The answer drifts past the evidence. A model handed a relevant passage can still add a plausible sentence the passage doesn't support. This is what citations catch, and why an uncited claim inside a cited answer deserves suspicion.
- The source is wrong. Grounding means faithful to your documents, not faithful to reality. Feed it an outdated contract and you get outdated answers, correctly cited.
- Chunking artefacts. A table split across two passages, or a clause whose meaning depends on a definition forty pages earlier, retrieves badly. Structure that spans chunks is the weak point of the whole approach.
The general mechanism behind the second one is worth understanding on its own — see why AI makes things up.
When it's worth the setup
| Situation | Use |
|---|---|
| One document, one question | Paste it into a chat |
| One long document you'll query repeatedly | Either, leaning RAG |
| Dozens of documents on one topic | RAG |
| Answers someone else will rely on | RAG — for the citations, not the retrieval |
| Mixed media: PDFs, video, audio, web pages | RAG, if it handles all of them |
| General knowledge questions | Neither — just ask a model |
What this looks like as a product
Neurobase is our version of this: you give a Neuron sources — PDFs, text files, video, YouTube, web pages, images, audio — and it indexes them into a focused assistant that answers with citations back to the original material. Answers can be tested in the Lab before you rely on them, and several Neurons can be chained into a workflow with operations like Summarize, Compare, Extract, and Translate.
If you want the concrete version rather than the concept, start with turning a folder of PDFs into an assistant.
FAQ
What is RAG?
Retrieval-Augmented Generation: the system searches your documents first, puts the relevant passages in the model's context, and asks it to answer from those — with citations.
What problem does it solve?
Training cutoffs, private documents the model never saw, and confident invention. Citations let you check rather than trust.
Is it the same as uploading a file to a chatbot?
No. Attaching a file works for one document in one conversation. RAG indexes a collection once and retrieves per question, across many conversations.
Does RAG stop AI making things up?
It reduces it substantially, not entirely. Retrieval can miss, answers can drift past the evidence, and sources can be wrong. Citations make those failures checkable.