RAG — retrieval-augmented generation — means the system searches your own documents first and hands the model those passages before it answers. The model replies with your facts instead of whatever it absorbed in training. For a sales or support team, that is the difference between an assistant that invents a refund policy and one that quotes yours.
The name comes from a 2020 paper by Patrick Lewis and colleagues, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, which combined a pre-trained generation model — parametric memory, knowledge baked into weights — with a searchable index of documents, described as non-parametric memory. That split is still the whole idea: what the model knows, versus what it can look up.
If you want the basics of models and tools first, read what an AI agent is. This article is about making retrieval actually work in a revenue team.
In short
- RAG is for prose knowledge. Exact facts about records belong in a query, not a vector search.
- Retrieval quality is a corpus problem long before it is a model problem.
- Permissions must apply at retrieval time, or you have built a leak.
- Measure retrieval and answering separately, against a fixed set of real questions.
- Customer-facing answers keep a human in the loop until the evidence says otherwise.
How it works, in four steps
AWS defines RAG as optimising a model's output so it "references an authoritative knowledge base outside of its training data sources before generating a response," and describes the mechanics as: prepare external data as numerical representations in a vector database; retrieve the most relevant documents for a query; augment the prompt with what was retrieved; and keep the external data current through real-time or batch updates (AWS).
The same page names why teams reach for it: models hallucinate when they lack an answer, their training data is static and goes out of date, and the same terminology means different things in different sources. It also names the benefits that matter commercially — it is cheaper than retraining a model, it lets you control and change the information sources, and it allows the system to present answers with source attribution so a user can verify them.
That last one is the one to insist on. An answer without a citation is an opinion.
The decision that comes first
Before anyone builds a vector database, answer one question: where does the true answer live?
| The question | Where the answer lives | Right technique |
|---|---|---|
| "How many open opportunities does Acme have?" | A table | Query the CRM through its API |
| "What's our refund policy for annual plans?" | A document | RAG |
| "Which of our customers churned last quarter?" | A table | Query or report |
| "How did we answer this security questionnaire last time?" | Past documents | RAG |
| "What tone should replies use?" | Nowhere — it's behaviour | System prompt, or fine-tuning at scale |
| "What did the customer say on the call in March?" | A transcript | RAG over transcripts |
Semantic search is approximate by design. That is a feature when you are looking for the passage that talks about cancellation windows, and a bug when you need a count that must be exact. Teams that skip this decision end up with a chatbot that is confidently off by three on every number — see CRM architecture for why the record system should stay the arbiter of record facts.
Four options, not two:
- Paste the context. If the relevant document is small and always the same, just include it in the prompt. No infrastructure required. Start here more often than you'd think.
- Tool-call a query. Exact facts from systems of record.
- RAG. Large or changing prose knowledge.
- Fine-tune. Shape format, tone and behaviour — not facts. If the vocabulary here is new, LLMs explained for operators covers the underlying mechanics.
Most useful sales assistants combine 2 and 3: retrieve the policy, query the account.
Where RAG earns its place in revenue work
- Objection and question handling. A rep asks how to respond to "you're more expensive than X" and gets the current positioning with a link to the source, not a 2023 battlecard someone found in Drive.
- Policy and pricing questions. Discount authority, contract terms, cancellation, data residency. High-frequency, high-consequence, and entirely documented somewhere.
- Security questionnaires and RFPs. Retrieval over past approved answers is one of the highest-return uses in the category, because the corpus is already curated and the output is reviewed anyway.
- Call preparation. Retrieve prior notes, transcripts and tickets for this account, then summarise. Read-only and internal, which makes it a good first build.
- Support drafting. Draft a reply grounded in the help centre, with citations, for an agent to approve.
Notice the pattern: the best first projects are internal, read-only and reviewed. That is also how you earn the right to make something customer-facing later.
The quality chain
Answer quality is a chain, and it snaps at the weakest link. Debug in this order.
1. Corpus
Most "RAG doesn't work" complaints are corpus complaints. If three documents describe the discount policy differently, retrieval will find one of them and the model will answer it confidently. Curation beats clever retrieval.
The MitHub knowledge audit — run this before you build anything:
| Question | Why it decides the build |
|---|---|
| Who owns this document? | No owner means no updates, and no one to ask when it's wrong. |
| Is there exactly one canonical answer per topic? | Duplicates and drafts are the main source of confident wrong answers. |
| How often does it change? | Weekly-changing content needs a refresh path, not a one-off upload. |
| Who is allowed to read it? | This becomes a retrieval filter, not a policy note. |
| What happens when the answer is wrong? | Defines whether a human approves the output or only audits it. |
If more than a third of your documents have no owner, the first project is not RAG. It is a cleanup.
2. Chunking and context
Documents are split into chunks so retrieval can return small, relevant pieces. Anthropic's write-up on contextual retrieval identifies the flaw plainly: individual chunks "lack sufficient context," using the example of a chunk that states revenue growth without naming the company or the quarter (Anthropic).
Their fix is to prepend a short, chunk-specific explanation before embedding and indexing. The reported results, on their evaluation: contextual embeddings reduced the top-20-chunk retrieval failure rate by 35%, from 5.7% to 3.7%; combining with BM25 keyword search reduced it by 49%, to 2.9%; and adding reranking reduced it by 67%, to 1.9% (Anthropic).
Three practical takeaways, regardless of your stack:
- Chunks need self-contained context. A passage that makes no sense alone will not be retrieved well, and will not be understood well if it is.
- Hybrid beats pure semantic. Keyword matching still wins on product codes, error codes, contract clause numbers and names.
- Reranking is worth the extra step when precision matters more than latency.
3. Retrieval and permissions
Retrieval must filter by what the person asking is allowed to see. This is not a later phase. A system that retrieves from every document in the company and then politely declines to mention the sensitive one has still put that content into the model's context, and often into a log.
Design rule: apply access control on the retrieval query, not on the generated answer.
4. Generation, citations and abstention
Three requirements for any answer the business relies on:
- Cite sources with links, so the reader can check. AWS names source attribution as a way for users to verify claims (AWS).
- Allow "I don't know." A system that must always answer will always answer, including when retrieval returned nothing relevant. Explicitly instruct abstention and make it a measured outcome, not a failure.
- Distinguish quoted from inferred. "The policy says X" and "based on X, probably Y" are different claims and should look different.
Evaluating it properly
You cannot evaluate a knowledge system by trying it a few times. Build a golden set: 30 to 50 real questions people actually asked, each with a known correct answer and the document that contains it. Collect them from support tickets and internal chat, not imagination.
Then measure two things separately, because they have different fixes:
| Metric | What it asks | If it's bad, fix |
|---|---|---|
| Retrieval hit rate | Was the right document in the retrieved set at all? | Corpus, chunking, hybrid search, reranking |
| Answer correctness | Given the right passage, was the answer right? | Prompt, model, output format |
| Citation accuracy | Do the cited sources actually support the claim? | Prompt, post-check |
| Abstention rate | How often does it correctly say it doesn't know? | Instructions, retrieval threshold |
| Freshness lag | Days between a document changing and the index reflecting it | Ingestion pipeline |
Re-run the golden set on every change — new documents, new prompt, new model, new chunking. Without it, you are guessing, and every "improvement" is a coin flip.
The risks, named
Confident wrong answers. Retrieval reduces invention; it does not eliminate it. If the wrong passage is retrieved, the model will answer from the wrong passage, fluently.
Stale and contradictory sources. The most dangerous document in your corpus is last year's pricing page, because it is well written and completely wrong.
Prompt injection through retrieved content. OWASP describes indirect prompt injection, where an LLM reads external content such as a website or file whose content "alters the behavior of the model in unintended or unexpected ways," and recommends segregating and clearly denoting untrusted content, applying least privilege to the model's access, and using human-in-the-loop controls for privileged operations (OWASP). If your corpus includes anything a customer or vendor can write into — tickets, emails, uploaded PDFs — treat retrieved text as data, never as instructions.
Over-trust. The more polished the interface, the less people verify. Citations and visible uncertainty are the counterweight.
Cost and latency. Every question costs a retrieval plus a generation. For the ten questions asked a hundred times a day, a cached, human-written answer is better than an AI one.
A rollout that earns trust
- Pick one narrow domain with an owner: refund and cancellation policy, or the security questionnaire library. Not "all company knowledge."
- Clean the corpus first. One canonical document per topic; archive the rest so retrieval cannot reach it.
- Build the golden set before the system, from real questions.
- Ship internal and read-only, with citations on every answer and an obvious "report a bad answer" button.
- Measure for two weeks against the golden set and against real usage; review every reported bad answer.
- Only then consider customer-facing use, with a person approving the first outbound replies — the tiering logic is in human in the loop.
- Assign maintenance. A corpus with no owner degrades to a liability in about a quarter.
When not to use RAG at all
- The knowledge fits in a prompt and rarely changes. Paste it.
- The answer is an exact fact in a system of record. Query it.
- The corpus is a mess and nobody will own it. Fix the mess; that is the project.
- The real problem is that ten people give ten different answers. RAG will faithfully reproduce all ten.
That last case is worth sitting with. A retrieval system is a mirror of your documentation. If the mirror shows chaos, the tool is working — the input isn't. Which is exactly why MitHub's method puts diagnosis before building, and why the best first AI project in most companies is smaller and less glamorous than the one people ask for.
Ready to build one? The free foundations of the Faculty of Revenue Reverse Engineering walk from mapping a process to proving the result, and agentic workflows covers what happens when retrieval becomes one tool among several.