What is RAG? How chatbots answer from your own documents
RAG, or retrieval-augmented generation, is a method where a chatbot first searches your own uploaded documents for the relevant passage, then asks the AI model to write its answer based on that passage rather than from memory alone. This grounds replies in your actual policies and catalogue, making them far more accurate than an AI model guessing from its general training.
- RAG works in two steps: retrieve the relevant passage from your documents, then generate an answer using it.
- This reduces hallucination because the model is answering from real, retrieved text instead of guessing from training data.
- Upload documents and URLs that are specific, current and directly relevant — not your entire website archive.
- Stale documents produce stale answers, so a knowledge base needs periodic review, not a one-time upload.
- On Convo360, an AI agent's knowledge base is built by uploading documents or URLs, and RAG runs automatically behind every reply.
What RAG means, in plain terms
Retrieval-augmented generation, or RAG, is a two-step process. First, when a customer asks a question, the system retrieves the most relevant passage from a set of documents you've provided, a policy page, a product sheet, an FAQ. Second, it generates a reply by handing that retrieved passage to the AI model along with the question, so the model answers using your actual text rather than whatever it remembers from general training.
The name describes exactly what happens: retrieve first, then generate. Neither step alone is enough: retrieval without generation is just a search box, and generation without retrieval is a model guessing from memory.
A useful way to picture it: imagine a new store assistant who, before answering any customer question, is handed the one relevant page from your policy folder to read first, then asked to explain it in their own words. They're not reciting from memory. They're reading, then explaining. That's RAG, running automatically behind every chatbot reply.
Why this reduces wrong answers
An AI model asked a question with no supporting documents answers from patterns learned during training, which can sound fluent and confident while being entirely wrong for your specific business, which is often called hallucination. RAG narrows the model's job: instead of 'recall what a typical return policy looks like', it becomes 'summarise this specific paragraph I just gave you', which is a much easier, more reliable task for the model to do accurately.
It doesn't eliminate mistakes entirely. The model can still misread a passage, or a question can fall outside everything you've uploaded, but it dramatically narrows the gap between what the bot says and what your business actually does.
What to upload, and what not to
Good candidates for a knowledge base are documents that are specific, current and written to answer real questions: your FAQ page, return and shipping policy, product specification sheets, pricing pages, or a services brochure. URLs work too, pointing the agent at a live policy page means it can pull from whatever's currently published.
Avoid dumping in everything you have. A ten-year archive of old blog posts, outdated price lists, or internal notes not meant for customers all add noise that can confuse retrieval and occasionally surface an answer you didn't intend to give out. Keep the knowledge base to what you'd be comfortable a new front-desk hire reading and quoting directly.
Why freshness matters
RAG answers are only as current as the documents behind them. If your festive return window changes from 15 days to 7 days and the uploaded policy document still says 15, the bot will confidently quote the old number — it has no way of knowing the document is outdated. This is the most common cause of a RAG-powered bot giving a wrong answer that isn't the AI's fault at all.
Treat your knowledge base like a living shelf, not a filing cabinet: review it whenever a policy, price or catalogue changes, and remove documents that no longer apply rather than leaving them to compete with the updated version.
RAG versus a plain AI model with no documents
| Plain LLM, no documents | RAG (with knowledge base) | |
|---|---|---|
| Source of answers | General training data only | Your uploaded documents, retrieved per question |
| Accuracy on business-specific facts | Often wrong or generic | Grounded in your actual policy or catalogue |
| Needs updating when policy changes | Cannot be updated by you at all | Yes, update the document, answers follow |
| Best for | General conversation, broad questions | Order policies, product details, business-specific FAQs |
Common misunderstandings
A frequent assumption is that uploading one document 'trains' the model permanently, like teaching it a new skill. It doesn't. RAG doesn't retrain the underlying LLM at all; it simply retrieves the relevant passage fresh for every question, which is exactly why updating a document takes effect immediately with no retraining delay.
Another misunderstanding is expecting RAG to work well with vague, unstructured documents. A knowledge base built from clear, well-organised pages with headings and specific facts retrieves far better than one long, meandering document trying to cover everything at once.
A third misunderstanding is assuming RAG guarantees an answer exists for every question. If a customer asks something genuinely outside every document you've uploaded, a brand-new scenario your policies never anticipated, the correct behaviour is for the agent to say so and hand over to a human, not stretch an unrelated passage to fit. A well-written system prompt should explicitly instruct the agent to do this rather than force an answer.
How this works on Convo360
Every Convo360 AI agent has a knowledge base you build by uploading documents or pointing it at URLs, no technical setup beyond that is required. Once documents are added, RAG runs automatically behind every customer question: the agent retrieves the relevant passage and answers using your model of choice, whether OpenAI, Anthropic Claude or Google Gemini. See /docs/ai-agents/ and the step-by-step guide at how to train your AI agent's knowledge base for setup.
Because the knowledge base sits alongside the system prompt and temperature setting on the same agent, you can pair a strict, low-temperature style with a tightly scoped knowledge base for something like warranty queries, or a warmer, more conversational style with a broader knowledge base for general product discovery, both drawing on the same underlying RAG mechanism.
Step by step
- Gather your current documentsCollect your FAQ, return/shipping policy, pricing page and product sheets — whatever a customer might realistically ask about.
- Upload to the AI agent's knowledge baseIn the agent settings, upload the documents or add the relevant page URLs directly.
- Test with real customer questionsUse Preview / Try-it to ask the questions your customers actually send, and check the answers match your real policy.
- Review after every policy changeWhenever pricing, returns or catalogue details change, update or replace the relevant document immediately.
Example conversation
Set up your AI agent's knowledge base
Start free in minutes — no code, no credit card. Bring your own AI keys and pay no per-message markup.
Frequently asked questions
What does RAG stand for in AI chatbots?
Retrieval-augmented generation — a method where the chatbot first retrieves a relevant passage from your documents, then generates its answer based on that passage rather than from general training data alone.
Does RAG completely stop an AI chatbot from giving wrong answers?
No, but it significantly reduces it. The model still relies on the documents you've provided being accurate and current, and can occasionally misread a passage or face a question outside what's uploaded.
What kind of documents should I upload to a chatbot's knowledge base?
Specific, current and customer-facing content — FAQs, return and shipping policies, product sheets and pricing pages. Avoid outdated archives or internal notes not meant for customers.
Does RAG retrain the AI model itself?
No. RAG doesn't change or retrain the underlying model — it retrieves the relevant passage fresh for each question, which is why updating a document takes effect immediately with no retraining needed.
How often should I update my chatbot's knowledge base?
Whenever a policy, price or catalogue detail changes. A knowledge base with outdated documents will produce outdated answers even though the AI itself is working correctly.
Can I use a website URL instead of uploading a file?
Yes, most RAG-based knowledge bases, including Convo360's, accept URLs alongside uploaded documents, letting the agent reference a live page directly instead of a downloaded copy that can quietly go out of date.
Related reading
Available across India — see WhatsApp & AI calling automation by state and city.