RAG for business documents: how an AI assistant on your company files works

A business AI assistant that answers from your own documents almost always relies on RAG: a search engine finds the right passages in your procedures, contracts or product sheets, then a language model writes an answer from those passages and cites them. It is worth building when your people spend time hunting for information that already exists in writing. Whether it works depends on three things demos rarely show: document quality, access permissions and data protection.

This guide is written for business owners and operations leaders, not machine learning engineers. It covers how RAG works, where it pays off, the three ways to get one, the data protection questions, the limits, and what it actually costs, with sourced figures.

RAG, explained without jargon

A language model, the engine behind ChatGPT, Claude or Gemini, only knows what it read before it was released. It has never seen your returns policy, this year's price list or the contract amendment signed last month. Ask anyway and it will answer confidently, and often wrongly.

RAG, short for retrieval-augmented generation, fixes that without changing the model. Picture a librarian and a writer. The librarian takes the question, pulls the three or four passages that answer it from your files, and hands them over. The writer drafts the answer from those passages only, and notes where each fact came from.

The four steps behind the chat window

1. Collect: documents are gathered from where they live: shared drives, SharePoint, Google Drive, Confluence, a help desk, a document management system.
2. Split and index: each document is cut into passages, and each passage is turned into a vector, a numerical fingerprint of its meaning. Its source, date and access rights are stored alongside.
3. Retrieve: for each question, the system finds the passages closest in meaning, ideally combined with exact keyword matching, which matters for part numbers, product codes and clause references.
4. Generate: the model receives the question and the selected passages, and writes an answer that links back to the original documents.

When a document changes, you re-index that document. Nothing needs retraining.

Use cases that work

RAG performs best when the answer exists in text and the question keeps coming back.

It works poorly in three situations, and knowing them early saves a project.

• Number crunching: "What was revenue by region last quarter?" belongs in a reporting tool connected to your data, not in a document search.
• Live data: today's stock level or an order's status lives in your ERP or CRM. The assistant then needs connectors to query those systems, which turns it into an agent. Our breakdown of what a custom AI agent costs covers that step.
• Summarising everything: "Summarise all our contracts" goes beyond what retrieving a handful of passages can do reliably.

Three ways to get a document assistant

1. An off-the-shelf assistant

Business plans of mainstream assistants let you upload files to a project or connect a document space. For example, Mistral listed its Le Chat Team plan at USD 24.99 per user per month, excluding tax, on its pricing page on 6 November 2026. This is the fastest route for a modest set of non-sensitive documents and simple permissions, and the right first step for many small businesses.

2. A managed building block from a cloud provider

Major providers sell document search as a service. OpenAI's file search tool, for example, was priced at USD 0.10 per GB of storage per day (first GB free) and USD 2.50 per 1,000 calls, on top of model usage, according to its pricing page on 6 November 2026. AWS (Bedrock Knowledge Bases), Google Cloud and Microsoft Azure offer equivalents. You still need an application around it: screens, source connectors, permissions and monitoring.

3. A custom assistant integrated with your systems

A custom build makes sense when permissions are fine-grained (by client, department or site), when the assistant must live inside an existing business application, or when data must stay in your own environment.

Data protection: the questions to settle first

As soon as your documents hold personal data (customer names, employee files, supplier correspondence), data protection law applies to the assistant like any other processing. In the EU and the UK, that means the GDPR and the UK GDPR; the UK regulator's guidance on AI and data protection is a practical reference. Elsewhere, check your local rules and any sector obligations.

• Deployment model. The French regulator, the CNIL, states in its Q&A on generative AI (18 July 2024) that for personal data or sensitive and strategic documentation, such as a RAG corpus, an on-premises deployment generally appears safer. Running the assistant in your own cloud account, in the region of your choice, is the usual middle ground.
• Reuse by the provider. Confirm in the contract that prompts and documents are not used to train or improve the provider's models, and switch that option off where it exists.
• Minimisation. Index only what the use case needs. Individual HR files and health information have no place in a procedures assistant.
• Processor agreements with every provider, and a clear answer on where data, backups and logs are stored.
• Transparency towards employees and, where the risk is high, a data protection impact assessment.

Our GDPR checklist for business applications goes through these steps, and our guide to hosting a business application in Europe compares the hosting options. If you serve EU users, the AI Act also adds a transparency duty: people should know they are interacting with an AI.

Permissions: what demos leave out

An assistant that reads every company document can answer anyone with any document. A sales rep asking about a colleague's salary must not receive an extract of an employment contract, even if the file sits on a shared drive.

The design rule is simple: permissions are enforced at retrieval, not in the prompt. Each indexed passage carries the permissions of its source document, and the search only returns passages the user may read. The model cannot leak what it never received. Telling the model "do not reveal confidential documents" protects nothing.

Two practical consequences. Permissions must stay in sync with the source (a file removed from a shared folder must leave the index), and the filtering logic must be tested like security code, at every change.

Limits to accept before you start

• Hallucinations shrink, they do not vanish. Require clickable citations, and an explicit "I don't know" when no passage fits.
• Documents set the ceiling. Three contradictory versions of the same procedure produce contradictory answers. Cleaning sources is step one, not an afterthought.
• Scans and complex tables resist. A scanned PDF without text recognition, or a dense spreadsheet pasted into a document, reads badly. Plan dedicated handling for those formats.
• Prompt injection. A document can contain text that hijacks the model's behaviour. The OWASP Top 10 for LLM applications ranks it first. An assistant that only answers limits the damage; an agent that acts in your systems needs stronger guardrails.
• Quality must be measured. Write fifty real questions with expected answers, and rerun them whenever the model, the chunking or the document set changes.

What a document assistant really costs

Costs fall into five buckets.

1. Document preparation: inventory, clean-up, removal of duplicates and outdated versions. Usually underestimated, and partly internal work.
2. Development: source connectors, indexing, the question interface, permissions, conversation logs.
3. Model usage, billed by the volume of text sent and generated, counted in tokens.
4. Hosting of the index and the application.
5. Running it: monitoring answer quality, onboarding new documents, keeping up with model changes.

For model usage, here is a back-of-the-envelope estimate, with assumptions to replace by your own. Fifty people ask ten questions per working day, about 10,000 questions a month. Each question sends 6,000 tokens to the model (the question plus retrieved passages) and receives 500. Using the price points we recorded on 3 November 2026 from Anthropic, OpenAI and Google's official pages, detailed in our AI agent cost guide:

The spread is wide, and an entry-level model is often enough once retrieval does its job well. Only a test on your own questions settles it.

A five-step method

1. Pick one use case with a defined audience and real question volume: internal support for one product line, quality procedures for one site.
2. Clean the documents in that scope, and name who keeps them up to date.
3. Write the test questions and expected answers before building anything.
4. Build a first assistant on that scope, with permissions from day one, and give it to a small group of users.
5. Measure: share of correct answers, unanswered questions, time saved. Then widen the scope.

If the assistant must live inside an existing tool, our guide to adding AI features to a business application covers the integration side.

When not to build custom

If your documents fit in one workspace, everyone may read everything and no personal data is involved, an off-the-shelf assistant will do the job faster and cheaper. Custom work earns its keep when permissions, integration or hosting become requirements.

Where to start

Start with a question your team asks ten times a week and whose answer already exists in writing. It is the best test of value you can run.

For the next step, we build a free visual prototype, presented over video within 5 working days: you see the question screen, the cited answers and the permission model before anything is developed. Our comparison of prototype, MVP and proof of concept explains what that first step includes.