What is RAG?

A plain explanation of retrieval augmented generation, the way most organisations get AI to answer from their own documents.

Last updated 30 October 2026

The short answer

RAG, short for retrieval augmented generation, lets a language model answer questions from your own documents. When someone asks a question, the system first finds the most relevant passages in your documents, then gives them to the model with the question, so the answer draws on your material rather than only on what the model learned in training.

It is the usual way organisations build AI search over policies, contracts, research or case files, and every part of it can run inside your own estate.

How RAG works

01
Ingestion
Documents are collected from the stores you choose, such as file shares, SharePoint or a document management system, and split into passages.
02
Embeddings
Each passage is turned into an embedding, a list of numbers that captures its meaning, and stored in a vector database.
03
Retrieval
A question is turned into an embedding in the same way and matched against the vector database to find the passages closest in meaning.
04
Answer
The model receives the question with those passages and writes an answer from them, ideally pointing back to the passages so people can check it.

Why not just ask the model?

A language model only knows what was in its training data. That stops at a date, and it never included your internal documents. Asked about your policies or your contracts, a model on its own can only guess, and it can guess fluently.

RAG changes the question the model is answering. Instead of "what do you know about this?", it becomes "answer this from these passages". That makes answers more relevant, lets people see where they came from, and lets the system say it found nothing. It reduces wrong answers; it does not remove them, which is why answers should be checked against their sources.

RAG or fine tuning?

Fine tuning changes the model itself by training it further on your examples. RAG leaves the model alone and gives it the right material at the moment of the question. They solve different problems.

RAG suits facts that change

When documents are added or updated, they are indexed again and the next answer uses them. No retraining is needed.

RAG can respect permissions

Retrieval can be limited to documents the person asking is already allowed to open. Knowledge trained into a model cannot be withheld from one user and shown to another.

Fine tuning suits style and format

It can teach a model a house style, a structured output or a narrow task it performs often.

Many platforms use both

A model tuned for the task, answering from documents retrieved at question time.

Keeping RAG private

Every part of a RAG pipeline handles your documents, so every part needs a home you are happy with:

The embedding model. Easy to overlook: if embeddings are made by calling an external service, every passage of every document is sent to it.
The vector database. It holds your passages and their embeddings, so it is as sensitive as the documents themselves.
The language model. It sees each question and the passages retrieved for it.
Logs. Questions and answers often contain the most sensitive material of all, so decide what is kept and where.

In a private AI platform all four run inside your cloud tenancy or data centre, and access tests with accounts that should be refused are part of acceptance. Read what it takes to run an LLM on premises for the hardware and software involved.

Where RAG goes wrong

Most disappointing RAG systems fail for ordinary reasons, not exotic ones:

the documents are out of date, duplicated or contradict each other, so the right passage competes with the wrong one
passages are split badly, cutting a table or a clause in half
retrieval misses the passage that holds the answer, and the model fills the gap
permissions are not carried through, so search shows people documents they could not otherwise open
nobody measures answer quality, so problems surface only when someone relies on a wrong answer

Each of these can be found early by testing with real questions from the people who will use the system. That is what a proof of concept is for.

Start with your documents.

Tell us which documents your people search most and which must never move. We will come back with a scope.

Thank you. Your request is on its way. We will reply from info@tlgsolutions.co.uk to arrange a first discussion.

That did not send. Please email info@tlgsolutions.co.uk instead, and we will reply from there.