Skip to content
Kuza Ndoto Academy

Retrieval-Augmented Generation

IntermediateChapter 6 · Page 23 of 265 min readUpdated 15 Nov 2026

Retrieval-augmented generation (RAG) lets an AI model answer questions using your documents: policies, notes, manuals or records it never saw during training. Before the model answers, a search step retrieves the most relevant passages and adds them to the prompt, so the answer is grounded in real sources.

Why RAG?

An LLM on its own has three problems for real organisations:

  • It doesn't know your private information, such as a SACCO's loan rules.
  • Its knowledge stops at a training cut-off and goes out of date.
  • When it doesn't know, it may hallucinate a plausible answer.

Retraining or fine-tuning a model every time a document changes is slow and expensive. RAG instead keeps knowledge in a searchable store that you can update at any time, and lets the answer cite which passage it came from.

The RAG pipeline

Preparation (done once, and again when documents change)

  1. Load the documents (PDFs, web pages, text files).
  2. Chunk them into small passages, typically a few hundred tokens each.
  3. Embed each chunk: an embedding model turns it into a vector that captures its meaning.
  4. Store the vectors and the chunk text in a vector database or index.

Answering (for every question)

  1. Embed the question with the same embedding model.
  2. Retrieve the chunks whose vectors are most similar to the question's vector.
  3. Augment the prompt: insert the retrieved chunks with instructions to answer only from them.
  4. Generate the answer with the LLM, ideally citing the sources.
Preparation (once, and when documents change) Load & chunkdocuments Embedchunks Vectorstore top-matching chunks Answering (every question) Question Embedquestion Vectorsearch Promptcontext +question LLM Groundedanswer
Figure 1. The two halves of RAG. The vector store built during preparation is searched for every question, and the retrieved chunks ground the LLM's answer.

Chunking

Whole documents are too long to put in a prompt and too broad to match a specific question, so they're split into chunks. Chunks usually overlap a little, so a sentence cut at a boundary still appears whole in one chunk. Here is a simple word-based chunker:

def chunk_words(text, size, overlap):
    words = text.split()
    step = size - overlap
    chunks = []
    for start in range(0, len(words), step):
        chunks.append(" ".join(words[start:start + size]))
        if start + size >= len(words):
            break
    return chunks

text = "Loans need two guarantors and members must save for three months first"
for i, chunk in enumerate(chunk_words(text, size=5, overlap=2)):
    print(i, chunk)
0 Loans need two guarantors and 1 guarantors and members must save 2 must save for three months 3 three months first

Each chunk repeats the last two words of the previous one. Real systems usually split by tokens, sentences or headings, so each chunk is a meaningful unit. Chunks that are too small lose context; chunks that are too large dilute the match and waste prompt space.

Embedding models map text with similar meaning to nearby vectors, even when the words differ: "How many people must back my loan?" lands close to "Loans require two guarantors". Closeness is usually measured by cosine similarity, the cosine of the angle between two vectors: 1 means same direction, 0 means unrelated.

Worked example with tiny 2-number vectors: question q = (0.9, 0.1), chunk A = (0.8, 0.2), chunk B = (0.1, 0.9). cos(q, A) = (0.72 + 0.02) / (√0.82 × √0.68) ≈ 0.74 / 0.747 ≈ 0.99. cos(q, B) = (0.09 + 0.09) / (√0.82 × √0.82) = 0.18 / 0.82 ≈ 0.22. Chunk A is retrieved. Real embeddings have hundreds or thousands of numbers, and vector databases (such as FAISS, Chroma or PostgreSQL with pgvector) search millions of them quickly.

A mini RAG retriever

This example uses TF-IDF from scikit-learn, a keyword-based method, in place of a neural embedding model, so it runs with no extra installs. The pipeline shape is the same:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

docs = [
    "The SACCO office opens at 8 am and closes at 5 pm on weekdays.",
    "Loans require two guarantors who are active members of the SACCO.",
    "Savings earn interest, which is paid to members at the end of each year.",
]
query = "How many guarantors do I need?"

vectorizer = TfidfVectorizer(stop_words="english")
doc_vectors = vectorizer.fit_transform(docs)          # "embed" and store the chunks
query_vector = vectorizer.transform([query])          # "embed" the question

scores = cosine_similarity(query_vector, doc_vectors)[0]
best = scores.argmax()
print(best, docs[best])
1 Loans require two guarantors who are active members of the SACCO.

Only the second document shares a meaningful word ("guarantors") with the question, so it scores highest. Notice a weakness of keywords: the question "Can I get a loan?" would not match "Loans" because the words differ. Neural embeddings handle this kind of variation far better. Next, build the grounded prompt:

prompt = f"""Answer the question using only the context below.
If the answer is not in the context, say you don't know.

Context:
{docs[best]}

Question: {query}"""
print(prompt)
Answer the question using only the context below. If the answer is not in the context, say you don't know. Context: Loans require two guarantors who are active members of the SACCO. Question: How many guarantors do I need?

This prompt is then sent to an LLM, which should answer "two guarantors who are active members", grounded in the retrieved text. In practice you retrieve the top 3 to 5 chunks, not just one, and include their source names so the answer can cite them.

Making RAG work well

  • Retrieval quality is everything. If the right chunk isn't retrieved, the model can't use it. Test retrieval separately with real questions.
  • Hybrid search (keywords plus embeddings) helps with names, codes and numbers.
  • Attach metadata (document title, date, section) to chunks so you can filter and cite.
  • Keep the index up to date when documents change, and remove outdated versions.
Common mistake

Assuming RAG removes hallucination. It reduces it, but the model can still misread a passage, mix chunks together, or answer from its own memory. Instruct it to use only the context, ask for citations, and check answers in testing. Also respect access rights: a staff chatbot must not retrieve documents the user isn't allowed to see.

Key points

  • RAG retrieves relevant passages from your documents and adds them to the prompt.
  • The pipeline: load, chunk, embed, store; then embed the question, retrieve, augment, generate.
  • Vector search finds chunks by meaning, using cosine similarity between embeddings.
  • Good chunking and retrieval matter most; grounding reduces but doesn't eliminate errors.
Up next · New chapter: Responsible AIBias & FairnessHow models can be unfair, and what to do.
→

Tip: use the ← and → keys to move between pages.