Building RAG from scratch

RAG. Vector databases. Embeddings. Semantic search. Chunking.

If you’ve read anything about AI this year, you’ve probably seen these words so often they don't mean anything anymore. And when you try to look them up, it's either complicated linear algebra or a really vague explanation. So let's start from the beginning.

RAG is pretty simple: the model itself doesn’t know your data, so we fetch it for it.

When you ask a question, the system first retrieves relevant text from your documents - whether those are notes, company documents, or documentation - and includes it in the prompt. The model then answers using that context instead of relying on training data alone and hallucinating.

But how does it work??

First, how does a word become a number? (or: what are vector embeddings?)

A computer might be able to compare the content of two sentences directly, but it can't compare their semnatic meanings. It can only compare numbers. So text has to become numbers, and not just any numbers; ones that capture meaning.

Imagine plotting every word in the dictionary on a giant map, where similar words end up near each other. "Dog" and "puppy" sit close together. "Dog" and "stapler" sit much farther apart. "King" and "queen" sit near each other, but shifted in a consistent direction: "king" will be closer to "man", whereas "queen" will be closer to "woman." That map is what a vector embedding model builds, and is also the way LLMs themselves use and process information.

The same trick extends from single words to whole sentences and paragraphs, producing a list of numbers called an embedding.

flowchart LR
    A["text: a word or sentence"] --> B[Embedding Model]
    B --> C["a list of numbers
    [0.12, -0.87, 0.33, ...]"]
    C --> D["plotted on the meaning map"]

Texts with similar meanings end up with similar vectors, even if they don't share a single word. For example, "my car wouldn't start" and "the vehicle failed to turn on" land close together, because the embedding captures meaning, not spelling. That's the whole trick behind semantic, vector-based search: compare lists of numbers, not words. Let's pivot back to RAG.

Two ways to think about retrieval

Vector retrieval uses the embedding idea above. Every chunk of a document gets converted into a list of numbers ahead of time and stored. A question gets converted the same way, and whichever stored chunks land closest on the map get pulled out with the highest priority and are passed to the model.

Vectorless retrieval skips embeddings. There are two types, old-school keyword matching, and having a back-and-forth with the model using a table of contents (i.e. PageIndex). Let's explore both ideas.

Building the chunking step

Vector search and keyword search both need documents broken into small pieces first, because there's no way to point at a specific paragraph inside one giant embedding. The simplest way to do this is to split every chunk_size words into a seperate chunk - if processing PDFs (often coming out of OCR), another way is page-based (spoiler!).

def chunk_text(text, chunk_size=200):
    words = text.split()
    chunks = []
    start = 0
    while start < len(words):
        end = start + chunk_size
        chunks.append(" ".join(words[start:end]))
    return chunks

Building vector retrieval

import numpy as np
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")

chunks = [
    "Cats are popular pets",
    "Dogs enjoy playing fetch",
    "Birds can fly really long distances"
]

vectors = model.encode(chunks, normalize_embeddings=True)

def search(query):
    query_vector = model.encode([query], normalize_embeddings=True)[0]
    scores = vectors @ query_vector
    best = np.argmax(scores)
    return chunks[best]

Each chunk gets mapped to its list of numbers and stored. A search compares the question's numbers against every stored chunk and returns whichever are closest. Now, for example, let's say we were to execute the query

search("Which animal likes to play with a ball?")

Since the idea of balls is most closely mapped to the game of fetch, the question has the highest cosine similarity with "Dogs enjoy playing fetch", and that is what will be printed. This is how things work with a LLM too, but generally with much larger chunks

Building vectorless retrieval

Keyword search (BM25)

BM25 counts how often the words in a question appear in each chunk, weighted so rare words (often subject-specific vocabulary) matter more than common ones. No neural network, no embeddings, just word frequency math. It shines when questions use the same terminology as the source, product codes, names, exact phrases embeddings tend to blur.

from rank_bm25 import BM25Okapi
import numpy as np

chunks = [
    "Cats are popular pets",
    "Dogs enjoy playing fetch",
    "Birds can fly really long distances"
]

bm25 = BM25Okapi([text.lower().split() for text in chunks])

def search(query):
    scores = bm25.get_scores(query.lower().split())
    best = np.argmax(scores)
    return chunks[best]

Since BM25 is looking at actual word matches and ignoring semantic meanings completely, the query we used before:

search("Which animal likes to play with a ball?")

would probably not get us the result we wanted. However, a query containing one of the words in the knowledge base, such as

search("Which animal likes to play fetch?")

would instantly yield the right answer. This makes BM25 a fast and effective lookup method when the user uses the same words that appear in their knowledge base - but it may miss synonyms with the same actual meaning.

Reasoning over structure (PageIndex)

BM25 and vector search both split a document into arbitrary chunks, then score each chunk independently. On the other hand, PageIndex works by building a tree based on the document's actual structure, its pages, sections, and headings. Each node in the tree gets a short AI-generated summary. When a question is asked, the LLM reasons over these summaries to decide which branch of the document is most likely to contain the answer.

For example, imagine a 100-page user manual:

If someone asks a query like "How do I fix the application when it won't start?", the LLM won't compare the question against every paragraph. Instead, it first selects the Troubleshooting section. It will recieve summaries of specific subsections in response, then search within that branch for the exact answer. This back-and-forth lets it ignore most of the document and focus only on the pages that are likely to be relevant. This way it can choose exactly what it needs without filling the model's context (and using so, so, so many tokens).

The code below splits by page, rather than by section, which is a slightly more involved process and depends on the structure of your document. However, the structure and concept still holds.

class PageIndex:
    def __init__(self, pages):
        self.pages = pages
        self.summaries = [llm_call(f"Summarize this page:\n\n{p}") for p in pages]

    def find_correct_page(self, query):
        prompt = f"""
You are given page summaries. Pick the most relevant page index.

Summaries:
{chr(10).join(f"{i}: {s}" for i, s in enumerate(self.summaries))}

Question: {query}

Return only the index.
"""

        idx = int(llm_call(prompt).strip())
        return self.pages[idx]

This often shines in situations with long, structured documents.

The generation step

However the chunks got found, the last step is the same: hand them to the model as context, then ask the question.

def answer_question(query, store, top_k=3):
    chunks = store.search(query, top_k=top_k)
    context = "\n\n".join(chunks)

    prompt = f"""Answer the question using only the context below.
If the context doesn't contain the answer, say so.

Context:
{context}

Question: {query}"""

    return llm_call(prompt)

Hybrid: vector first, then enrich

While building a RAG recently I learned that an actually useful RAG system involves a couple different layers.

1. Vector search first, over the whole collection, for broad semantic recall, this narrows things down to a handful of candidate pages. Feeding in a huge table of contents for larger documents isn't ideal for token consumption.
2. Pull in PageIndex summaries for those candidate pages, giving the model the surrounding structural context, not just the raw chunk.
3. Run fuzzy keyword/BM25 matching at the same time to catch exact terms vector search tends to blur, company IDs, ticket numbers, product codes. (for example, C-101 and C-102 would have very similar vectors but would be distinct to an exact matching algorithm).

Vector search decides which pages are in play, PageIndex explains where those pages sit in the document and what they're about, and keyword matching makes sure a literal string like a company ID doesn't get lost in translation.

Have fun RAG-ing!