Why RAG exists

A large language model only "knows" what was in its training data, frozen at a cutoff date, blended into its weights. That creates three practical problems:

1. Staleness — it can't know anything that happened after training.
2. No private knowledge — it has never seen your company's docs, tickets, or database.
3. Hallucination — asked about something it half-knows, it will confidently invent an answer.

Retrieval-Augmented Generation (RAG) fixes all three the same way: instead of relying on the model's parametric memory, you retrieve relevant text from an external source at query time and inject it into the prompt as context. The model then answers from the provided passages, not from memory.

The pipeline

RAG has two phases. Indexing happens once (or on a schedule) over your corpus. Querying happens on every user request.

INDEXING (offline):     documents → chunk → embed → store vectors + text in an index
QUERYING (per request): question → embed → similarity search → top-k chunks → prompt + generate

1. Embeddings

An embedding is a fixed-length vector of floats (e.g. 1536 dimensions) that represents the meaning of a piece of text. An embedding model is trained so that texts with similar meaning land close together in that vector space, and unrelated texts land far apart.

from openai import OpenAI          # the OpenAI SDK; `import` ~ JS `import`

client = OpenAI()                  # reads your API key from the env var OPENAI_API_KEY

def embed(text: str) -> list[float]:      # `text: str` = param typed as string; `-> list[float]` = returns a float array
    resp = client.embeddings.create(      # call the embeddings endpoint
        model="text-embedding-3-small",   # which embedding model to use
        input=text,                       # the text to turn into a vector
    )
    return resp.data[0].embedding         # data[0] = first (only) result; .embedding = the vector itself

v = embed("How do I reset my password?")
len(v)   # -> 1536   (len() is Python's Array.length / .length)

"Close" is measured with a similarity metric, most commonly cosine similarity — the cosine of the angle between two vectors, from -1 (opposite) to 1 (identical direction). Here it is in code, because you'll implement exactly this in the first exercise:

import numpy as np                 # numpy = the array-math library (think a fast, vectorised math kit)

def cosine_similarity(a: list[float], b: list[float]) -> float:
    a, b = np.array(a), np.array(b)         # turn the plain float lists into numpy arrays so we can do vector math
    dot = a @ b                             # `@` = dot product: sum of a[i]*b[i] over every dimension
    norm = np.linalg.norm(a) * np.linalg.norm(b)   # norm() = vector length (magnitude); multiply the two lengths
    return float(dot / norm)                # cosine = dot product divided by the product of lengths → range -1..1

# Higher score = more similar in meaning:
cosine_similarity(embed("reset my password"),
                  embed("I forgot my login credentials"))   # ≈ 0.6  (same intent, different words)
cosine_similarity(embed("reset my password"),
                  embed("what are your business hours"))     # ≈ 0.1  (unrelated)

2. Chunking

You rarely embed a whole document — it's too coarse (one vector can't represent 40 pages) and blows the context window. Instead you split documents into chunks (e.g. a few hundred tokens each), and embed each chunk. Chunking is a genuine design choice:

• Too large → the chunk mixes many topics, so its embedding is a blurry average and retrieval gets imprecise; you also waste context tokens.
• Too small → a chunk loses the surrounding context needed to be meaningful ("It costs €49" — what costs €49?).
• Overlap — sliding a window with some overlap (e.g. 10–20%) keeps sentences that straddle a boundary intact.

Here's the core chunker in code. It's a sliding window: take size tokens, step forward by size - overlap, repeat — so consecutive chunks share overlap tokens.

def chunk_tokens(tokens: list[str], size: int = 200, overlap: int = 30) -> list[list[str]]:
    step = size - overlap              # how far the window advances each time (smaller step = more overlap)
    chunks = []                        # accumulator (an empty list, like `const chunks = []`)
    for start in range(0, len(tokens), step):   # range(0, N, step) = 0, step, 2*step, …  (like a for-loop with i += step)
        window = tokens[start : start + size]   # slice: tokens from `start` up to (not including) `start+size`
        chunks.append(window)                   # push this window onto the result (like Array.push)
    return chunks

# A token here is just a word for illustration; in production you'd split on a real tokenizer.
words = "the quick brown fox jumps over the lazy dog".split()   # .split() with no arg splits on whitespace → a list
chunk_tokens(words, size=4, overlap=2)
# -> [['the','quick','brown','fox'],        # window 1
#     ['brown','fox','jumps','over'],        # window 2 shares 'brown','fox' with window 1 (that's the overlap)
#     ['jumps','over','the','lazy'],
#     ['the','lazy','dog']]                  # last window is short — that's fine

The same list comprehension, written compactly, is what you'll compare against in the exercise:

# Identical logic, one line. Read it as: "for each start in the range, take that window."
def chunk_tokens(tokens, size=200, overlap=30):
    step = size - overlap
    return [tokens[i : i + size] for i in range(0, len(tokens), step)]

3. Indexing / storing

Each chunk's embedding is stored in a vector index alongside the original text and metadata (source, title, timestamp). A vector database (pgvector, Qdrant, Azure AI Search, …) does the approximate-nearest-neighbor search efficiently at scale. For learning, a plain Python list is enough:

# Build the tiny "index": for each chunk keep its text AND its embedding, side by side.
chunks = ["...", "...", "..."]                       # your chunked passages (strings)
index = [(text, np.array(embed(text))) for text in chunks]   # list of (text, vector) pairs — a list of tuples
# A tuple `(a, b)` is just a fixed 2-element pair, like a readonly [a, b] in JS.

4. Retrieval

At query time you embed the user's question and ask the index for the top-k most similar chunks (e.g. k=4). Those chunks are your evidence.

def retrieve(question: str, k: int = 4) -> list[str]:
    q = np.array(embed(question))                       # embed the question into the SAME space as the chunks
    # For every stored (text, vector) pair, score it against the question, keep (score, text):
    scored = [(cosine_similarity(q, vec), text) for (text, vec) in index]
    scored.sort(reverse=True)                           # sort by score descending (highest similarity first)
    return [text for (score, text) in scored[:k]]       # scored[:k] = first k items; return just their text

5. Generation

Finally you build a prompt that contains the retrieved chunks and the question, and instruct the model to answer only from the provided context — and to say it doesn't know if the context is insufficient.

def answer(question: str) -> str:
    context = "\n\n".join(retrieve(question))   # join the retrieved chunks into one string, blank line between each
    prompt = f"""Answer the question using ONLY the context below.
If the context doesn't contain the answer, say you don't know.

Context:
{context}

Question: {question}
"""                                             # triple-quoted f-string = multi-line template literal with {…} interpolation
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],   # a list with one message dict (dict ~ JS object)
    )
    return resp.choices[0].message.content      # pull the text out of the response

That "only from the context / say you don't know" instruction is what turns retrieval into grounding — it's the difference between RAG and just pasting search results next to a hallucinating model.

Evaluating RAG

RAG has two failure surfaces, so you evaluate both the retriever and the generator:

Metric	Question it answers	Failure it catches
Context recall	Did retrieval fetch the chunks that contain the answer?	Missing evidence (retriever failure)
Context precision	Are the retrieved chunks actually relevant (not noise)?	Distracting/irrelevant retrieval
Faithfulness / groundedness	Is the answer supported by the retrieved context?	Hallucination despite good context
Answer relevance	Does the answer actually address the question?	On-topic-but-useless answers

A key diagnostic habit: if the answer is wrong, first check whether the right chunk was even retrieved. The one-line habit that saves hours is to print the retrieved chunks next to the answer:

def debug_answer(question: str):
    hits = retrieve(question)                   # the chunks the retriever chose
    print("RETRIEVED:")
    for i, text in enumerate(hits):             # enumerate() gives (index, item) pairs, like JS .entries()
        print(f"  [{i}] {text[:80]}…")          # text[:80] = first 80 chars; f-string formats the line
    print("ANSWER:", answer(question))
# Now read the two together:
#   • answer-bearing chunk NOT in RETRIEVED  → retriever failure (fix chunking / k / embeddings)
#   • it IS retrieved but the answer is wrong → generator/faithfulness failure (tighten the prompt or model)

If recall failed, no amount of prompt tuning helps. If recall was fine but the answer is still wrong, that's a generation/faithfulness problem.

Pitfalls

• Garbage retrieval, confident answer. If you don't instruct the model to abstain, it will answer from memory when retrieval fails — reintroducing the hallucination you were avoiding.
• Embedding/model mismatch. Different models for query vs. documents ⇒ meaningless similarity.
• Chunk size chosen by vibes. It materially changes quality; treat it as a tunable evaluated against your golden set.
• No evaluation. Without measuring recall/faithfulness you're flying blind and can't tell a retriever bug from a generation bug.

Recap

• RAG grounds an LLM in external, current, private knowledge by retrieving relevant text at query time and instructing the model to answer only from it.
• The pipeline is embed → chunk → index → retrieve → generate; indexing is offline, retrieval and generation are per-request.
• Embeddings put meaning in a vector space; cosine similarity finds the closest chunks; use the same embedding model for queries and documents.
• Evaluate the retriever (context recall/precision) and the generator (faithfulness, answer relevance) separately — and diagnose wrong answers by checking retrieval first.
• You now have a commented, worked example for every exercise: cosine_similarity (Ex 1), chunk_tokens (Ex 2), the assembled pipeline (Ex 3), and debug_answer (Ex 4).

import numpy as np
from openai import OpenAI
client = OpenAI()

def embed(t): return client.embeddings.create(model="text-embedding-3-small", input=t).data[0].embedding
def cos(a, b):
    a, b = np.array(a), np.array(b)
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

texts = ["How do I reset my password?",
         "I forgot my login credentials",
         "What are your business hours?",
         "The mitochondria is the powerhouse of the cell"]
vs = [embed(t) for t in texts]
for t, v in zip(texts[1:], vs[1:]):
    print(f"{cos(vs[0], v):.3f}  {t}")

def chunk(tokens, size=200, overlap=30):
    step = size - overlap
    return [tokens[i:i+size] for i in range(0, len(tokens), step)]

import numpy as np
from openai import OpenAI
client = OpenAI()

chunks = [ ... ]  # ~10 short passages from your source
emb = [np.array(embed(c)) for c in chunks]

def retrieve(q, k=3):
    qv = np.array(embed(q))
    sims = [float(qv @ e / (np.linalg.norm(qv)*np.linalg.norm(e))) for e in emb]
    return [chunks[i] for i in np.argsort(sims)[::-1][:k]]

def answer(q):
    ctx = "\n\n".join(retrieve(q))
    prompt = (f"Answer ONLY from the context. If it's not there, say you don't know.\n\n"
              f"Context:\n{ctx}\n\nQuestion: {q}")
    return client.chat.completions.create(model="gpt-4o-mini",
        messages=[{"role":"user","content":prompt}]).choices[0].message.content