Why LLMs hallucinate

A hallucination is confident, fluent output that is wrong or unsupported. It's not a bug to be fully patched — it's inherent to how the model works: an LLM predicts the most probable next token, not the true one. It has no built-in model of truth and no signal for "I don't actually know this," so when its parametric memory is thin, it produces a plausible-sounding guess with the same confidence as a fact.

Two flavors worth separating (they need different fixes):

• Intrinsic / unfaithful — the output contradicts the provided source (e.g. the context said €49, the answer says €59). A faithfulness failure — fixable with grounding + verification.
• Extrinsic / factual — the output is wrong about the world with no source involved. Needs grounding (give it a source) or abstention.

Reliability is therefore not one switch but a stack of defenses, each catching what the previous missed.

Defense 1: grounding + abstention

The foundation (from RAG, Track 3): don't rely on parametric memory — retrieve the facts and instruct the model to answer only from them, and to abstain when they're insufficient. Abstention is the single most important reliability instruction: a model that says "I don't know" is infinitely more trustworthy than one that invents.

def grounded_answer(question: str) -> str:
    context = retrieve(question)                          # bring real facts into the prompt (grounding)
    prompt = (
        "Answer using ONLY the context. If the context does not contain the answer, "
        "reply exactly 'I don't know.' Do not use outside knowledge.\n\n"   # forced abstention — the key line
        f"Context:\n{context}\n\nQuestion: {question}"
    )
    return llm(prompt, temperature=0)                      # temp 0 → less improvisation on facts

Defense 2: cite sources & verify (chain-of-verification)

Make claims checkable. First, require the model to cite which context chunk supports each statement — citations turn "trust me" into "check line 3," and they make unsupported claims visible. Second, for high-stakes output, run a verification pass: a second model call that checks each claim against the source and flags anything unsupported. This "chain-of-verification" (generate → verify → revise) measurably cuts hallucination:

def verify_claims(answer: str, context: str) -> dict:
    # A SECOND LLM call acting as a fact-checker against the source (the faithfulness judge from Track 3):
    report = llm(
        "For each claim in the ANSWER, say whether the CONTEXT supports it. "
        'Return JSON: {"unsupported": ["claim", ...]}.\n\n'
        f"CONTEXT:\n{context}\n\nANSWER:\n{answer}",
        temperature=0,
    )
    return json.loads(report)                             # {"unsupported": [...]} — empty means fully grounded

def answer_with_verification(question: str) -> str:
    context = retrieve(question)
    draft = grounded_answer(question)                     # first pass
    issues = verify_claims(draft, context)               # verification pass
    if issues["unsupported"]:                             # if any claim isn't grounded…
        return llm(f"Revise to remove unsupported claims {issues['unsupported']}:\n{draft}")   # …revise it out
    return draft

Defense 3: constrain the output

You can't hallucinate an invalid category if the format won't allow it. For classification/extraction, constrain generation to a schema or enum (structured outputs / function calling from Track 4). The model physically can't return a label outside the set:

schema = {"type": "object", "properties": {
    "category": {"type": "string", "enum": ["billing", "technical", "sales"]},   # ONLY these three are possible
    "confidence": {"type": "number"},
}, "required": ["category"]}
# With structured-output/enforced-schema mode, the model can't invent a fourth category or malformed output.

Constraints don't help open-ended prose, but for anything structured they eliminate a whole class of "made-up value" errors for free. Pair with a confidence field and route low-confidence cases to a human.

Defense 4: the verification layer (and human-in-the-loop)

For high-stakes output, wrap generation in a reliability layer that decides whether to ship, revise, or escalate. It composes the defenses: ground → generate → verify → check confidence → gate:

def reliable_answer(question: str):
    context = retrieve(question)
    draft = grounded_answer(question)
    if not context.strip():                              # nothing retrieved → don't answer from memory
        return {"action": "abstain", "text": "I don't have information on that."}
    issues = verify_claims(draft, context)
    if issues["unsupported"]:                            # unsupported claims → escalate, don't silently ship
        return {"action": "escalate_to_human", "text": draft, "issues": issues["unsupported"]}
    return {"action": "answer", "text": draft}           # grounded + verified → safe to return

The pattern: abstain when you can't ground, escalate when verification fails, answer only when both pass. For irreversible actions (medical, legal, financial, sending, deleting), add the LangGraph human-in-the-loop gate (Track 4) so a person approves before anything ships.

Pitfalls

• Treating hallucination as fully fixable. It's inherent to next-token prediction — reduce and contain, don't assume a cure.
• No abstention instruction. Without an explicit "say I don't know," the model fills gaps with confident invention.
• Citations you never check. Requiring citations helps only if you (or a verifier) actually validate them.
• Skipping verification on high stakes. For medical/legal/financial output, a generate-only pipeline is negligent — add a verify pass and human gate.
• Mismatched defense depth. Over-guarding a low-stakes toy wastes effort; under-guarding a high-stakes system is dangerous.

Recap

• Hallucination is inherent: LLMs predict probable, not true, tokens — separate intrinsic (unfaithful to source) from extrinsic (factually wrong) errors.
• Layer defenses: grounding + abstention (answer only from context, else "I don't know"), citation + verification (chain-of-verification: generate → verify → revise), structured-output constraints (no invalid values), and a reliability layer that abstains/escalates/answers.
• Add human-in-the-loop for irreversible, high-stakes actions.
• You can't reach 100% — fail safe and match defense depth to stakes.
• Worked examples map to the exercises: grounding+abstention (Ex 1), citation+verification (Ex 2), enum constraints (Ex 3), the reliability layer (Ex 4).

STRONG = ("Answer using ONLY the context. If it's not there, reply exactly \"I don't know.\"\n\n"
          "Context:\n{ctx}\n\nQuestion: {q}")
WEAK   = "Context:\n{ctx}\n\nQuestion: {q}"     # no abstention instruction

q = "What is the CEO's home address?"           # not in the corpus
ctx = "\n\n".join(retrieve(q))
print(llm(STRONG.format(ctx=ctx, q=q)))          # -> "I don't know."
print(llm(WEAK.format(ctx=ctx, q=q)))            # -> often a confident invented answer

import json
def verify_claims(answer, context):
    r = llm('For each claim in ANSWER, is it supported by CONTEXT? '
            'Return JSON {"unsupported": [...]}.\n\n'
            f"CONTEXT:\n{context}\n\nANSWER:\n{answer}", temperature=0)
    return json.loads(r)

draft = "The Pro plan is €49/month and includes a free laptop."   # laptop is unsupported
print(verify_claims(draft, "The Pro plan is €49/month."))          # {"unsupported": ["...free laptop..."]}

schema = {"type":"object","properties":{
    "category":{"type":"string","enum":["billing","technical","sales"]},
    "confidence":{"type":"number"}},"required":["category","confidence"]}

out = classify_structured("asdkfj random gibberish", schema=schema)   # still returns one of the 3
if out["confidence"] < 0.6:
    route_to_human(out)          # low confidence -> review instead of trusting it

def reliable_answer(q):
    ctx = "\n\n".join(retrieve(q))
    if not ctx.strip():
        return {"action": "abstain", "text": "I don't have information on that."}
    draft = grounded_answer(q)
    issues = verify_claims(draft, ctx)
    if issues["unsupported"]:
        return {"action": "escalate", "text": draft, "issues": issues["unsupported"]}
    return {"action": "answer", "text": draft}

print(reliable_answer("unanswerable question"))   # abstain
print(reliable_answer("question that triggers an unsupported claim"))  # escalate
print(reliable_answer("well-grounded question"))  # answer