Why eval is the whole game

Anyone can build an LLM demo that works on the three inputs they tried. Shipping a product means knowing it works on the inputs you haven't tried — and not silently regressing when you change a prompt, swap a model, or update a dependency. That knowledge comes only from evaluation: measuring output quality against a dataset, repeatably. Job posts list eval as a differentiating skill precisely because most people skip it and ship on vibes. Two modes:

• Offline eval — run a fixed dataset before shipping. A regression gate; fast, repeatable, CI-friendly.
• Online eval — measure real production traffic (sampled, traced, sometimes user-signal-based). Catches what the dataset didn't anticipate. (The Observability topic covers the production side.)

The eval dataset

Everything starts with a dataset of test cases — inputs plus (usually) a reference of what "good" looks like. It's the same golden set discipline from Track 3, generalized:

from dataclasses import dataclass, field

@dataclass
class EvalCase:
    input: str                                   # the input to the feature under test
    reference: str = ""                          # expected/ideal output (for reference-based metrics)
    tags: list = field(default_factory=list)     # slices: "edge-case", "adversarial", "easy", by feature area…

dataset = [
    EvalCase("Refund a €50 order", "category=refund", tags=["easy"]),
    EvalCase("I hate this, cancel everything", "category=cancellation", tags=["adversarial"]),
    # …30–200 cases spanning easy, hard, edge, and adversarial inputs…
]

The tags matter: aggregate scores hide failures. Slicing by tag ("we're 95% overall but 60% on adversarial") is where the actionable signal lives.

Metric types: reference-based vs. reference-free

Two families, chosen by whether a single correct answer exists.

Reference-based (compare output to a known-good reference):

• Exact / normalized match — for classification, extraction, structured output. Cheap, deterministic, unambiguous.
• Embedding similarity — for free text with a reference: cosine between output and reference embeddings (semantic, not exact).
• BLEU / ROUGE — n-gram overlap (translation/summarization heritage); weak proxies for quality — use sparingly.

def exact_match(output: str, reference: str) -> float:
    return 1.0 if output.strip().lower() == reference.strip().lower() else 0.0   # deterministic 0/1

def embedding_similarity(output: str, reference: str) -> float:
    return cosine_similarity(embed(output), embed(reference))    # semantic closeness for free-text refs (0–1)

Reference-free (no single right answer — judge quality directly), the LLM-as-judge you met in Track 3: score an output against a rubric with a separate model call. Essential for open-ended tasks (summaries, chat, creative) where no reference string can be exhaustive.

Pairwise comparison

For subjective quality, absolute scores are noisy ("is this a 7 or an 8?"). Humans and LLM judges are far more reliable at comparisons: "is A better than B?" This is how you compare two prompts/models, and the basis of preference data and leaderboards (Elo/Bradley-Terry):

def pairwise_judge(prompt: str, output_a: str, output_b: str) -> str:
    verdict = llm(                                    # ask the judge to PICK, not to score
        f"Question: {prompt}\n\nAnswer A:\n{output_a}\n\nAnswer B:\n{output_b}\n\n"
        "Which answer is better? Reply only 'A' or 'B'.",
        temperature=0,
    )
    return verdict.strip()                             # aggregate win-rates over the dataset → which config wins

# To reduce position bias, run each pair BOTH orders (A,B) and (B,A) and require agreement.

Report win rate ("prompt v2 beat v1 on 68% of cases") — a far more trustworthy signal than comparing two absolute-score averages.

The regression suite in CI

Metrics become an engineering safeguard when they run automatically on every change and fail the build if quality drops below a threshold. This is the payoff — a bad prompt edit is caught in CI, not by users:

def run_eval(feature, dataset) -> dict:
    scores = [score_case(feature, c) for c in dataset]         # score every case
    by_tag = {}                                                 # per-slice aggregates
    for c, s in zip(dataset, scores):
        for t in c.tags:
            by_tag.setdefault(t, []).append(s)
    return {"overall": mean(scores),
            "by_tag": {t: mean(v) for t, v in by_tag.items()}}  # overall + sliced

# In CI: fail the build if quality regresses past a threshold.
result = run_eval(current_feature, dataset)
assert result["overall"] >= 0.85, f"Eval regressed: {result}"   # a failing assert blocks the merge
assert result["by_tag"]["adversarial"] >= 0.70, "Adversarial slice regressed"   # guard the weak slice too

Store results over time so you can see trends, and version the dataset alongside the code. A change that raises the overall score but tanks the adversarial slice is a regression you'd miss without slicing.

Common mistakes

• Evaluating on data you tuned on. Keep eval cases separate from prompt-tuning examples, or you measure memorization.
• A single aggregate number. Slice by tag; the overall score hides the failures that matter.
• Over-trusting BLEU/ROUGE. N-gram overlap is a weak quality proxy — prefer embedding similarity or a judge for free text.
• Unaudited LLM judges. Judges err — use temperature 0, rubrics, reasons, order-swapping, and periodic human spot-checks (Track 3).
• No CI gate. An eval you run manually and forget is not a safeguard; wire it into the build.

Recap

• Evaluation is what separates a demo from a product — measuring quality against a dataset, repeatably; offline (regression gate) and online (production) are complementary.
• Build a tagged eval dataset and slice results — aggregates hide failures.
• Choose metrics by task: reference-based (exact match, embedding similarity) when a right answer exists; reference-free (LLM-as-judge) for open-ended output.
• Use pairwise comparison (with order-swapping) and win rate for subjective quality.
• Wire a regression suite into CI with thresholds — including on weak slices — so quality is a gate, not a hope.
• Worked examples map to the exercises: reference-based scoring (Ex 1), embedding-vs-judge (Ex 2), pairwise win rate (Ex 3), and an end-to-end eval design (Ex 4).

from dataclasses import dataclass, field
from statistics import mean

@dataclass
class EvalCase:
    input: str; reference: str = ""; tags: list = field(default_factory=list)

data = [
    EvalCase("Refund my order", "refund", ["easy"]),
    EvalCase("Cancel my plan now", "cancellation", ["easy"]),
    EvalCase("this is garbage, I want out", "cancellation", ["adversarial"]),
    EvalCase("charged twice??", "billing", ["adversarial"]),
]

def exact(o, r): return 1.0 if o.strip().lower() == r.strip().lower() else 0.0
scores = [exact(classify(c.input), c.reference) for c in data]

by_tag = {}
for c, s in zip(data, scores):
    for t in c.tags: by_tag.setdefault(t, []).append(s)
print("overall:", mean(scores), "by_tag:", {t: mean(v) for t, v in by_tag.items()})

def emb_sim(o, r): return cosine_similarity(embed(o), embed(r))

def judge(summary, source):
    v = llm(f"Score 0-1 how faithful AND concise this summary is of the source. "
            f"Reply just the number.\n\nSOURCE:\n{source}\n\nSUMMARY:\n{summary}", temperature=0)
    return float(v.strip())

print("emb:", emb_sim(candidate, reference_summary))   # needs a reference; semantic closeness
print("judge:", judge(candidate, source_text))          # no reference needed; rubric-based

def pick(q, a, b):
    return llm(f"Q: {q}\nA:\n{a}\nB:\n{b}\nWhich is better? Reply 'A' or 'B'.", temperature=0).strip()

wins_v2 = 0; counted = 0
for case in dataset:
    o1, o2 = summarize_v1(case.input), summarize_v2(case.input)
    r1 = pick(case.input, o1, o2)          # v1 as A, v2 as B
    r2 = pick(case.input, o2, o1)          # swapped: v2 as A, v1 as B
    # Agreement means: r1 says B (v2) AND r2 says A (v2). Count only decisive, bias-controlled cases.
    if r1 == "B" and r2 == "A": wins_v2 += 1; counted += 1
    elif r1 == "A" and r2 == "B": counted += 1
print(f"v2 win rate: {wins_v2}/{counted}")