What LLMOps is (and why it's hard)

LLMOps is the practice of deploying, monitoring, versioning, and maintaining LLM applications in production. It borrows from MLOps (versioning, experiments, monitoring) and DevOps (CI/CD, rollback), but LLM systems add unique operational hazards:

• A prompt is code — but often untracked. A one-line prompt edit can swing quality as much as a code change, yet teams change prompts in a text box with no version history.
• Your core dependency is a third party that changes. The model provider updates, deprecates, or silently tweaks the model under you — behavior can shift with no change on your side.
• Non-determinism. Same input, different output — so "it works on my machine" is even weaker than usual.
• Quality is fuzzy and drifts. No compiler catches a subtly worse answer; regressions are silent (which is why Eval + Observability are prerequisites).

The through-line of this whole track: make LLM behavior measurable (evals), visible (observability), and managed (LLMOps).

Version everything as an experiment

Treat every artifact that affects output as versioned, and every change as an experiment you can compare and roll back. At minimum version: prompts, model + params, eval datasets, and retrieval configs.

from dataclasses import dataclass, field
import datetime

@dataclass
class PromptVersion:
    id: str                                   # e.g. "support-classifier@v3"
    template: str                             # the prompt text (the thing that actually changed)
    model: str                                # which model it's paired with
    params: dict                              # temperature, top_p, etc.
    eval_score: float = 0.0                   # its score on the golden set (Eval Design topic)
    created: str = field(default_factory=lambda: datetime.date.today().isoformat())

# A registry lets you compare versions, promote the best, and roll back to a known-good one:
REGISTRY = {
    "v2": PromptVersion("support-classifier@v2", "...", "gpt-4o-mini", {"temperature": 0}, eval_score=0.86),
    "v3": PromptVersion("support-classifier@v3", "...", "gpt-4o-mini", {"temperature": 0}, eval_score=0.91),
}
# Because each version carries its eval score, "promote v3" is a data-backed decision, and "roll back to v2"
# is one lookup away.

Tools formalize this: prompt registries (Langfuse, PromptLayer) and experiment trackers (MLflow, Weights & Biases) store versions, params, datasets, and metrics so runs are comparable and reproducible.

Reproducibility

To debug or compare, you must be able to reproduce a run — which means logging everything that determined the output alongside it: the exact prompt version, model and its version string (not just "gpt-4o-mini" but the dated snapshot), params, and retrieved context. Pin versions explicitly; a floating model alias can change under you:

# Pin the DATED model snapshot, not a floating alias, so behavior is stable and reproducible:
CONFIG = {"model": "gpt-4o-mini-2024-07-18", "temperature": 0, "prompt_version": "v3"}   # fully pinned

def generate(question, config=CONFIG):
    out = call_model(question, **config)
    log(trace_id=..., config=config, output=out)   # store the CONFIG with the output → reproducible later
    return out

The change lifecycle: gate, roll out, roll back

A prompt or model change follows a deploy pipeline like any code change — but the gate is an eval, not just unit tests:

1. Change a prompt/model in a branch.
2. Eval gate — run the offline regression suite (Eval Design); block if it regresses (esp. weak slices).
3. Staged rollout — release to a canary (small % of traffic), compare online metrics/cost/latency against the current version.
4. Promote or roll back — if the canary holds, ramp to 100%; if online metrics dip, roll back instantly.

def route(request):
    # Canary: send a small slice of traffic to the new version, the rest to the proven one.
    version = "v3" if hash(request.user_id) % 100 < 10 else "v2"   # 10% canary on v3, 90% on v2
    out = generate(request, REGISTRY[version])
    log_metric(version=version, latency=..., cost=..., user_signal=...)   # compare v3 vs v2 on LIVE traffic
    return out
# If v3's online metrics (quality signal, cost, latency) beat or match v2, ramp up; if they dip, route 100% to v2.

Keeping the previous version one config-flip away makes rollback instant — the most important safety property when a change misbehaves in production.

Drift and model deprecations

Even with zero changes on your side, an LLM system degrades over time — drift:

• Model drift / deprecation — the provider retires or updates the model version you pinned. You must migrate, and the new version can score differently on your evals — so re-run the golden set on every model change before switching.
• Data drift — real inputs shift away from what you built and evaluated on (new slang, new products, new attack patterns), so quality quietly drops on the new distribution.
• Prompt drift — accumulated ad-hoc prompt tweaks erode a once-tuned prompt.

Your defense is the loop you've already built: online eval over sampled traces (Observability) surfaces drift as a metric trend, and the golden set re-run catches model-change regressions before they ship.

Incident management

When quality breaks in production (a bad deploy, a provider change, a drift threshold crossed), you need a runbook, not improvisation:

1. Detect — an online-eval metric or alert fires (error rate, quality score, cost/latency spike).
2. Mitigate first — roll back to the last known-good version (instant, thanks to versioning) before diagnosing. Stop the bleeding.
3. Diagnose — use traces (Observability) to localize: which prompt/model version, which span, what changed.
4. Fix + gate — correct it, run the eval suite, canary, promote.
5. Post-mortem — add the failing case to the golden set so this specific regression can never ship silently again.

That last step is how the system gets more reliable over time: every incident becomes a permanent eval case.

Pitfalls

• Untracked prompts. Editing prompts with no version history makes regressions un-diagnosable and rollback impossible.
• Floating model aliases. Unpinned model names change under you — pin dated snapshots for stability/reproducibility.
• No eval gate on changes. Shipping a prompt/model change without the regression suite is shipping on hope.
• No canary / no rollback path. Big-bang releases with no quick rollback turn a bad change into an outage.
• Ignoring drift. Without online eval, silent model/data drift degrades quality with no alarm.
• Diagnosing before mitigating. In an incident, roll back first; investigate second.

Recap

• LLMOps operates LLM apps over time; it's harder than DevOps because prompts are untracked code and your core dependency (the model) changes under you.
• Version everything (prompts, model+params, datasets, configs) as comparable experiments, and log configs so runs are reproducible — pin dated model snapshots.
• Ship changes through an eval gate → canary rollout → promote/rollback, keeping the previous version one flip away.
• Combat drift and deprecations by re-running the golden set on every model change and watching online eval trends.
• Run incidents by a runbook: detect → mitigate (rollback) first → diagnose via traces → fix+gate → post-mortem into the golden set.
• Worked examples map to the exercises: a prompt registry (Ex 1), a canary router (Ex 2), a model-deprecation re-eval (Ex 3), and an incident runbook (Ex 4).

from dataclasses import dataclass

@dataclass
class PromptVersion:
    id: str; template: str; model: str; params: dict; eval_score: float

REGISTRY = {
    "v2": PromptVersion("clf@v2", "...", "gpt-4o-mini-2024-07-18", {"temperature":0}, 0.86),
    "v3": PromptVersion("clf@v3", "...", "gpt-4o-mini-2024-07-18", {"temperature":0}, 0.91),
}
live = "v2"

def promote():
    best = max(REGISTRY, key=lambda k: REGISTRY[k].eval_score)
    return best      # -> "v3"

def rollback(current="v3", previous="v2"):
    return previous  # instant known-good fallback

def route(user_id):
    version = "v3" if hash(user_id) % 100 < 10 else "v2"   # deterministic 10% canary
    # ... generate with REGISTRY[version], then:
    log_metric(version=version, quality=..., cost=..., latency=...)
    return version

# Decision after collecting metrics:
#   v3 quality >= v2 AND cost/latency acceptable  -> ramp v3 to 100%
#   v3 quality dips OR cost/latency worse          -> route 100% to v2 (rollback)

old = run_eval(feature_with("gpt-4o-mini-2024-07-18"), golden)   # baseline on the retiring model
new = run_eval(feature_with("gpt-4o-2025-xx-xx"),      golden)   # replacement model, SAME prompt + dataset
print(old["overall"], new["overall"], new["by_tag"])

# Decision:
#   new >= old on overall AND weak slices        -> safe swap, pin the new dated version
#   new drops (esp. on a slice)                   -> the prompt was tuned to the old model; re-tune before swapping