Why LLM apps need observability

A traditional app is deterministic: same input, same output, and a stack trace tells you what broke. LLM apps are the opposite — non-deterministic, multi-step (retrieve → maybe several tool calls → generate), and they fail silently and semantically: no exception, just a subtly wrong answer. When a user reports "the agent gave a bad answer," you need to see exactly what happened: what was retrieved, the exact prompt sent, which tools ran with what arguments, the raw response, and the tokens/latency/cost of each. That's observability — and without it you're debugging blind.

Traces and spans

The core abstraction: a trace = one full request end to end; spans = the individual steps inside it, nested to show structure. A RAG agent request might be:

Trace: "answer user question"                      (2.3s, €0.004, 1,850 tokens total)
├─ Span: retrieve            (0.2s)  query -> 4 chunks
├─ Span: LLM call [agent]    (0.9s, 1,100 tok)  decides to call a tool
├─ Span: tool: get_weather   (0.1s)  Paris -> "rainy"
└─ Span: LLM call [final]    (1.1s, 750 tok)  grounded answer

You capture this by wrapping each step. Manually, a span records inputs, outputs, timing, and token usage:

import time, contextlib

@contextlib.contextmanager                             # makes a `with`-usable span (like a try/finally scope)
def span(trace: list, name: str, **meta):
    start = time.perf_counter()                        # start the clock
    record = {"name": name, "meta": meta}              # what this step is
    try:
        yield record                                   # the wrapped work runs here; it can add to `record`
    finally:
        record["latency_s"] = time.perf_counter() - start   # always capture duration, even on error
        trace.append(record)                           # add this span to the trace

def traced_llm(trace, prompt):
    with span(trace, "llm_call", model="gpt-4o-mini") as s:
        resp = client.chat.completions.create(model="gpt-4o-mini",
            messages=[{"role":"user","content":prompt}])
        s["prompt"] = prompt                                    # capture INPUT
        s["output"] = resp.choices[0].message.content          # capture OUTPUT
        s["tokens"] = resp.usage.total_tokens                  # capture token usage (for cost)
        return s["output"]

What to capture

For every LLM span, log at minimum:

• Inputs & outputs — the exact prompt (post-templating) and raw response. This is what you'll actually read when debugging.
• Model & params — model id, temperature, etc. (a silent model swap is a common regression cause).
• Tokens — prompt + completion counts → drives cost.
• Latency — per span, so you can find the slow step.
• Metadata — user/session/trace id, app version, feature tag — for filtering and slicing.
• Errors & retries — what failed and how it recovered.

Nested spans for multi-step agents

The value compounds for agents: a nested trace shows the whole decision path, so you can see which step went wrong — the retriever fetched junk, or the model ignored good context, or a tool errored:

def traced_agent(question):
    trace = []                                          # one trace per request
    with span(trace, "agent_request", question=question):
        with span(trace, "retrieve") as s:
            s["chunks"] = retrieve(question)            # record what retrieval returned
        answer = traced_llm(trace, build_prompt(question, s["chunks"]))   # nested LLM span
    return answer, trace                                # return the answer AND its full trace
# Reading the trace: bad answer + junk in the retrieve span = retriever bug; good chunks + bad answer = generator bug.

This is the check-retrieval-first diagnostic from RAG, now backed by data instead of guesswork — the trace shows you which stage failed.

Cost & latency

Tokens convert to money via per-model pricing; aggregating across traces gives you the dashboards that keep an app affordable and fast:

PRICE = {"gpt-4o-mini": {"in": 0.15/1e6, "out": 0.60/1e6}}   # $ per token (illustrative; check current pricing)

def span_cost(model, prompt_tokens, completion_tokens):
    p = PRICE[model]
    return prompt_tokens * p["in"] + completion_tokens * p["out"]   # cost of one LLM call

# Aggregate across many traces to answer the questions that actually matter operationally:
#   • cost per request (and per user/feature)   • p50/p95 latency, and WHICH span dominates it
#   • token trends over time                     • error rate per model/endpoint

These are the metrics a dashboard (Langfuse, Grafana) surfaces: a prompt change that doubles token usage, a p95 latency spike traced to one slow tool, a cost-per-user creeping past your margin — all invisible without observability, obvious with it.

Tools & connecting to eval

You rarely hand-roll all this. Purpose-built LLM observability platforms capture traces, cost, and latency and add dashboards, and most auto-instrument LangChain/LangGraph/LlamaIndex:

• Langfuse — open-source, self-hostable tracing + evals + prompt management (a common default).
• LangSmith — LangChain's hosted tracing/eval platform.
• Phoenix (Arize) — open-source tracing + eval, strong on RAG.
• OpenLLMetry / OpenTelemetry — vendor-neutral instrumentation you can export to any backend.

The real payoff is the loop: traces feed online evaluation (Eval Design topic). Sample production traces, run your judges/metrics on them, and you catch quality drift the offline golden set never anticipated — closing the loop from "we shipped it" to "we know it's still good."

Pitfalls

• No tracing until something breaks. Instrument from day one; you can't retro-debug a request you didn't capture.
• Logging outputs but not inputs. Without the exact prompt/context you can't reproduce or explain a bad answer.
• Ignoring cost/latency until the bill or the SLA hurts. Track tokens and p95 continuously.
• Leaking PII into traces. Prompts/outputs are user data — redact, restrict access, set retention.
• Traces you never look at. Observability only pays off if you read traces and feed them into online eval.

Recap

• LLM apps are non-deterministic, multi-step, and fail silently, so ordinary logging isn't enough — you need tracing.
• A trace is one request; spans are its nested steps. Capture inputs/outputs, model/params, tokens, latency, metadata, errors per span.
• Aggregate tokens→cost and per-span latency into dashboards; watch cost-per-request and p95.
• Langfuse/LangSmith/Phoenix (over OpenTelemetry) auto-instrument frameworks; the key payoff is feeding production traces into online eval to catch drift.
• Treat traces as sensitive data — redact PII, restrict access, set retention.
• Worked examples map to the exercises: a span wrapper (Ex 1), nested agent trace (Ex 2), cost/latency aggregation (Ex 3), and trace-driven debugging (Ex 4).

import time, contextlib

@contextlib.contextmanager
def span(trace, name, **meta):
    start = time.perf_counter(); rec = {"name": name, "meta": meta}
    try:
        yield rec
    finally:
        rec["latency_s"] = round(time.perf_counter() - start, 3); trace.append(rec)

def traced_llm(trace, prompt):
    with span(trace, "llm_call", model="gpt-4o-mini") as s:
        r = client.chat.completions.create(model="gpt-4o-mini",
            messages=[{"role":"user","content":prompt}])
        s["prompt"] = prompt
        s["output"] = r.choices[0].message.content
        s["tokens"] = r.usage.total_tokens
        return s["output"]

trace = []
traced_llm(trace, "Say hi in one word.")
print(trace[0])   # {'name':'llm_call','meta':{'model':...},'latency_s':...,'prompt':...,'output':...,'tokens':...}

def traced_agent(question):
    trace = []
    with span(trace, "agent_request", question=question):
        with span(trace, "retrieve") as s:
            s["chunks"] = broken_retrieve(question)   # returns irrelevant chunks
        answer = traced_llm(trace, build_prompt(question, s["chunks"]))
    return answer, trace

answer, trace = traced_agent("What does the Pro plan cost?")
retrieve_span = next(s for s in trace if s["name"] == "retrieve")
print("retrieved:", retrieve_span["chunks"])   # junk -> retriever bug, confirmed from the trace

import statistics
PRICE = {"gpt-4o-mini": {"in": 0.15/1e6, "out": 0.60/1e6}}   # illustrative $/token

def cost(model, pt, ct):
    p = PRICE[model]; return pt*p["in"] + ct*p["out"]

calls = [("gpt-4o-mini", 900, 200), ("gpt-4o-mini", 1200, 350), ("gpt-4o-mini", 700, 150)]
costs = [cost(*c) for c in calls]
lat = [1.1, 2.4, 0.8]
print("total $:", round(sum(costs), 5))
print("p95 latency:", statistics.quantiles(lat, n=20)[-1])   # ~95th percentile

{"spans": [
  {"name": "retrieve", "chunks": ["The Pro plan is €49/month."]},
  {"name": "llm_call", "prompt": "...Context: The Pro plan is €49/month... Q: Pro plan price?",
   "output": "The Pro plan costs €59/month.", "tokens": 320}
]}