AIMentor — AI Expertise Learning Platform

13 questions · pass mark 80% · 20 minute limit. The timer auto-submits at zero.

Time remaining: 20:00

Question 1easy
Why do LLM apps need observability beyond ordinary logging?
Because they are always slowThey are non-deterministic, multi-step, and fail silently/semantically — a stack trace won't show a subtly wrong answerBecause logging is illegal for AIThey never produce errors
Question 2easy
In LLM observability, what is a trace vs. a span?
A trace is one token; a span is one wordA trace is one full request end-to-end; spans are the nested steps inside it (retrieval, LLM calls, tools)They are the same thingA span is the whole request; a trace is a single step
Question 3medium
What should you capture for each LLM span? (Select all that apply)
The exact prompt (post-templating) and raw outputModel id and params, and token countsLatency and metadata (user/session/trace id, version, feature tag)Only the final answer, nothing else
Select all that apply.
Question 4medium
Why capture the exact prompt/context, not just the output?
To make traces biggerWithout the exact input you can't reproduce or explain a bad answerPrompts are needed for billing onlyIt speeds up the model
Question 5medium
How does a nested agent trace support the 'check retrieval first' diagnostic?
It hides the retrieval stepIt shows what retrieval returned vs. what the model produced, so you can tell a retriever bug from a generator bug from dataIt retrains the retrieverIt only records the final answer
Question 6medium
How is the cost of an LLM call computed for a dashboard?
A flat fee per request regardless of lengthPrompt and completion token counts multiplied by the model's per-token input/output pricesBy the number of tools calledBy latency in seconds
Question 7hard
Which operational questions do cost/latency dashboards answer? (Select all that apply)
Cost per request, user, and featurep50/p95 latency and which span dominates itToken trends and error rate per model/endpointThe model's exact training data
Select all that apply.
Question 8hard
What is the key payoff of connecting production traces to evaluation?
It reduces token cost automaticallySampling live traces and running your judges/metrics on them catches quality drift the offline golden set never anticipatedIt replaces the need for a golden setIt fine-tunes the model on traffic
Question 9medium
Which are purpose-built LLM observability tools?
NumPy, Pandas, MatplotlibLangfuse, LangSmith, Phoenix (over OpenTelemetry/OpenLLMetry)Docker, Kubernetes, Terraformpgvector, Qdrant, Pinecone
Question 10hard
Why are traces a data-governance concern?
They are too large to storeThey contain prompts and outputs — often sensitive user data — so require PII redaction, access control, and retention limitsThey slow the model downThey cannot be encrypted
Question 11hard
A trace shows the retrieve span returned the correct chunk and the prompt included it, but the output contradicts it. What does this localize the bug to?
The retrieverThe generator (a faithfulness / intrinsic-hallucination failure) — fix grounding/temperature/verificationThe embedding modelThe vector database index
Question 12medium
What is the nested-step unit within a trace called (one word)?
Question 13hard
Which latency percentile (e.g. p__) is commonly watched to catch tail-latency spikes? Give the number.

0 / 13 answered

Observability — Final Exam