AIMentor — AI Expertise Learning Platform

Untimed. Answers are graded when you submit, with explanations and spaced-repetition scheduling.

Question 1easy
What is LLMOps?
A way to train models fasterThe practice of deploying, monitoring, versioning, and maintaining LLM applications in productionA prompt-writing techniqueA type of vector database
Question 2medium
What makes LLM systems uniquely hard to operate vs. ordinary software? (Select all that apply)
A prompt is code-like but often changed with no version historyThe core dependency (the model) is a third party that updates or deprecates under youNon-determinism and fuzzy, drifting quality with no compiler to catch regressionsThey cannot be deployed at all
Select all that apply.
Question 3medium
Which artifacts should be versioned as experiments? (Select all that apply)
PromptsModel id and params (temperature, etc.)Eval datasets and retrieval configsOnly the final source code
Select all that apply.
Question 4medium
Why pin a dated model snapshot (e.g. gpt-4o-mini-2024-07-18) instead of a floating alias?
It is cheaperA floating alias can be repointed to a new snapshot at any time, silently changing behavior; a dated pin is stable and reproducibleDated snapshots are fasterThe API rejects aliases
Question 5medium
What makes an LLM run reproducible?
Using the largest modelLogging everything that determined the output: exact prompt version, dated model, params, and retrieved contextSetting temperature to 1.0Deleting old traces
Question 6hard
In the change lifecycle, what gates a prompt/model change before rollout?
A manual smoke test onlyThe offline eval regression suite (blocking on overall and weak-slice thresholds)A vibe checkNothing — LLM changes can't be gated
Question 7hard
What is a canary rollout for a prompt/model change?
Releasing to 100% of users immediatelySending a small % of live traffic to the new version and comparing online metrics before ramping upTesting only on the golden setDeleting the old version first
Question 8medium
Why keep the previous version one config-flip away?
To save storageSo rollback is instant — the key safety property when a change misbehaves in productionTo confuse attackersIt has no benefit
Question 9hard
A provider deprecates your pinned model. What must you do before switching to the replacement?
Switch immediately; models are interchangeableRe-run the golden set on the replacement model and compare scores; re-tune the prompt if it regressesIncrease temperature to compensateDelete the eval suite
Question 10medium
What is 'data drift' in an LLM system?
The model file becoming corruptedReal production inputs shifting away from what you built and evaluated on, quietly degrading qualityThe temperature changingLatency increasing
Question 11hard
In an incident, what is the correct FIRST action after detection?
Write a detailed post-mortemMitigate — roll back to the last known-good version before diagnosingRetrain the modelDiagnose the root cause fully before touching anything
Question 12hard
What is the value of the post-mortem step (adding the failing case to the golden set)?
It makes the incident report longerThe specific regression can never ship silently again — every incident permanently strengthens the eval suiteIt deletes the bad versionIt has no lasting effect
Question 13medium
What experiment-tracking tool (name one) is commonly used to version LLM/ML runs, params, and metrics?
Question 14hard
Complete the incident rule: '____ before diagnose' — roll back first, investigate second.

0 / 14 answered

LLMOps Lifecycle — Practice quiz