AIMentor — AI Expertise Learning Platform

13 questions · pass mark 80% · 20 minute limit. The timer auto-submits at zero.

Time remaining: 20:00

Question 1easy
Why is evaluation described as what separates a demo from a product?
It makes the model respond fasterIt tells you the feature works on inputs you haven't tried and hasn't silently regressed after a changeIt reduces token costIt replaces the need for prompts
Question 2easy
What is the difference between offline and online evaluation?
Offline runs a fixed dataset before shipping (a regression gate); online measures real production trafficOffline needs no datasetOnline is only for trainingThey are the same
Question 3medium
Why tag eval cases (easy / adversarial / edge) and slice results?
Tags speed up scoringA single aggregate score hides failures; slicing shows e.g. '95% overall but 60% on adversarial'Tags are required by the APITo reduce the dataset size
Question 4medium
When is a reference-based metric (exact match, embedding similarity) appropriate?
When there is a known-good reference answer to compare againstOnly for creative writingNever for classificationOnly when no reference exists
Question 5medium
For open-ended output (summaries, chat) with no single correct answer, which approach fits?
Exact string matchReference-free evaluation, e.g. LLM-as-judge against a rubricBLEU onlyNo evaluation is possible
Question 6medium
Why prefer pairwise comparison over absolute scores for subjective quality?
Pairwise is cheaper to computeHumans and LLM judges are far more reliable at 'is A better than B?' than at assigning an absolute scoreAbsolute scores are illegalPairwise needs no judge
Question 7hard
In pairwise LLM judging, why run each pair in both orders (A,B) and (B,A)?
To double the dataset sizeTo control for position bias — judges often favor whichever option comes first; require agreementBecause the API caches resultsIt has no effect
Question 8hard
What turns eval metrics into an actual engineering safeguard?
Running them manually once at launchWiring a regression suite into CI that fails the build when quality drops below a thresholdPosting scores in a chat channelIncreasing temperature
Question 9hard
Which are common evaluation mistakes? (Select all that apply)
Evaluating on the same data you tuned the prompt onReporting only a single aggregate numberOver-trusting BLEU/ROUGE as quality measures for free textSlicing results by tag
Select all that apply.
Question 10medium
Why should the eval dataset be kept separate from prompt-tuning examples?
To save disk spaceOtherwise you measure memorization of the tuning data, not generalizationBecause the API requires itIt makes scoring faster
Question 11hard
A change raises the overall eval score but drops the adversarial slice below its threshold. How should CI treat it?
Pass — overall went upFail — a per-slice threshold guards the weak slice, and this is a regression you'd otherwise missIgnore slices entirelyOnly warn, never block
Question 12medium
What is the labelled dataset of test cases an eval runs against commonly called (two words)?
Question 13hard
For subjective quality, what metric (two words) reports how often config B beats config A across the dataset?

0 / 13 answered

Eval Design — Final Exam