AIMentor — AI Expertise Learning Platform

15 questions · pass mark 80% · 20 minute limit. The timer auto-submits at zero.

Time remaining: 20:00

Question 1easy
Why does RAG evaluation score the retriever and generator separately?
To use two different embedding modelsA single accuracy score can't tell whether a wrong answer came from bad retrieval or bad generationBecause retrieval is always the problemTo reduce the number of API calls
Question 2easy
What is a 'golden set' in RAG evaluation?
The most expensive model availableA curated set of test cases with questions, expected answers, and the ids of the relevant chunksThe vector index itselfA cache of previous answers
Question 3medium
What should a good golden set include? (Select all that apply)
Hard questions where the answer is buried or split across chunksAbstain cases the corpus genuinely cannot answerOnly easy, happy-path questionsA manageable size (e.g. 30–100 well-chosen cases) rather than thousands of sloppy ones
Select all that apply.
Question 4medium
Context recall measures:
Of the chunks that should have been retrieved, how many wereOf the retrieved chunks, how many were relevantWhether the answer is grounded in the contextHow fast retrieval runs
Question 5medium
Context precision measures:
Of the chunks that should have been retrieved, how many wereOf the retrieved chunks, how many were actually relevantHow relevant the answer is to the questionThe cosine similarity threshold
Question 6hard
You measure high context recall but low context precision. What does this point to?
The answer chunk isn't being retrieved — widen retrievalThe answer chunk IS retrieved but surrounded by irrelevant ones — add reranking or metadata filteringThe generator is hallucinatingThe embedding model is broken
Question 7medium
Which metric catches an answer that invents facts even though the retrieved context was correct?
Context recallContext precisionFaithfulness (groundedness)Latency
Question 8medium
Why is 'LLM-as-judge' used for faithfulness and answer relevance?
Because these have no cheap deterministic formula, so a model scores against a rubricBecause it is always cheaper than computing recallBecause judges never make mistakesBecause embeddings cannot be compared otherwise
Question 9hard
Which practices reduce LLM-as-judge error? (Select all that apply)
Use temperature 0 for deterministic gradingGive a crisp rubric and ask for a reasonPeriodically spot-check the judge's verdicts against human labelsTrust the judge fully and never audit it
Select all that apply.
Question 10medium
What is the purpose of a RAG regression harness?
To generate embeddings fasterTo run the metrics over the golden set and produce a scorecard you compare across changesTo replace the vector databaseTo rewrite user queries automatically
Question 11hard
Between two configs on the same golden set, recall is unchanged but faithfulness rises from 0.78 to 0.91 after adding reranking. What does this indicate?
Retrieval got betterThe change helped the generator ground better (tighter context), not first-pass retrieval coverageThe golden set is invalidThe embedding model changed
Question 12medium
What is the difference between offline and online RAG evaluation?
Offline uses the golden set before shipping; online samples and scores live production traffic to catch driftOffline needs the internet, online does notThey are the same thingOnline evaluation does not use metrics
Question 13easy
Which are established frameworks that package RAG metrics like context precision/recall and faithfulness?
NumPy and PandasRagas and DeepEvalDocker and Kubernetespgvector and Qdrant
Question 14medium
Name the labelled ground-truth dataset (two words) that RAG metrics are measured against.
Question 15hard
Which generator-side metric (one word) measures whether every claim in the answer is supported by the retrieved context?

0 / 15 answered

RAG Evaluation — Final Exam