On this page
Key takeaways
- Debug a wrong RAG answer in order: retrieval, then chunking, then the question, and only then the model.
- Permission filters must apply at query time, before generation, not after.
- Build the eval set from real questions and grow it from real failures. There is no magic size.
- Compare models by cost per completed answer and quality, not price per token.
Retrieval augmented generation (RAG) and evaluation are where AI engineering interviews get practical. Anyone can describe a vector database. Fewer people can explain why a RAG system gave a wrong answer, or prove that a change made it better.
This bank has twenty questions in two parts. For each we list what a strong answer covers. Use them to find gaps, not to memorise lines.
Part 1: RAG
Try it first: turn on practice mode to hide the answers, think through each question, then reveal.
A RAG assistant gives a wrong answer to a policy question. Walk me through what you check.
What a strong answer covers
first, did the right document come back from retrieval? Then, was it chunked so the answer survived intact? Then, was the question ambiguous? Only after that, the model and the prompt. Most candidates jump straight to the model.
How do you choose a chunking strategy?
What a good answer covers
chunk by document structure (sections, headings, tables) rather than a fixed character count, keep enough context for each chunk to stand alone, and test chunk choices against an eval set.
What is hybrid search and when does it help?
What a good answer covers
combining keyword search with vector search, so exact terms such as product codes and names are found as well as paraphrases.
What does a re-ranker do?
What a good answer covers
takes a larger set of retrieved candidates and orders them by relevance to the question before they go to the model, trading some latency for precision.
How do you make retrieval permission aware?
What a good answer covers
filter by the user's access at query time, so a user never sees content from a document they could not open themselves. Filtering after generation is too late.
When should you not use RAG?
What a good answer covers
exact lookups, structured tables and live values belong to an API or a database query, not to text retrieval.
Your documents change every week. What breaks?
What a good answer covers
stale index entries, renamed terms, deleted documents still being retrieved. A good answer includes a re-index schedule and a check that source and index match.
How do you handle a question the documents do not answer?
What a good answer covers
detect low retrieval relevance, say that the answer is not in the sources, and do not let the model fill the gap.
How do you show sources to the user?
What a good answer covers
cite the retrieved passages or documents used, link to them, and test that citations actually support the answer.
How would you reduce the cost of a RAG system?
What a good answer covers
retrieve fewer but better chunks, put the stable part of the prompt first so caching can help, and route simple questions to a smaller model.
Part 2: Evals
How do you build your first eval set?
What a good answer covers
real past questions, including messy and out of scope ones; expected answers written with a domain owner; a grading method agreed in advance.
How big should the eval set be?
What a good answer covers
big enough to cover the main question types and known failure cases; start small and grow it from real failures. There is no magic number.
How do you evaluate retrieval separately from generation?
What a good answer covers
check whether the right document or passage was in the retrieved set for each question, independent of the final answer.
When is LLM as judge acceptable?
What a good answer covers
when the judge is checked against human grades on a sample, the rubric is specific, and the judge model is not simply grading its own outputs without checks.
What is a regression eval?
What a good answer covers
running the full eval set on every prompt, model, retrieval or data change, and blocking the change if previously passing cases fail.
Accuracy improved but users are less happy. What happened?
What a good answer covers
the eval set does not match real traffic, or it measures the wrong thing (correct but too long, correct but missing a citation).
How do you turn user complaints into evals?
What a good answer covers
every complaint becomes a test case the same week, with the expected answer confirmed by the owner.
What do you monitor after launch?
What a good answer covers
answer quality on sampled traffic, retrieval hit rate, refusal rate, latency, cost per answer, and new question types appearing.
How do you compare two models for this system?
What a good answer covers
same eval set, same retrieval, cost per completed answer and latency alongside quality, not price per token alone.
Explain your eval approach to a non-technical owner.
What a good answer covers
"here are real questions, here is what correct looks like, here is how often we get it right today, and here is how we will know if it gets worse."
How to practise
Take one RAG system you have built or used. Answer questions 1, 7, 11 and 15 for that system specifically. If you can not answer one, that is the gap to work on.
The AIXL Academy Forward-Deployed AI Engineering cohort covers retrieval, evals and production handover on a real build, assessed on what you ship.
Written by
AIXL Academy
AIXL Academy trains engineers to build and ship AI systems for real customers. Subscribe by RSS or follow the AIXL Academy page on LinkedIn for new question banks.
Keep reading
Interview questions
LLM evals cheat sheet: what to check, how, and when
A one page guide to evaluating LLM and agent systems: code checks, rubrics, model judges, eval sets and when to run them.
Interview questions
AI engineering system design interview questions, with what good answers cover
Common system design questions for AI engineer and forward deployed engineer interviews, and what a strong answer covers for each.