# RAG and evals interview questions for AI engineers

URL: https://aixl.academy/blog/rag-evals-interview-questions/
Published: 2026-11-06
Updated: 2026-11-06
Author: AIXL Academy
Topics: rag, llm evals, interview questions, ai engineering

Retrieval augmented generation (RAG) and evaluation are where AI engineering interviews get practical. Anyone can describe a vector database. Fewer people can explain why a RAG system gave a wrong answer, or prove that a change made it better.

This bank has twenty questions in two parts. For each we list what a strong answer covers. Use them to find gaps, not to memorise lines.

## Part 1: RAG

**1. A RAG assistant gives a wrong answer to a policy question. Walk me through what you check.**
Strong answer: first, did the right document come back from retrieval? Then, was it chunked so the answer survived intact? Then, was the question ambiguous? Only after that, the model and the prompt. Most candidates jump straight to the model.

**2. How do you choose a chunking strategy?**
Covers: chunk by document structure (sections, headings, tables) rather than a fixed character count, keep enough context for each chunk to stand alone, and test chunk choices against an eval set.

**3. What is hybrid search and when does it help?**
Covers: combining keyword search with vector search, so exact terms such as product codes and names are found as well as paraphrases.

**4. What does a re-ranker do?**
Covers: takes a larger set of retrieved candidates and orders them by relevance to the question before they go to the model, trading some latency for precision.

**5. How do you make retrieval permission aware?**
Covers: filter by the user's access at query time, so a user never sees content from a document they could not open themselves. Filtering after generation is too late.

**6. When should you not use RAG?**
Covers: exact lookups, structured tables and live values belong to an API or a database query, not to text retrieval.

**7. Your documents change every week. What breaks?**
Covers: stale index entries, renamed terms, deleted documents still being retrieved. A good answer includes a re-index schedule and a check that source and index match.

**8. How do you handle a question the documents do not answer?**
Covers: detect low retrieval relevance, say that the answer is not in the sources, and do not let the model fill the gap.

**9. How do you show sources to the user?**
Covers: cite the retrieved passages or documents used, link to them, and test that citations actually support the answer.

**10. How would you reduce the cost of a RAG system?**
Covers: retrieve fewer but better chunks, put the stable part of the prompt first so caching can help, and route simple questions to a smaller model.

## Part 2: Evals

**11. How do you build your first eval set?**
Covers: real past questions, including messy and out of scope ones; expected answers written with a domain owner; a grading method agreed in advance.

**12. How big should the eval set be?**
Covers: big enough to cover the main question types and known failure cases; start small and grow it from real failures. There is no magic number.

**13. How do you evaluate retrieval separately from generation?**
Covers: check whether the right document or passage was in the retrieved set for each question, independent of the final answer.

**14. When is LLM as judge acceptable?**
Covers: when the judge is checked against human grades on a sample, the rubric is specific, and the judge model is not simply grading its own outputs without checks.

**15. What is a regression eval?**
Covers: running the full eval set on every prompt, model, retrieval or data change, and blocking the change if previously passing cases fail.

**16. Accuracy improved but users are less happy. What happened?**
Covers: the eval set does not match real traffic, or it measures the wrong thing (correct but too long, correct but missing a citation).

**17. How do you turn user complaints into evals?**
Covers: every complaint becomes a test case the same week, with the expected answer confirmed by the owner.

**18. What do you monitor after launch?**
Covers: answer quality on sampled traffic, retrieval hit rate, refusal rate, latency, cost per answer, and new question types appearing.

**19. How do you compare two models for this system?**
Covers: same eval set, same retrieval, cost per completed answer and latency alongside quality, not price per token alone.

**20. Explain your eval approach to a non-technical owner.**
Covers: "here are real questions, here is what correct looks like, here is how often we get it right today, and here is how we will know if it gets worse."

## How to practise

Take one RAG system you have built or used. Answer questions 1, 7, 11 and 15 for that system specifically. If you can not answer one, that is the gap to work on.

The AIXL Academy Forward-Deployed AI Engineering cohort covers retrieval, evals and production handover on a real build, assessed on what you ship.