# AI engineering system design interview questions, with what good answers cover

URL: https://aixl.academy/blog/ai-engineering-system-design-interview-questions/
Published: 2026-11-08
Updated: 2026-11-08
Author: AIXL Academy
Topics: ai engineering, system design, interview questions, rag, agents

AI engineering interviews now include a system design round, and it is not the same as a classic one. Interviewers want to see how you turn a vague business request into an AI system that can be tested, run and handed over.

Below are questions we see often, with what a strong answer covers. Use them to practise out loud.

## A structure for any answer

Whatever the question, a strong answer usually moves through six parts:

1. **Clarify the task and the user.** Who uses it, for what decision, and what happens if it is wrong.
2. **Data and access.** Where the data lives, who may see what, and how fresh it must be.
3. **Retrieval or tools.** Whether the system searches documents, calls systems, or both.
4. **Evals and the ship bar.** How you will know it works before it ships.
5. **Latency and cost.** A budget for each, and where they go.
6. **Logging and handover.** What is recorded and who runs it after launch.

Interviewers notice when you skip parts 4 to 6. Most candidates do.

## Question 1: Design an assistant that answers employee questions from internal policies

A strong answer covers:

- Clarifying which policies, which users and which languages.
- Chunking by document structure, with version and owner on each document so old versions leave the index.
- Filtering retrieval by the user's permissions at query time.
- An eval set of real questions, including ones where the right answer is "I do not know".
- Showing the source document with each answer.

## Question 2: Design an agent that updates customer records from support emails

A strong answer covers:

- The exact actions allowed, and which need a person to approve.
- Structured output validated against a schema, plus checks on meaning: dates, ids, amounts.
- What happens on failure: one retry, then hand to a person.
- An audit record for every run: who, what was read, which tools, what changed.

## Question 3: Design a system that answers questions over a product database and manuals

A strong answer covers:

- Separating the two: query the database with a tool for numbers and status, search the manuals for text.
- Hybrid search for part numbers and codes, with re ranking.
- How the agent chooses between tool and retrieval, and how you test that choice.

## Question 4: The assistant is too slow. How do you fix it?

A strong answer covers:

- Measuring each step at the 95th percentile, not the average.
- A latency budget split across retrieval, model calls and tools.
- Specific fixes: a smaller model for simple steps, fewer chunks, prompt caching with a stable prefix, tools that return less.
- Showing progress to the user for longer tasks.

## Question 5: Costs doubled last month. Walk us through it.

A strong answer covers:

- Cost per completed task, not per token.
- Finding the route or step that changed: more calls per task, longer prompts, retries, a cache that stopped hitting.
- Routing simple requests to a smaller model, with evals on both paths.

## Question 6: How would you know this system is ready to ship?

A strong answer covers:

- An eval set from real cases, agreed with the business.
- A ship bar and critical cases that block release on a single failure.
- A low access test user for permission checks.
- A plan to run the evals on every prompt, model, retrieval and tool change.

## Question 7: The build team leaves next month. What do you hand over?

A strong answer covers:

- A runbook for the person on call.
- The eval set and how to add a case.
- One named owner and an escalation path.
- An overlap period where the new owner handles real issues.

## How to practise

Pick one question a day. Talk through the six parts in under fifteen minutes, then write down what you skipped. Practising with real constraints, like a messy dataset or a strict latency target, is closer to the interview than practising in the abstract.

## Practise with feedback

If you want to practise system design on realistic builds, with your work reviewed, the AIXL Academy Forward-Deployed AI Engineering cohort is built around exactly these scenarios.

Details and enrolment: [AIXL Academy Forward-Deployed AI Engineering cohort](https://aixl.academy/workshops/forward-deployed-ai-engineering/?utm_source=aixl.academy&utm_medium=blog&utm_campaign=fde-cohort&utm_content=aixl-2026-11-16-blog-cta)

## Sources

- None: AIXL Academy curriculum practice, no external facts or figures.