AI engineering system design interview questions, with what good answers cover

AI engineering interviews now include a system design round, and it is not the same as a classic one. Interviewers want to see how you turn a vague business request into an AI system that can be tested, run and handed over.

Below are questions we see often, with what a strong answer covers. Use them to practise out loud.

01 / 11A structure for any answer

Whatever the question, a strong answer usually moves through six parts:

1. Clarify the task and the user. Who uses it, for what decision, and what happens if it is wrong.
2. Data and access. Where the data lives, who may see what, and how fresh it must be.
3. Retrieval or tools. Whether the system searches documents, calls systems, or both.
4. Evals and the ship bar. How you will know it works before it ships.
5. Latency and cost. A budget for each, and where they go.
6. Logging and handover. What is recorded and who runs it after launch.

Interviewers notice when you skip parts 4 to 6. Most candidates do.

02 / 11Question 1: Design an assistant that answers employee questions from internal policies

A strong answer covers:

• Clarifying which policies, which users and which languages.
• Chunking by document structure, with version and owner on each document so old versions leave the index.
• Filtering retrieval by the user's permissions at query time.
• An eval set of real questions, including ones where the right answer is "I do not know".
• Showing the source document with each answer.

03 / 11Question 2: Design an agent that updates customer records from support emails

A strong answer covers:

• The exact actions allowed, and which need a person to approve.
• Structured output validated against a schema, plus checks on meaning: dates, ids, amounts.
• What happens on failure: one retry, then hand to a person.
• An audit record for every run: who, what was read, which tools, what changed.

04 / 11Question 3: Design a system that answers questions over a product database and manuals

A strong answer covers:

• Separating the two: query the database with a tool for numbers and status, search the manuals for text.
• Hybrid search for part numbers and codes, with re ranking.
• How the agent chooses between tool and retrieval, and how you test that choice.

05 / 11Question 4: The assistant is too slow. How do you fix it?

A strong answer covers:

• Measuring each step at the 95th percentile, not the average.
• A latency budget split across retrieval, model calls and tools.
• Specific fixes: a smaller model for simple steps, fewer chunks, prompt caching with a stable prefix, tools that return less.
• Showing progress to the user for longer tasks.

06 / 11Question 5: Costs doubled last month. Walk us through it.

A strong answer covers:

• Cost per completed task, not per token.
• Finding the route or step that changed: more calls per task, longer prompts, retries, a cache that stopped hitting.
• Routing simple requests to a smaller model, with evals on both paths.

07 / 11Question 6: How would you know this system is ready to ship?

A strong answer covers:

• An eval set from real cases, agreed with the business.
• A ship bar and critical cases that block release on a single failure.
• A low access test user for permission checks.
• A plan to run the evals on every prompt, model, retrieval and tool change.

08 / 11Question 7: The build team leaves next month. What do you hand over?

A strong answer covers:

• A runbook for the person on call.
• The eval set and how to add a case.
• One named owner and an escalation path.
• An overlap period where the new owner handles real issues.

09 / 11How to practise

Pick one question a day. Talk through the six parts in under fifteen minutes, then write down what you skipped. Practising with real constraints, like a messy dataset or a strict latency target, is closer to the interview than practising in the abstract.

10 / 11Practise with feedback

If you want to practise system design on realistic builds, with your work reviewed, the AIXL Academy Forward-Deployed AI Engineering cohort is built around exactly these scenarios.

Details and enrolment: AIXL Academy Forward-Deployed AI Engineering cohort

11 / 11Sources

• None: AIXL Academy curriculum practice, no external facts or figures.