Multi-Tenant LLM Serving
A serving design round. One model, several internal teams, one pool of accelerators, and the interviewer wants to know how you would divide it up. Expect the requirements to move.
The questions in this round
Six product teams want the same model on one pool of accelerators. Design the serving layer.
One of those teams runs an overnight batch job that would happily consume every accelerator you have. Another serves a live product.
The live product now wants a 32,000-token context, and concurrency has to stay where it is.
What fails first here, and what does the on-call engineer see when it does?