LLM Context Capacity Planning
A product manager has promised a customer a 32,000-token context. You are the one who has to say whether that is true.
The questions in this round
A product manager wants a 32,000-token context at 16 concurrent requests on one accelerator. Can we, and what would you change if not?
It does not fit. The PM asks what it would take. Give me three options and what each one costs.
They pick lower concurrency. What breaks at two o’clock on a Monday?