Capacity Planning for AI Systems
A design-document round for a customer-support assistant: open-weight models, a hundred concurrent users, grounded answers, under two seconds, plus analytics and a cost estimate.
The questions in this round
Design it: open-weight models, a hundred concurrent users, answers grounded in the company help centre, under two seconds, and finance wants a monthly figure.
Your budget puts retrieval at 300 milliseconds and generation at 1.2 seconds. Measured, generation takes 2.4. Where does the extra second come from?
Finance reads your estimate and halves the budget. Nothing about the promise changes.
The brief asks for analytics. What goes on the dashboard, and what do you look at first when someone says it got worse?