LLM Cost Optimization
A cost exercise with real numbers on the table. A hundred thousand users, ten requests each, two thousand input and four hundred output tokens. Finance wants the bill halved.
The questions in this round
A hundred thousand users, ten requests each, two thousand input and four hundred output tokens. Finance wants the bill halved. Where do you start?
You route easy requests to a smaller model. How do you decide what counts as easy, and how will you know you were right?
The cache you added is at a 60% hit rate, and support is reporting stale and slightly-wrong answers. What went wrong?