Production LLM Quality Monitoring
An operations round. The interviewer asks how you would find out your LLM feature is broken, and is listening for whether you describe a web-service dashboard.
The questions in this round
How would you know your LLM feature is broken before your users tell you?
Your dashboard is green. 200s, p95 fine, no errors. A customer emails to say the answers have been wrong for a week. What was missing?
You add sampled online judging. What does it still not catch?