LLM Evaluation and Release Gates

An average score can hide a broken use case

Keep evaluation cases grouped by the job the application performs: retrieval, grounded answers, refusals, structured output, or tool selection. Include cases from actual failures after removing sensitive information.

Version the inputs, expected checks, grading method, and candidate configuration. A changed grader can move a score even when the application did not change.