The Tuesday Prompt Tweak
A prompt change shipped on Tuesday. It improved one kind of answer and broke another, and the team found out on Friday from a customer.
Catching a regression that the average hides
An evaluation set reports one number, and one number can stay flat while one category improves and another collapses. The release gate has to look at the categories, not the total.
Your task
Complete EvalHarness.run one part at a time. Part one is in the brief below; the next arrives once its tests pass.
Pairing tasks
Part 1 — Score the run
Run a candidate answerer over the evaluation set and report how it did: the accuracy across every case, and the accuracy within each slice.
- Report the accuracy across every case as overall.
- Report one accuracy per slice as by_slice.
Part 2 — Stop the average hiding it
Here is the case that matters. A change makes the answerable questions better and the unanswerable ones worse, and the total barely moves. Your report has to make that visible, and the gate has to name the slice that regressed rather than reporting a slightly lower number.
- Keep each slice’s accuracy independent, so an offsetting move cannot cancel out.
- Check that the gate blocks the release and names the regressed slice.
Explain why an aggregate score can rise while a slice falls, and state the tolerance you would set for a gate that runs on every prompt change.
Interviewer follow-up questions
- A slice drops two points out of twelve. Is that a regression or noise, and how would you tell?
- Why does an unanswerable-question slice deserve separate reporting?
- What do you do when the gate blocks a change you know is an improvement?
- Where does this gate sit relative to sampled judging on live traffic?
Skills: Evaluation sets · Slice analysis · Release gates · Regression testing