CodeBaobab
Data & Systems50 minutes

Eval Runner

Build a small evaluation runner.

Python

Task

You need a small evaluation runner before comparing two model changes. A scripted model and sixty tagged cases are supplied. Build the report, keep failed calls visible, and show which previously passing cases now fail. Start with a working evaluator before adding extra features.

Requirements

  1. Implement evaluate(cases, options) using the supplied fake_model. No network or API key is needed.
  2. Support exact-match and contains-all scoring using the normalization rules in docs/spec.md. Reject invalid case definitions.
  3. Return one result per case, including model exceptions. Keep every attempted case in the accuracy denominator.
  4. Report overall and per-tag results separately. A case with two tags counts once overall. Sort results and regressions by id, and tag summaries by tag.
  5. Compare against options.baseline and report pass-to-fail regressions. Preserve inputs and use the exact report shape in the spec.
How you are graded
  • Decomposition 15%
  • Prompting the assistant 15%
  • Verification 20%
  • Judgment about AI output 15%
  • Correctness 25%
  • Communication 10%

Scores use evidence from your session. A short or incomplete session may not provide enough evidence for every skill. See a sample scorecard

Interview guidance

I will review whether your numbers mean what their labels say. Use the AI assistant to build the runner, then walk me through a small report by hand.

  • AI is allowed and expected: your own Claude, OpenAI or Gemini key, or the free in-browser model.
  • You are graded on how you drive and check the AI, not on how much you prompt.
  • Run the checks; read what fails.
  • A new requirement will arrive partway through.
  • Too little AI use is reported as “not enough to grade”, never as a fail.
You will answer 7 debrief questions at the end
  • What did you delegate to the AI, and why?
  • Name one AI suggestion you rejected or changed, and why.
  • Choose one block the AI helped write. Explain what each line does and which test verifies it.
  • What is still risky, and what would you test or ship next?
  • Why is the overall accuracy not the average of the tag accuracies?
  • How does a model exception affect the report and its denominator?
  • Which part of the generated aggregation did you verify by hand?