Data & Systems45 minutes
Prediction Review
Compute confusion counts and precision/recall.
Python
Task
Build a classifier-review application that shows how changing a decision threshold affects correct predictions, false alarms, and missed cases.
Requirements
- Each row has a unique id, score in [0, 1], and integer label 0 or 1. Predict positive when score >= threshold. Reject duplicate IDs with a ValueError containing id.
- Return tp, fp, tn, fn, precision, recall, and f1. Round defined measurements to six decimal places. Precision = tp/(tp+fp), recall = tp/(tp+fn), and f1 = 2*tp/(2*tp+fp+fn).
- Return None for a measurement with a zero denominator, not a made-up perfect score. Empty rows therefore return four zero counts and three None measurements.
- Threshold and scores must be finite numbers in [0, 1], not booleans. Labels must be integers 0 or 1, not booleans. Raise ValueError containing threshold, score, or label for invalid values. Do not mutate input.
How you are graded
- Decomposition 15%
- Prompting the assistant 15%
- Verification 20%
- Judgment about AI output 15%
- Correctness 25%
- Communication 10%
Scores use evidence from your session. A short or incomplete session may not provide enough evidence for every skill. See a sample scorecard
Interview guidance
An ML team reviews a classifier at a chosen threshold before release. Use the assistant to help write the evaluation in Python. I want to see how you handle the cases where a measurement is undefined, because a made-up perfect score here can ship a bad model.
- AI is allowed and expected: your own Claude, OpenAI or Gemini key, or the free in-browser model.
- You are graded on how you drive and check the AI, not on how much you prompt.
- Run the checks; read what fails.
- A new requirement will arrive partway through.
- Too little AI use is reported as “not enough to grade”, never as a fail.
You will answer 7 debrief questions at the end
- What did you delegate to the AI, and why?
- Name one AI suggestion you rejected or changed, and why.
- Choose one block the AI helped write. Explain what each line does and which test verifies it.
- What is still risky, and what would you test or ship next?
- Which validation in the assistant's code would have accepted bad input, and how did you test it?
- How does weighting the sampled negatives change precision, and why does recall stay the same in the first follow-up check?
- How would you choose a release threshold without tuning on the final test set?