Training Script Review
Review an AI-written training script and fix how it measures its model.
Task
An AI assistant wrote this training script for a churn model. It runs, and it reports 97 percent accuracy on a table where few customers churn. Before anyone trusts that number, review the script against the team's evaluation rules in docs/evaluation.md. Rank your findings by how much they change what the number means, then fix the blocking ones so the script reports an honest result.
Requirements
- train(request) trains a logistic regression and returns a report. request has data (a list of {x, y} rows, or {rows, seed, positive_rate} for the built-in table generator) and the optional seed, epochs, learning_rate and patience.
- The script must follow every rule in docs/evaluation.md. The checks test each rule through the report.
- The report has the keys that docs/evaluation.md lists. Every float is rounded to 6 decimals. {show: [keys]} returns only those keys of the report (ValueError containing show for a key the report does not have).
- Keep the model, the split function and the settings as they are. The task is how the model is measured, not how good it is.
- Invalid input raises ValueError naming the problem: rows (fewer than 5), label, features, epochs. The script already checks its input.
How you are graded
- Decomposition 15%
- Prompting the assistant 15%
- Verification 20%
- Judgment about AI output 15%
- Correctness 25%
- Communication 10%
Scores use evidence from your session. A short or incomplete session may not provide enough evidence for every skill. See a sample scorecard
Interview guidance
An assistant wrote this training script, and it reports 97 percent accuracy on our churn table. Review it before we trust the number. Rank your findings by how much they change what the number means, and fix the blocking ones. You can use the AI assistant for the review, but its opinion is not evidence. For each finding, show me the line.
- AI is allowed and expected: your own Claude, OpenAI or Gemini key, or the free in-browser model.
- You are graded on how you drive and check the AI, not on how much you prompt.
- Run the checks; read what fails.
- A new requirement will arrive partway through.
- Too little AI use is reported as “not enough to grade”, never as a fail.
You will answer 8 debrief questions at the end
- What did you delegate to the AI, and why?
- Name one AI suggestion you rejected or changed, and why.
- Choose one block the AI helped write. Explain what each line does and which test verifies it.
- What is still risky, and what would you test or ship next?
- What were your findings, in rank order? Which one changes the meaning of the reported number the most?
- What is the honest result for the example table, in one sentence a product manager could act on?
- Which parts of the script did you decide to leave alone, and why?
- The assistant gave you an opinion of the script. Where did it help, and where did you have to check it?