Data & Systems45 minutes
Feature Drift Monitor
Compute per-feature PSI drift between training and live data.
Python
Task
Build a feature drift monitor that compares live model inputs with the training sample and flags the features that have shifted.
Requirements
- features maps each feature name to "numeric" or "categorical"; anything else, or an empty mapping, raises ValueError containing feature. reference and live are nonempty lists of row objects (ValueError containing sample otherwise). Every row must have every feature (ValueError containing feature). Other keys are ignored.
- Numeric values must be finite numbers, not booleans (ValueError containing numeric). Bin them with edges from the REFERENCE sample only: sort it, take the values at positions n*1//4, n*2//4 and n*3//4, and drop duplicate edges. A value goes in the bin numbered by how many edges are at or below it, so values outside the reference range fall in the first or last bin.
- Categorical values must be strings (ValueError containing categorical). The bins are the reference categories plus one bin for every live category the reference never saw.
- PSI = sum over bins of (live% - ref%) * ln(live% / ref%), where each % is the bin count divided by that sample's size and a 0% is replaced by 0.0001. Round PSI to four decimals; status is "stable" below 0.1, "watch" below 0.25, otherwise "drift". Return [{feature, psi, status}] sorted by psi descending, then feature ascending. Do not mutate input.
How you are graded
- Decomposition 15%
- Prompting the assistant 15%
- Verification 20%
- Judgment about AI output 15%
- Correctness 25%
- Communication 10%
Scores use evidence from your session. A short or incomplete session may not provide enough evidence for every skill. See a sample scorecard
Interview guidance
This monitor decides whether we page someone about a model's inputs: a false "drift" costs someone a night's sleep, and a false "stable" leaves a degraded model in production. Use the AI assistant as you would at work. Before you accept its binning code, tell me where the bin edges come from and what happens to an empty bin.
- AI is allowed and expected: your own Claude, OpenAI or Gemini key, or the free in-browser model.
- You are graded on how you drive and check the AI, not on how much you prompt.
- Run the checks; read what fails.
- A new requirement will arrive partway through.
- Too little AI use is reported as “not enough to grade”, never as a fail.
You will answer 7 debrief questions at the end
- What did you delegate to the AI, and why?
- Name one AI suggestion you rejected or changed, and why.
- Choose one block the AI helped write. Explain what each line does and which test verifies it.
- What is still risky, and what would you test or ship next?
- Where did the assistant's binning differ from the stated rule, and which check or hand calculation showed it?
- Why should a live window with nine values report insufficient data rather than a PSI?
- Before paging someone about a drifted feature, what else would you check about the feature and the model?