Train/Serve Skew

Task

The fraud model has an offline AUC of 0.86, and online results are much worse. The model was trained on features from the training pipeline. In production, the scoring service builds its features with a serving pipeline that another team wrote from the same raw fields. You have forty logged serving requests, each matched to its training row by id, and a parity request that counts the rows where the two pipelines disagree. Find out why the model sees different features in production, and fix it so the two pipelines cannot drift apart again.

Requirements

1. run(request) answers one request about one row or a list of rows. A row has id, amount (a number or null), account_age_days (a number or null) and category (a string or null). Rows are never changed.
2. {op: "train_features", row} and {op: "serve_features", row} each return {features}: [log amount, account age in days, category index]. For the same row the serving features must equal the training features exactly. The training pipeline is the reference: the model was trained on its output, so what it computes does not change.
3. {op: "score", row} returns {features, score}: the serving features and the model's probability of fraud, rounded to 6 decimals.
4. {op: "parity", rows} returns {rows, mismatched, first}: how many rows were compared, how many have serving features that differ from their training features, and the id of the first such row (null when there is none).
5. An amount or account age that is not a number or null raises ValueError containing its field name. rows that are not a list raise ValueError containing rows. An unknown op raises ValueError containing op.