Redaction Pull Request
Review an assistant-written pull request that redacts personal data from prompt logs and fix the blocking findings.
Task
An assistant wrote this pull request. It removes email addresses and phone numbers from prompts before they are logged. Review it as if it ships today: read the code against the logging policy, rank what you find, and fix the findings that block the release. The checks call one function with a model call and look at the log records it would write.
Requirements
- handle(request) answers {op: "log", event} with the list of log records for one model call, and {op: "leaks", event, secrets} with {leaked}: the secrets found anywhere in those records. An event is {request_id, messages: [{role, content}], error?}. Roles are system, user, assistant and tool (ValueError containing role, content or request_id for a bad event; ValueError containing op for an unknown op). Do not change the arguments.
- No personal data in any record. An email address or a phone number becomes a typed placeholder: [EMAIL_1], [PHONE_1], and so on. Nothing made from the value is logged: no hash, no part of it, and no table of values.
- Placeholders are numbered per kind in reading order across the whole event. The same address (ignoring case) or the same number (ignoring punctuation and a leading country code 1) gets the same placeholder everywhere in that event.
- Phone numbers are ten digits, with an optional +1 or 1 in front, optional parentheses around the first three digits, and a space, dot, hyphen or nothing between the groups. Email addresses may have capitals, dots, hyphens and plus tags.
- The first record is {event: "prompt", level: "info", request_id, messages, redacted}. Each logged message is {role, content, chars}: content is redacted and then cut to 120 characters, and chars is the length of the original content. redacted is {EMAIL, PHONE}: how many different values of each kind the event held.
- When the event has an error, a second record follows: {event: "model_error", level: "error", request_id, message}, where message is "failed: <error> | prompt: <last user message>". The error text is redacted, and the last user message appears as it was logged ("" when there is none).
How you are graded
- Decomposition 15%
- Prompting the assistant 15%
- Verification 20%
- Judgment about AI output 15%
- Correctness 25%
- Communication 10%
Scores use evidence from your session. A short or incomplete session may not provide enough evidence for every skill. See a sample scorecard
Interview guidance
An assistant wrote this pull request. It redacts personal data before we log prompts. Review it as if it ships today. Rank your findings, tell me which ones block the release and why, and fix those. Use the AI assistant however you like, but the review is yours: I will ask how you proved each finding.
- AI is allowed and expected: your own Claude, OpenAI or Gemini key, or the free in-browser model.
- You are graded on how you drive and check the AI, not on how much you prompt.
- Run the checks; read what fails.
- A new requirement will arrive partway through.
- Too little AI use is reported as “not enough to grade”, never as a fail.
You will answer 7 debrief questions at the end
- What did you delegate to the AI, and why?
- Name one AI suggestion you rejected or changed, and why.
- Choose one block the AI helped write. Explain what each line does and which test verifies it.
- What is still risky, and what would you test or ship next?
- Give your findings in rank order. Which one would you fix first if you had ten minutes, and why?
- Was there anything you flagged at first and then decided to leave alone? How did you decide?
- What did you ask the assistant to do in this review, and what did you keep for yourself?