Data & Systems40 minutes

Flag Rollout

Find why a 20 percent feature rollout flips for the same user.

Python

Task

A feature flag service decides who sees the new checkout while it rolls out to 20 percent of users. Support says some users see the new checkout on one visit and the old one on the next. The team's own tests pass. Read the ticket, find the cause in the rollout code, and fix it so that every user gets one stable answer.

Requirements

  1. evaluate(flags, request) answers one request. flags maps a flag name to its config, such as {"percent": 20}. percent is an integer from 0 to 100 (ValueError containing percent otherwise). An unknown flag raises ValueError containing flag. Do not change the arguments.
  2. {op: "bucket", flag, user} returns {bucket}: an integer from 0 to 99, computed from the flag name and the user ID as docs/bucketing.md defines. The same flag and user always give the same bucket. user is a non-empty string (ValueError containing user otherwise).
  3. {op: "check", flag, user} returns {enabled, reason}. A flag at percent P is on for exactly P of the 100 buckets, the lowest numbered ones. reason is "rollout" when the flag is on and "outside_rollout" when it is off.
  4. {op: "audience", flag, users} returns {enabled, count}: the users from the list who get the flag, in the order given, and how many there are. users is a list (ValueError containing users otherwise). Any other op raises ValueError containing op.
How you are graded
  • Decomposition 15%
  • Prompting the assistant 15%
  • Verification 20%
  • Judgment about AI output 15%
  • Correctness 25%
  • Communication 10%

Scores use evidence from your session. A short or incomplete session may not provide enough evidence for every skill. See a sample scorecard

Interview guidance

Support says some users see the new checkout on one visit and the old one on the next. The rollout is set to 20 percent, and the team's own tests pass. Find the cause and fix it. Use the AI assistant however you like. Before you edit anything, tell me where you think the problem is and how you would prove it.

  • AI is allowed and expected: your own Claude, OpenAI or Gemini key, or the free in-browser model.
  • You are graded on how you drive and check the AI, not on how much you prompt.
  • Run the checks; read what fails.
  • A new requirement will arrive partway through.
  • Too little AI use is reported as “not enough to grade”, never as a fail.
You will answer 7 debrief questions at the end
  • What did you delegate to the AI, and why?
  • Name one AI suggestion you rejected or changed, and why.
  • Choose one block the AI helped write. Explain what each line does and which test verifies it.
  • What is still risky, and what would you test or ship next?
  • What made the same user get different answers, and how did you prove it before you changed code?
  • The team's tests passed the whole time. What would a test need to do to catch this?
  • Which fixes did you or the assistant consider and drop, and why?