AI Agents & RAG55 minutes
Device Reliability Agent
Build the step function of a read-only investigation agent.
Python
Task
Build a Device Reliability Agent that investigates barcode-scanner telemetry and reports emerging failures before they disrupt customers.
Requirements
- Return a new state with the same seven fields: status, steps, pending, observations, answer, seen. Never mutate inputs. Statuses are ready, tool_pending, done, or stopped. Terminal states are unchanged. max_steps is an integer from 1 to 20, not a boolean; invalid values raise ValueError containing max_steps.
- A decision event contains action. A tool action is {type: "tool", id, name, arguments}. Allow inspect_fleet, read_device_logs, read_releases, and search_runbooks only. IDs must be nonblank, previously unused strings; arguments must be a dictionary. Invalid actions raise ValueError containing action. Keep only id, name, arguments in pending, append the ID to seen, add one step, and set tool_pending. Only ready states accept decisions; otherwise raise ValueError containing pending.
- A tool_result event contains call_id and a dictionary result. Match it to pending, append {call_id, name, result} to observations, clear pending, and return to ready. A result whose call_id is already in observations is an unchanged replay. An unmatched result raises ValueError containing call_id. A result can contain an error; record it so the model can decide what to do next.
- A final action is {type: "final", text, citations}. Require nonblank text and a nonempty list of string citations. Each citation must match a document ID in a previous observation’s result.documents list. Otherwise raise ValueError containing citation. Set done, add one step, and store {text, citations} in answer.
- Before accepting any decision, if steps >= max_steps, set stopped and clear pending instead. Tool results do not consume a model step. Unknown event types raise ValueError containing event. The HTTP layer validates size and structure; this function enforces ordering, tool policy, and evidence.
How you are graded
- Decomposition 15%
- Prompting the assistant 10%
- Verification 20%
- Judgment about AI output 20%
- Correctness 25%
- Communication 10%
Scores use evidence from your session. A short or incomplete session may not provide enough evidence for every skill. See a sample scorecard
Interview guidance
This agent investigates failing warehouse scanners. You are writing the orchestration step, not the model: it decides which tool calls are allowed, records results, and checks that the final answer cites real evidence. Use the assistant as much as you like, but tell me where you think it is most likely to be wrong about policy, and show me that you checked.
- AI is allowed and expected: your own Claude, OpenAI or Gemini key, or the free in-browser model.
- You are graded on how you drive and check the AI, not on how much you prompt.
- Run the checks; read what fails.
- A new requirement will arrive partway through.
- Too little AI use is reported as “not enough to grade”, never as a fail.
You will answer 7 debrief questions at the end
- What did you delegate to the AI, and why?
- Name one AI suggestion you rejected or changed, and why.
- Choose one block the AI helped write. Explain what each line does and which test verifies it.
- What is still risky, and what would you test or ship next?
- Name one place where the assistant got the agent's policy wrong. How did you notice, and what did you change?
- Why is the step budget checked before the per-tool limit, and what would an operator see if the order were reversed?
- What evidence would justify recommending a rollout pause, and what must still require an engineer's approval?