Data & Systems45 minutes

Structured Output Parser

Write tests for a parser that pulls JSON out of model replies and fix what they show.

JavaScript

Task

This module pulls one JSON object out of a model reply and checks it against a schema. It fails in production now and then. You have forty real replies and two tests. Write tests that pin down what the parser must do, then fix the parser. Your tests are graded too: each one must state the exact result, and the suite must fail on wrong versions of the parser.

Requirements

  1. check(request) is the only function the checks call. {op: "parse", reply, schema} returns parseReply(reply, schema). reply must be a string (Error containing reply). Any other op raises an Error containing op. Do not change the arguments.
  2. Finding the object. If the reply has fenced blocks, the object comes from the first fenced block that holds one. Otherwise it comes from the prose. The answer is the first balanced {...} span, reading left to right, that parses as a JSON object. Braces inside JSON strings do not count. When nothing parses, return {ok: false, reason: "no_json"}.
  3. Repair. Before parsing, a comma that is followed only by white space and then } or ] is removed. Text inside JSON strings is never changed.
  4. Validation. A schema is a list of {name, type, required}; type is string, number, boolean, array or object. A valid object returns {ok: true, value}, with the value exactly as parsed and extra keys kept. Otherwise return {ok: false, reason: "invalid", errors}, where errors lists {field, problem} in schema order. problem is "missing" for an absent required field and "wrong_type" for a value of another type. Values are never converted, and null is a wrong type.
  5. Tests. TESTS holds {name, reply, schema, expected}. {op: "suite"} returns {failed}: the names of tests that fail on your parser. {op: "catches", variant} returns {caught}: true when a test that passes on your parser fails on that variant (Error containing variant when unknown). {op: "covers", outcome} returns {covered}: true when a passing test expects that outcome ("ok", or a reason).
How you are graded
  • Decomposition 15%
  • Prompting the assistant 15%
  • Verification 20%
  • Judgment about AI output 15%
  • Correctness 25%
  • Communication 10%

Scores use evidence from your session. A short or incomplete session may not provide enough evidence for every skill. See a sample scorecard

Interview guidance

This module pulls a JSON object out of model replies, and it fails in production now and then. Here are forty real replies and two tests. Write tests first, then fix the parser. Use the AI assistant however you like. I will ask you why each test exists and how you know its expected value is right.

  • AI is allowed and expected: your own Claude, OpenAI or Gemini key, or the free in-browser model.
  • You are graded on how you drive and check the AI, not on how much you prompt.
  • Run the checks; read what fails.
  • A new requirement will arrive partway through.
  • Too little AI use is reported as “not enough to grade”, never as a fail.
You will answer 7 debrief questions at the end
  • What did you delegate to the AI, and why?
  • Name one AI suggestion you rejected or changed, and why.
  • Choose one block the AI helped write. Explain what each line does and which test verifies it.
  • What is still risky, and what would you test or ship next?
  • Pick one of your tests. Which wrong parser does it catch, and how do you know its expected value is right?
  • What does your parser now refuse that the shipped one accepted? Who downstream is affected by that change?
  • Where did the assistant help most, and which of its suggestions did you turn down?