Hybrid Catalog Search
Find and fix why a hybrid keyword and vector search misses products.
Task
Search over a product catalog combines a keyword ranking with a vector ranking. Customers say it misses products they can describe. You have the search code, a catalog with precomputed embeddings, fifteen labeled queries and an eval that reports recall. Find what is wrong, fix one thing at a time, and be ready to defend each decision with a number.
Requirements
- run(catalog, request) answers one request. A product is {id, sku, title, tags, category, in_stock, vec}. A query is {text, vec, filters?}; filters may set category, in_stock or both. {op: "search", query, k} returns {results}: at most k product ids, best first. Do not change the arguments.
- Two rankings. Keyword: 5 points for each query word found in the title and 2 for each found in the tags; products with 0 points are left out. Vector: cosine similarity between the query vec and the product vec. Each ranking is best first, ties by id, and keeps its best 6. {op: "rankings", query} returns {keyword, vector} as [id, score] pairs, cosines rounded to 4 places.
- Fusion is reciprocal rank fusion: a product gets 1 / (60 + rank) from each ranking it appears in, rank counted from 1, and the two are added. Results are ordered by that sum, ties by id, then cut to k.
- Filters come first: a product that fails a filter is not ranked by either ranking and is never returned.
- Recall. {op: "recall", runs} takes runs of {id, relevant, results} and returns {recall, hits, total, missed}. A run is a hit only when the relevant product id is in its results. recall is hits / total rounded to 4 places (0.0 for no runs) and missed lists the ids of the runs that were not hits. {op: "eval", queries, k} searches each labeled query {id, text, vec, filters?, relevant} and returns the same report.
- Errors: k must be a positive integer (ValueError containing k), vec must match the catalog embedding size (vec), filters may hold only category and in_stock (filters), and any other op raises ValueError containing op.
How you are graded
- Decomposition 15%
- Prompting the assistant 15%
- Verification 20%
- Judgment about AI output 15%
- Correctness 25%
- Communication 10%
Scores use evidence from your session. A short or incomplete session may not provide enough evidence for every skill. See a sample scorecard
Interview guidance
Search over our product catalog combines keyword and vector scores. Customers say it misses products they can describe. Find out why and fix it. Change one thing at a time and give me a number for each change. Use the AI assistant however you like. At the end I will ask you to defend each decision.
- AI is allowed and expected: your own Claude, OpenAI or Gemini key, or the free in-browser model.
- You are graded on how you drive and check the AI, not on how much you prompt.
- Run the checks; read what fails.
- A new requirement will arrive partway through.
- Too little AI use is reported as “not enough to grade”, never as a fail.
You will answer 8 debrief questions at the end
- What did you delegate to the AI, and why?
- Name one AI suggestion you rejected or changed, and why.
- Choose one block the AI helped write. Explain what each line does and which test verifies it.
- What is still risky, and what would you test or ship next?
- In what order did you make your changes, and what was recall after each one? Which queries moved?
- Why did you combine the two rankings the way you did? What would make you choose a different method?
- One labeled query may still be missed. What would you do about it, and what would you not do?
- Which suggestion from the assistant did you turn down, and what evidence did you use?