The Cache That Must Not Leak
A multi-tenant assistant serves several companies from one deployment, and the same question costs the same money every time it is asked.
Reusing an answer without reusing it for the wrong person
Exact-match caching almost never hits, because users phrase the same question differently. Comparing embeddings raises the hit rate, and raises the risk with it: a cache keyed only on the text of a question will hand one company an answer computed from another company documents.
Your task
Complete SemanticCache.lookup one part at a time. Part one is in the brief below; the next arrives once its tests pass.
Pairing tasks
Part 1 — Reuse the closest answer
Questions arrive as embedding vectors. Return the stored response whose vector is closest to the incoming one, provided it clears the similarity threshold, and None when nothing is close enough.
- Return the response of the closest stored entry at or above the threshold.
- Return None when no stored entry is close enough.
Part 2 — Keep it inside the tenant
The cache now serves several customers, and the prompt template is versioned. Two customers can ask a near-identical question; one must never receive the other’s answer, and an answer produced under an old prompt version must not survive the change.
- Never return an entry stored under a different tenant, model or prompt version.
Part 3 — Report what it saved
Finance wants to know what the cache is actually buying. Every lookup has to count toward the reported rate, including the ones that miss for the boundary reasons you just added.
- Record a hit or a miss on every lookup so the reported rate reflects all of them.
State what belongs in the cache key and why each part must be there, then explain why the threshold should be tuned against false hits rather than for maximum coverage.
Interviewer follow-up questions
- Two questions differ only by the word "not". What does your cache do, and what should it do?
- Why is prompt version part of the key when the user never sees it?
- Would you rather raise the threshold or lower it, and what does each choice cost?
- How would you measure whether this cache is saving money without degrading answers?
Skills: Cosine similarity · Cache keys · Multi-tenancy · Cost control