RAG Answer Failure Diagnosis
An AI deep dive. The interviewer describes a system that retrieves well and answers badly, then waits.
The questions in this round
Retrieval returns the right passages, but the answer still contradicts them about a third of the time. What do you look at, and in what order?
You find the retrieved set often holds two passages that contradict each other: an old policy and the one that replaced it. Now what?
The fix works on your test set. How do you know it works in production, where you have no ground truth?