RAG Latency Diagnosis
A latency review. p95 is four seconds against a one-second target, and the team has already asked for bigger GPUs.
The questions in this round
A retrieval-augmented assistant has a p95 of four seconds against a one-second target, and the team wants bigger GPUs. Walk me through your diagnosis.
Here it is: 120ms embedding, 80ms search, 900ms rerank, 400ms prefill, 2.4s decode. Where do you go?
Product says users are fine as long as something appears quickly. Does that change your answer?