Repair the Production RAG Monitoring Tests

A deployed RAG assistant returns HTTP 200 even when retrieval is empty or its citations are invalid.

Monitoring the RAG behavior that HTTP status cannot see

The monitor records retrieval, citations, abstention, latency, and tenant boundaries. Its implementation works, while the tests currently treat every fluent response as healthy.

Your task

The application code is complete and read only. Repair only tests/test_rag_monitor.py so its expectations match the documented contract and still catch real regressions.

Pairing tasks

1. Read the implementation and written contract before running the suite.
2. Identify which expectations misunderstand the production behavior.
3. Repair only the editable pytest file; do not weaken or delete assertions.
4. Make every corrected test pass the working implementation and fail its matching regression.

Explain the production contract before editing a test. For every change, state why the old expectation was wrong and which regression the corrected assertion detects.

Interviewer follow-up questions

• Why does HTTP success miss RAG failures?
• Which metrics should be sliced by tenant and document type?
• When is abstention the correct production behavior?

Skills: LLM observability · RAG monitoring · Metrics · Alerting