Repair the LangChain RAG Tests

A customer-support assistant answers policy questions and hands uncertain cases to a person.

Testing a chatbot that must know when not to answer

The assistant invokes a LangChain-compatible retriever and answer Runnable, answers only from retrieved Documents, and keeps histories isolated by customer.

Your task

The application code is complete and read only. Repair only tests/test_chatbot.py so its expectations match the documented contract and still catch real regressions.

Pairing tasks

1. Read the implementation and written contract before running the suite.
2. Identify which expectations misunderstand the production behavior.
3. Repair only the editable pytest file; do not weaken or delete assertions.
4. Make every corrected test pass the working implementation and fail its matching regression.

Explain the production contract before editing a test. For every change, state why the old expectation was wrong and which regression the corrected assertion detects.

Interviewer follow-up questions

• What belongs inside the LangChain Runnable and outside it?
• Why is fluency not a success signal?
• Where would you enforce customer isolation in production?

Skills: LangChain · Runnables · RAG · Grounded output